Real-time Task Verification using Expert Demonstration Priors
Umar Khan · Examiner: Prof. Dr. Didier Stricker · Supervisor: MSc. Muhammad Asad Ali
A lot of human work is procedural: a fixed sequence of steps where both the order and the manner matter. Mistakes get expensive not because they are hard to see, but because nobody is watching.
One correct execution and a written instruction. No collection of recordings.
No recordings of the ways the procedure can go wrong.
Head-mounted, partial, moving. Every judgement is made while the task is still happening.
Their pace, habits and tool order all differ from the demonstrator's.
Trained on a fixed set of tasks, usually with error labels.
Assembly101 · IndustReal · CaptainCook4D · HoloAssist
Needs labelled errors
Flag executions that break an explicit procedure structure.
EgoPED · video-mined task graphs
Needs many runs to mine the graph
Model only correct execution; flag anything novel.
ATA · PREGO
Alarms it can't explain
And most report results with the true step boundaries supplied, which leaves open who finds the segments in the first place.
Can a general-purpose vision-language model, given one expert demonstration and no error supervision, follow a worker through a live stream and speak up when they deviate?
Egocentric recordings of people cooking, with every moment labelled as correct or as a type of error.
No visual templates, no reference frames, no learned parameters. Step names are the annotator's own words.
If an error clip's graph is built from its own annotation, only the Normal segments become steps. The steps done wrong disappear, and the system can't flag a step it doesn't know exists.
The fix is the one-shot assumption taken literally: each recipe's procedure comes from one normal clip.
Consequence: durations belong to another person. The reference allows 3.3 s to place the tortilla; this participant took 10.1 s.
| quesadilla_u1_a2_error_028 | Self | Ref. |
|---|---|---|
| Place tortilla on cutting board | – | ✓ |
| Use knife to scoop Nutella | – | ✓ |
| Spread Nutella onto tortilla | – | ✓ |
| Sprinkle cinnamon onto tortilla | – | ✓ |
| Put banana slices on tortilla | ✓ | ✓ |
| Fold tortilla | – | ✓ |
| Slice using knife | – | ✓ |
| Place tortilla wedges on plate | ✓ | ✓ |
| Steps in graph | 2 | 8 |
8 normal, 7 error, all five recipes. Used to compare system versions.
170 error, 17 normal · 11.9 h of video · 2,125 reference steps. The official held-out set.
Streaming only: frames once, in order, at 3 fps, with no oracle boundaries. No parameter changed between the two sets.
Both tiers run the same checkpoint, Qwen3.6-35B-A3B (FP8), served by vLLM with one GPU each. They differ in what they're asked and how the answer is read, not in their weights.
Constrain the answer to one letter and read the model's first-token distribution. Renormalise over the options and take the margin between the top two as confidence.
One forward pass, one token: a generative model becomes a classifier with a usable confidence score.
"Has step n been completed?"
Probabilities cluster around 0.53 / 0.47, close to a coin flip. At threshold 0.65 there are no false positives but also no detections; at 0.51, one detection and two false positives.
A not finished · B just completed
C already on the next step · D cannot tell
On the first clip tested, all 8 steps were found.
median boundary error
The single most important design choice in the system.
"What is happening right now?" Feeds the completion and step probes.
"Was this step done right?" Thinned evenly as the step grows. Feeds the correctness probe.
An earlier version cleared all frames at every step change, throwing away the evidence exactly when it was needed. Capping the buffer also caps the cost of the most expensive call.
Dense window, options A–D. B or C counts as evidence; an explicit A resets the count. The backbone of tracking.
Previous, current or next step, or "none of these". A moving average (ω = 0.35) is compared with a per-clip baseline to catch skipped steps and off-task activity.
Sparse buffer, deliberately neutral wording. Returns a short reading plus a yes/no.
On look-alike steps (pour grounds → pour water) the tracker raced several steps ahead in seconds.
Checking against the step probe didn't separate true from false completions. Time in step did.
mean boundary error
20 s cooldown. Nothing reaches the user without L1's verdict.
Two rules carry most of the precision:
1 · "The alert that woke you is not evidence."
2 · "A slow step is never a mistake by itself."
Stall verdicts went from 5 mistakes to 1 mistake + 8 ok. Telling the model that idling is normal didn't work; changing the decision rule did.
The frames clearly contradict the step. The only verdict that reaches the user.
No clear evidence of a deviation. Logged, then discarded.
The worker is fine; the tracker is on the wrong step. The cursor jumps silently.
Every false alarm in an earlier version began with the tracker losing sync, followed by the adjudicator blaming the worker. On the test split DESYNC fired 82 times across 70 clips: 82 false alarms that never reached the user.
system = ParityLM(PipelineConfig())
monitor = system.start_activity("clip.activity.json")
for frame in frame_source:
monitor.process_frame(frame)
if monitor.task_complete: break
report = monitor.summary()
The frame source is any iterable, so a file and a live camera differ only in the iterator.
Every process_frame call is timed. The loop is synchronous, so a window containing an L1 call is a window in which the stream was blocked, and its duration is that call's true cost.
Resumable across cluster jobs: a clip counts as done once its record exists, resuming requires matching settings, and no new clip starts close to the time limit.
<clip>.mp4the monitored video.activity.jsonthe graph it was checked against.timeline.jsonannotation, used only for scoring.run.jsondecisions, verdicts, per-window timingOpens a run-set folder directly. No server.
Driven by the measured timing: probes tick at their real rhythm, and when L1 is called the video stops while its reasoning appears over the seconds it actually took.
Drawing the pipeline honestly caught two mistakes: an expert-video panel (the system never sees expert footage) and an annotation labelled "ground truth" beside the tracker.
Coin-flip regime. ~74% of boundaries, 7.7 s error.
Median boundary error 0.7 s on the first clip.
Mean error 26.8 s → 16.9 s. No more racing ahead.
Stall mistakes 5 → 1. False alarms caused by the tracker are gone.
Smooth tracking, clean normal clips… and 0/7 errors detected.
Reference graph + correctness probe: 6/7 errors detected.
| v1 | v2 | v3 | v4 | |
|---|---|---|---|---|
| Normal clips clean | 8/8 | 8/8 | 8/8 | 4/8 |
| False alarms | 0 | 0 | 0 | 10 |
| Error clips detected | 1/7 | 0/7 | 0/7 | 6/7 |
| Clip-level outcome | 9/15 | 8/15 | 8/15 | 10/15 |
| Median boundary error | 6.3 s | 11.3 s | 8.7 s | 6.1 s* |
| Reasoning calls | 20 | – | 20 | 51 |
v1–v3 differ in hardware and buffer settings. v2's shorter dense window caused its boundary regression (confirmed by an A/B test) and was restored in v3.
v4 adds the reference graph and the correctness probe, and is carried to the test split unchanged.
* normal clips only
v1's single "detection" was luck: the tracker had stalled near an error, and the stall check happened to look at the right frames.
Once tracking was smooth, L1 was called 20 times in about 2 h of video, and almost never at an error.
v3 is an honest baseline: excellent tracking, and no way at all to notice wrong execution. Measure tracking and reporting separately, or artefacts look like progress.
error clips detected, v1 → v3
26 annotated error moments. The exact frames are supplied, the model is the same, and only the wording changes.
It wasn't perception, it was the question. The caution that keeps false alarms low also hides real mistakes, so checking alarms and checking correctness need different prompts.
error clips detected
clips judged correctly
false alarms, on 4 of 8 normal clips
Both changes come from one component. The correctness probe raised 25 of 51 L1 calls and 16 of them came back as mistakes, while stall checks behaved (17 of 19 ok). The precision loss is contained in one gate.
All other methods process the whole video offline. ZeProM-M uses a 35B model from the same family as ours; ZeProM-L uses 397B.
of 2,125 reference steps confirmed
on normal clips, where every step happens
median boundary error
within 10 s of the true end
The error is one-sided: 84% of boundaries fire early and carry 87% of the total error. The tracker falls more than 10 s behind the worker on only 7% of steps. 153 of 187 runs reach the final step.
per perception probe
per reasoning call, blocking the stream
Cost comes from images, not answer length. A 481-character verdict took 26.6 s; a 648-character one took 9.3 s. One correctness check at median cost leaves ~32 live frames unexamined.
| L1 trigger | Calls | Median | Max |
|---|---|---|---|
| Correctness | 329 | 10.7 s | 30.4 s |
| Off-task | 163 | 6.1 s | 17.2 s |
| Skipped step | 28 | 5.9 s | 17.8 s |
| Stall | 223 | 5.6 s | 22.9 s |
In one coffee clip, all 14 steps were confirmed within the first 200 s of 985 s, although the activity only starts at 492 s.
34 of 203 boundaries on normal clips are off by more than 60 s, even though the reference is correct.
30% of L1 calls were stall checks, and 181 of 223 said "ok". The worker was just slower.
Additions belong to no step, and "too short" slips can't be seen in still frames.