Grading only the final answer records that a model was wrong, not which fact it got wrong. STRAND pairs every target question with the atomic sub-questions it rests on, and credits a model only when the answer and all of its supporting facts are correct.
National University of Singapore · Centre for AI Research, VinUniversity
†Corresponding author — thong.nguyen@u.nus.edu
The primary metric
A model earns credit for a target question only if it answers that question and every supporting sub-question correctly. The denominator is all 977 targets, so a model cannot raise its score by being selectively consistent on the subset it happens to get right.
Faithful Accuracy (%) on STRAND. The three hatched rows are blind references: a majority-class predictor, a question-only predictor that never sees the video, and a caption-only predictor. Scores should be read against these rather than against 50%.
The benchmark
Rather than generating question–answer pairs from raw video, STRAND begins from a human-verified world model of each video — objects, their states, their relations, and the moments these change. Targets are composed from that record and paired with the atomic sub-questions they depend on.
Videos span sports, egocentric, surveillance and instructional domains, at roughly 22 videos per domain. Because that is thin, we report domain breakdowns as descriptive only and make no claim that the ordering of models is stable within any one domain.
Annotators anchor a discrete set of key timestamps at salient interactions, state transitions and occlusions.
Task-relevant entities receive canonical, visually grounded identifiers such as player_24.
An entity is registered only if it is re-identifiable — a human can point to the same entity at two
separated timestamps from visual evidence alone — and dynamic, meaning it takes part in at least one
state change or interaction. Static background elements are excluded.
Observable states and relations are recorded at each timestamp as typed tuples such as
(t, o₁, passing_to, o₂), drawn from a three-part schema of attribute, action and relation.
No bounding boxes: the failure mode STRAND targets is symbolic binding drift, not localisation error.
Entities are traced across the timeline into trajectories. Observation windows need not be contiguous — when an entity is occluded, leaves the frame and re-enters, the gap is recorded explicitly and an identity link is asserted across it only when re-identification is supported by visual evidence such as a jersey number. Otherwise the two segments stay distinct entities. Facts an annotator can infer but not observe are never recorded.
Llama-3.1-70B-Instruct organises the annotated facts into reasoning chains, composes a target question from a chain, and decomposes it into the atomic sub-questions the target draws on. The generator operates only on the symbolic fact tuples and never sees the video, so any error it introduces is one of phrasing or unsupported inference rather than invented visual detail. It is also disjoint from every evaluated model family, so no evaluated system shares pretraining lineage with the model that phrases its questions.
Annotators then review every generated item alongside the source video and check that the language is clear and natural, that the sub-questions are jointly sufficient to answer the target and that each probes a fact the target actually draws on, and that every answer is unambiguously grounded in visual evidence. Items carrying unsupported assumptions or unreliable identity bindings are discarded. An alternative sufficient fact set exists for 5.8% of targets.
| Model | Full video | Restricted input | ||
|---|---|---|---|---|
| Afaith | Atarget | Afaith | Atarget | |
| Single centre frame | ||||
| Gemini-3.1-Pro | 38.9 | 61.4 | 14.1 | 31.4 |
| Gemini-3-Flash | 28.7 | 48.7 | 12.5 | 29.8 |
| GPT-5 | 20.0 | 32.5 | 10.2 | 28.2 |
| Qwen3.5-27B | 45.5 | 68.4 | 13.8 | 32.7 |
| No visual input | ||||
| Question-only (Llama-3.1-70B) | — | — | 21.4 | 52.6 |
| Caption-only | — | — | 31.2 | 59.8 |
Restricted to the ordering sub-questions, the question-only predictor scores 49.3% — no better than guessing. The order of events has to be read off the video. Of the 382 temporally constrained targets, 361 (94.5%) carry an ordering sub-question.
Main results
No end-to-end model exceeds 68.4% target accuracy, and several fall below 50%. Three systems score below the 19.1% majority-class Faithful Accuracy: without reliable temporal tracking they cannot ground a target answer in the facts it rests on.
| Method | Afaithprimary ↑ | Atarget↑ | Asub↑ | Aconsreference |
|---|---|---|---|---|
| Ours | ||||
| Ours (Gemini-3-Flash) | 59.4 | 77.1 | 76.3 | 81.3 |
| Ours (Qwen3-VL-235B) | 54.2 | 74.0 | 72.6 | 78.1 |
| Proprietary end-to-end MLLMs | ||||
| Gemini-3.1-Pro | 38.9 | 61.4 | 64.9 | 69.2 |
| Gemini-3-Flash | 28.7 | 48.7 | 61.7 | 69.9 |
| Gemma-4-27B | 35.8 | 55.5 | 60.0 | 72.7 |
| GPT-5 | 20.0 | 32.5 | 43.2 | 70.5 |
| Claude-4.6-Sonnet | 30.2 | 48.3 | 62.9 | 73.2 |
| Open-weight end-to-end MLLMs | ||||
| InternVL3-78B | 29.7 | 49.9 | 66.1 | 71.0 |
| Qwen3-VL-32B Think | 46.1 | 65.0 | 70.6 | 78.1 |
| Qwen3.5-27B | 45.5 | 68.4 | 66.7 | 73.3 |
| Cosmos-Reason2-8B | 41.5 | 54.7 | 74.7 | 83.7 |
| VideoRFT-7B | 16.4 | 38.6 | 55.1 | 50.8 |
| Video-R1-7B | 4.3 | 24.8 | 19.6 | 21.9 |
| Structured pipelines | ||||
| VideoMind-7B | 18.9 | 50.3 | 37.9 | 43.2 |
| TraveLER | 22.0 | 50.6 | 41.8 | 49.3 |
| SeViLA | 26.1 | 51.4 | 59.6 | 56.8 |
The Gemini-3-Flash instantiation reaches 59.4 Afaith, ahead of Gemini-3.1-Pro by 20.5 points, with gains of 15.7, 11.4 and 12.1 on the remaining three metrics. That margin sets the scale of the effect; it does not isolate its cause, which the matched-budget comparisons and ablations address.
What remains hard
Splitting sub-question accuracy by probe type locates the residual errors. Both instantiations identify events far more reliably than they order them, and the gap is roughly 18 points on either backbone.
| System | Individual facts | Order of two events | Gap |
|---|---|---|---|
| Ours (Gemini-3-Flash) | 78.8 | 61.4 | −17.4 |
| Ours (Qwen3-VL-235B) | 75.4 | 56.2 | −19.2 |
| Model | Object halluc. | State misattr. | Temporal order |
|---|---|---|---|
| Ours (Gemini-3-Flash) | 12.4 | 34.1 | 53.5 |
| Gemini-3.1-Pro | 38.2 | 36.5 | 25.3 |
| Qwen3.5-27B | 33.7 | 38.4 | 27.9 |
Our framework suppresses object hallucination and state misattribution, so ordering becomes the dominant remaining failure at 53.5% of its errors — a 19.4-point margin over the next category. Because these are shares of each model's own errors, the shift reflects what is left once the other modes are reduced, and it matches the ordering gap above. Ordering is the capability STRAND isolates, and the one that remains open.
| Model | State changen = 200 | Identity trackingn = 277 | Action sequencingn = 300 | Multi-object interactionn = 200 |
|---|---|---|---|---|
| Ours (Gemini-3-Flash) | 70.0 | 49.8 | 52.0 | 73.0 |
| Gemini-3.1-Pro | 50.0 | 31.1 | 32.7 | 48.0 |
| Qwen3.5-27B | 58.0 | 37.9 | 38.0 | 55.0 |
Our framework
Because STRAND credits an answer only when the facts supporting it are correct as well, we build an answering system that represents those facts explicitly. A video becomes a set of object trajectories, each recording how one object's state changes over time. The representation is built once per video and reused by every question about it.
The video is partitioned into disjoint 15-second chunks and 60 frames are sampled from each. A VLM-based extractor processes every chunk independently, so extraction parallelises. A state is relational rather than purely attributive: it carries the predicate the object participates in together with its arguments.
Chunk-level observations contain duplicate objects, noisy states and identities fragmented across time. A deterministic symbolic aggregator consolidates them — it issues no model call, and its parameters (Δtmax, τconf) are fixed hyperparameters, not learned weights. Four steps: similarity scoring over visual attributes and spatial proximity; constraint filtering on temporal proximity and a confidence threshold; bipartite conflict resolution prioritised by temporal adjacency, giving a one-to-one identity mapping; and trajectory generation, in which observations are merged and sorted by timestamp and anything that matches nothing survives as a singleton rather than being discarded.
Not every trajectory bears on a given question, so a retriever selects the query-relevant subset. For the stronger backbone this acts mainly as a context-length control rather than a source of accuracy; the weaker backbone is markedly more sensitive to it.
The answer generator receives the retrieved trajectories, the question, and a fixed uniformly sampled 64-frame budget that is independent of the question — so among the video-derived inputs only the retrieved subset varies per query. Operating over structured trajectories rather than raw frames lets this stage address object identities, state transitions and temporal relations directly.
gemini-3-flash-preview for extraction, retrieval and generation.qwen3-vl-235b-a22b-thinking for extraction,
qwen3-235b-a22b for retrieval, qwen3.5-27b for generation.Unless ablated, both use 15-second chunks, 60 frames per chunk, 64 frames at the answerer, and identity linking. Every prompt is given verbatim in the paper's appendix, for our framework and for every baseline.
Evidence for the design
A harness brings extra compute as well as a different representation. The leaderboard above does not separate the two, so we hold the backbone fixed and equalise one resource at a time. These comparisons, not the end-to-end rows, are what the design claim rests on.
| Constraint | End-to-end baseline | Ours | Δ |
|---|---|---|---|
| Equal total frame budget (64 frames) | 41.5 | 54.2 | +12.7 |
| Equal number of model calls (single pass) | 39.8 | 49.3 | +9.5 |
| Equal token budget | 45.1 | 52.8 | +7.7 |
| Configuration | Gemini-3-Flash | Qwen3-VL-235B |
|---|---|---|
| Base configuration | 77.1 | 74.0 |
| No retrieval filter (full timeline) | 77.0tie | 64.6−9.4 |
| Random retrieval filter (top-k) | 76.5tie | 68.0−6.0 |
| LLM summarisation instead of symbolic aggregation | 76.7tie | 62.7−11.3 |
| Answerer lowered to 8 frames | 76.5tie | 63.6−10.4 |
| Dense prose captions instead of structured states | 72.4−4.7 | 61.2−12.8 |
| Shuffled trajectory order | 65.3−11.8 | 54.1−19.9 |
| Text-only answering (no frames at the answerer) | 73.6−3.5 | 58.1−15.9 |
| Configuration | Afaith | Atarget |
|---|---|---|
| Base — Δtmax = 15 s, τconf = 0.75, bipartite | 54.2 | 74.0 |
| Temporal constraint | ||
| Δtmax = 5 s (strict — fragments trajectories) | 50.4 | 71.2 |
| Δtmax = 30 s (relaxed — false-positive matches) | 51.1 | 72.8 |
| Confidence threshold | ||
| τconf = 0.90 (high precision) | 48.9 | 70.5 |
| τconf = 0.50 (high recall) | 51.2 | 71.8 |
| Conflict resolution | ||
| Greedy instead of bipartite matching | 47.5 | 69.4 |
Generality and cost
| Method | VideoHallucer | Video-MME | EgoSchema |
|---|---|---|---|
| Ours (Gemini-3-Flash) | 76.8 | 76.8 | 78.4 |
| Ours (Qwen3-VL-235B) | 71.4 | 73.6 | 76.4 |
| Gemini-3.1-Pro | 63.1 | 77.3 | 76.8 |
| GPT-5 | 67.5 | 76.2 | 75.4 |
| Qwen3-VL-235B | 59.8 | 72.6 | 72.1 |
| LLaVA-OV | 48.5 | 66.3 | 60.1 |
The Gemini instantiation leads the strongest baseline by 9.3 points on VideoHallucer and 1.6 on EgoSchema, and trails Gemini-3.1-Pro on Video-MME by 0.5. The Qwen instantiation improves on its own extractor backbone, run end-to-end, by 11.6 points on VideoHallucer.
| System | Pre-proc. (s) | Per-query (s) | Calls | Input tokens | Output tokens |
|---|---|---|---|---|---|
| Ours (Gemini-3-Flash)* | 14.2 | 3.2 | 91.6 | 962,400 | 18,100 |
| Ours (Qwen3-VL-235B) | 18.4 | 2.9 | 91.6 | 958,157 | 17,450 |
| Gemini-3.1-Pro* | 0.0 | 3.5 | 39.7 | 654,415 | 1,985 |
| Qwen3.5-27B | 0.0 | 2.8 | 39.7 | 654,415 | 1,985 |
| Cosmos-Reason2-8B | 0.0 | 2.1 | 39.7 | 654,415 | 1,985 |
| VideoMind-7B | 6.5 | 4.2 | 40.7 | 710,000 | 14,000 |
| TraveLER | 8.2 | 3.8 | 119.1 | 820,000 | 12,000 |
| SeViLA | 5.1 | 3.1 | 79.4 | 690,000 | 9,500 |
Our framework buys its 5.6-point target-accuracy gain over Qwen3.5-27B with more wall-clock time, more model calls and a larger token budget. That is the trade the budget-matched comparisons are designed to price.
Qualitative analysis
The paired evaluation surfaces reasoning that target-only scoring would credit in full.
These cases illustrate why target accuracy alone can overstate faithful video understanding. The quantitative form of the claim is the gap between Atarget and Afaith in the main table: Gemini-3.1-Pro answers 61.4% of targets correctly but supports only 38.9% of them fully.
Benchmark items
Each target is paired with the atomic sub-questions it rests on. Under Faithful Accuracy, every one of them has to be right. Play a clip and the dependency becomes plain: the answer turns on something that exists only across time.
Pure temporal ordering
Are the cookies smashed before the strawberries?No
Supporting sub-questions
Identity through occlusion
Is the white object under the brown object after the woman lifts up both objects?Yes
Supporting sub-questions
Paired target
Is the yellow object under the brown object after the woman lifts up both objects?No
State tracking across repeated trials
In the first experiment, did the yellow ball go to door number 2?Yes
Supporting sub-questions
Paired target
In the first experiment, did the yellow ball go to door number 1?No
Actor disambiguation over time
Does the blue team start playing first?Yes
Supporting sub-questions
Paired target
Does the black-haired woman start playing first?No