ANCHOR
An answer is only as strongas the evidence behind it.
Ground the answer.
Audit the reasoning.
Audio-Visually GrouNded Chain-of-THOught
Reasoning Benchmark
Does a model really see and hear what it claims? ANCHOR connects reasoning to the frames, regions, and sounds that justify an answer.
00:52Joel Julin and Souraja Kundu contributed equally (co-first authors).
Many worlds.
One grounding challenge.
From a sound in a crowded scene to an action unfolding over minutes: explore what ANCHOR asks models to understand.
Arc sizes organize the hierarchy, not frequency.
Percentages show the reported share of all QA pairs.
Categories were assigned by GPT-5.2. Rounded joint percentages may not sum exactly to category totals.
AVQA · Ego4D · THUMOS-14 · Animal-Bench · MAVERIX · COIN · MovieChat · Video-MME · LVBench
4.9 seconds to 114.27 minutes
Multiple-choice and open-ended questions.
See the evidence.
Then inspect the reasoning.
Twelve curated clips from ANCHOR reveal how questions, answers, reasoning traces, and spatial evidence fit together.
Examples are loaded from the supplied ANCHOR annotations. Videos load only when selected; playback is muted initially. Bounding boxes follow the nearest annotated frame.
Right answer.
Wrong reasons?
A plausible explanation can miss the evidence. ANCHOR makes the connection between perception and reasoning inspectable.
What is the amount of lime juice she uses?
00:52Representative frames reproduced from the paper; this is a still-frame walkthrough.
Measures juice.
The measuring cup makes the final measurement visible. The scene must be interpreted together with the spoken explanation.
“Half of a lime usually equates to half of an ounce.”
The later audio resolves the amount. Stopping at the earlier “add an ounce” instruction leads to the wrong answer.
Gemini-3-Flash
Reaches the reference answer, but offers generic cues without precise spatiotemporal grounding.
GPT-5.2
Answers “1 ounce” after relying on an earlier interval and missing the later audio evidence.
When did it happen?
Representative frames and timeframes tie claims to the relevant moment.
Where is the evidence?
Spatial bounding boxes identify the objects and regions a reasoning step references.
What can be heard?
Salient audio cues connect speech and acoustic events to the explanation.
Grounding makes
a measurable difference.
High answer scores do not guarantee faithful explanations. Controlled cue removal reveals what the model depends on.
Take away the evidence.
Answer accuracy · Gemini-3-Flash
Table 4 · Human-refined test set, 975 QA pairs. Binary answer correctness judged by GPT-5-nano. Differences are percentage points from each model’s full-cue result.
Vision is the foundation.
Across all three evaluated models, the audio-only setting causes the largest drop by a wide margin.
accuracy with audio alone
Spatial and audio cues have model-dependent effects.
The faithfulness gap
Paper-reported model scores. Higher is better for all four scores; fewer tokens means shorter responses.
| Model | Answer | Hallucination¹ | Divergence¹ | Conciseness | Tokens |
|---|
¹ Higher hallucination and divergence scores indicate better evidence support and less logical drift. Answer, hallucination, and divergence: three-judge 1–5 Likert scores. Conciseness: token-based quintile score, 1–5. These are not accuracy percentages.
Machine-assisted.
Human-refined.
A shared annotation interface connects video, audio, bounding boxes, timelines, and reasoning edits.
Curate
Select audio-visually grounded QA pairs from nine existing benchmarks.
Anchor
Generate initial reasoning traces and SAM3 proposals for spatial evidence.
Refine
26 participants correct evidence spans, boxes, and reasoning for 224 videos and 975 QA pairs.
Evaluate
Audit answer correctness, faithfulness, logical divergence, and conciseness.
Read the research overview
ANCHOR is a benchmark and evaluation framework for auditing perceptual grounding in multimodal reasoning. Models produce grounded chain-of-thought explanations that explicitly reference representative frames, spatial bounding boxes, and salient audio cues. The benchmark contains 2,319 videos and 8,272 question–answer pairs, including a human-refined subset of 224 videos and 975 QA pairs. Experiments expose a faithfulness gap: models that answer correctly can still produce explanations that are weakly grounded in perceptual evidence.
Scope and limitations: the full collection includes automatically generated reasoning traces; the human-refined subset is smaller. Coverage inherits the source datasets, and evaluation relies in part on model judges. See the paper for the complete protocol and limitations.
NeurIPS 2026 · Evaluations & Datasets Track · Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark
Cite this work
@inproceedings{julin2026anchor,
title={ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark},
author={Julin, Joel and Kundu, Souraja and Dahiya, Liza and Wei, George Z and
Xie, Liuyue and Yi, Jingru and Zhang, Xu and Yang, Hao and Jeni, Laszlo A.},
booktitle={Advances in Neural Information Processing Systems},
year={2026}
}