Evidence-Backed Video Question Answering

2026-07-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

Evidence-Backed Video Question Answering (E-VQA) is a new task designed to address the black-box nature of current Video Large Language Models (Video LLMs), which provide answers without verifiable visual grounding. Existing explainability methods struggle with complex video dynamics like occlusions. E-VQA requires models to output both a semantic answer and precise spatio-temporal evidence, including temporal segments and dense, tracked object segmentation masklets. To support this, the ST-Evidence benchmark was introduced, offering the first human-verified pixel-level grounding for discriminative and generative tasks. Evaluations revealed a critical decoupling between QA accuracy and true visual perception in state-of-the-art models, which scaling alone does not resolve. To bridge this, researchers developed ST-Evidence-Instruct, a 160k-scale dataset generated automatically, linking high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data achieved significant improvements, such as +27.2 t-mean and +13.8 J&F on a 7B model, establishing a robust baseline for explainable video understanding.

Key takeaway

For Machine Learning Engineers developing explainable video understanding systems, recognize that simply scaling Video LLMs does not guarantee verifiable visual grounding. You should prioritize integrating explicit spatio-temporal evidence mechanisms, such as dense object segmentation masklets and temporal segments, into your model architectures. Consider fine-tuning your grounded Video LLMs with datasets like ST-Evidence-Instruct to bridge high-level reasoning with fine-grained visual perception, establishing a more robust and transparent system.

Key insights

Explainable Video LLMs require precise spatio-temporal evidence, a gap current scaling efforts fail to bridge.

Principles

• QA accuracy can decouple from visual perception.
• Model scaling alone doesn't ensure grounding.
• Fine-grained grounding improves reasoning.

Method

E-VQA proposes jointly outputting semantic answers with temporal segments and dense, tracked object segmentation masklets. This is supported by automated dataset generation and fine-tuning grounded Video LLMs.

In practice

• Utilize ST-Evidence for pixel-level grounding.
• Leverage ST-Evidence-Instruct for fine-tuning.
• Access code and data at SalesforceAIResearch/EVQA.

Topics

• Video Question Answering
• Video LLMs
• Explainable AI
• Spatio-Temporal Grounding
• Object Segmentation
• ST-Evidence Benchmark
• Dataset Generation

Code references

• SalesforceAIResearch/EVQA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Grounding Video Reasoning in Physical Signals — Physical video understanding requires grounding events in time and space, beyond just naming them correctly.
• Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning — SegAnswer enhances MLLM visual reasoning by replacing bounding box "zoom-in" with precise pixel-level segmentation masks for better grounding.
• TRACE: Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generation — The "ground-before-reasoning" strategy improves multi-video event understanding by localizing evidence with a text LLM before visual reasoning.
• GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning — GeoWeaver grounds visual tokens with token-adaptive geometric evidence as a representational prerequisite for spatio-temporal reasoning.
• Q-Fold: Query-Aware Focus-Context Spatio-Temporal Folding for Long Video Understanding — Q-Fold improves long-video understanding by creating query-guided Focus-Context representations from temporal segments, balancing fidelity and coverage.
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs — MLLM video understanding benefits from a human-like "watch, remember, reason" functional decomposition.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.