Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

2026-07-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

A study reveals that Video Large Language Models (Video-LLMs) do not genuinely track characters in long-form videos. Despite open-source 7-8B models scoring 37-38% on InfiniBench's global appearance task, a nine-condition diagnostic protocol found shallow processing. This protocol tested three open-source Video-LLMs, referencing Gemini 2.5 Flash. When a named character in a question was swapped, models changed answers only 4-31% of the time, largely ignoring the specific individual. Shallow processing also appeared with a 13-28 point accuracy gap for gender-swapped names, indicating reliance on coarse gender cues. Open-ended questions caused open-source accuracy to drop 18-25 points, yielding no fully correct answers from 151 attempts. The bottleneck is how models link video to named individuals, not video input quantity. A diagnostic toolkit is now available.

Key takeaway

For Machine Learning Engineers evaluating Video-LLMs for character-centric applications, recognize that high benchmark scores may not reflect genuine character tracking. Your models likely rely on shallow cues like gender, not individual identity. You should use diagnostic toolkits to audit model understanding. Specifically, test robustness to named entity changes and performance on open-ended questions. This will help you avoid deploying models with misleading capabilities for critical tasks.

Key insights

Video-LLMs often fail true character tracking, relying on shallow cues despite benchmark scores.

Principles

• Video-LLM benchmark scores can mask shallow processing.
• Coarse gender cues are often prioritized over individual identity.
• Input quantity alone does not resolve character-video linking.

Method

A nine-condition diagnostic protocol was applied to Video-LLMs, involving named character swaps, gender-based name changes, and open-ended questioning, with Gemini 2.5 Flash as a reference.

In practice

• Audit Video-LLM benchmarks for true character understanding.
• Test model robustness to named entity changes.
• Evaluate performance with open-ended questions.

Topics

• Video-LLMs
• Character Tracking
• Long-Form Video
• Benchmark Auditing
• Diagnostic Protocol
• Gemini 2.5 Flash

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video — Video-LLMs often fail true character tracking, relying on superficial cues despite benchmark scores.
• Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur? — Video LLMs struggle with real-time mistake correction due to data scarcity, which Ego-CoMist addresses by generating synthetic training examples for proactive interventions.
• Lazy or Efficient? Towards Accessible Eye-Tracking Event Detection Using LLMs — An LLM-driven pipeline automates eye-tracking event detection, making complex analysis accessible without specialized programming.
• SVHighlights: Towards Extremely Long Sport Video Highlight Detection — The core challenge of long-form video highlight detection is addressed by a new benchmark and a multimodal, segment-based LLM approach.
• Did the Model See the Benchmark During Training? Detecting LLM Contamination — A lightweight method estimates LLM benchmark contamination by analyzing model behavior when training data is unknown.
• What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectors — Persona vectors diagnose LLM behavioral organization, revealing how training cultivates, exposes, or resists specific traits, rather than merely controlling them.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.