Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

2026-07-13 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

A new study titled "Do Video-LLMs Actually Watch?" reveals that current Video Large Language Models (Video-LLMs) struggle with genuine character tracking in long-form videos, despite achieving benchmark scores of 37-38% on tasks like InfiniBench's global appearance. Researchers applied a nine-condition diagnostic protocol to three distinct open-source Video-LLMs, with Gemini2.5Flash serving as a frontier reference. The findings indicate that models largely ignore the specific named character in questions, altering their answers only 4-31% of the time when the character's name was swapped. This behavior suggests reliance on coarse gender cues, as models reacted more significantly (a 13-28 point gap) when names were changed to a different gender. Furthermore, open-ended questioning reduced open-source model accuracy by 18-25 points, with no fully correct answers, compared to a 12-point drop for Gemini. The study concludes that the bottleneck is not video input quantity but how models link video content to named individuals, releasing a diagnostic toolkit.

Key takeaway

For AI Scientists and ML Engineers evaluating Video-LLMs for long-form character-tracking tasks, you must look beyond aggregate benchmark scores. Current models often fail to genuinely track specific individuals, instead relying on superficial cues like gender. You should integrate diagnostic protocols, such as name-swapping tests and open-ended questioning, into your evaluation pipeline. This approach will reveal true model capabilities and prevent deployment of systems with shallow understanding, especially for identity-critical applications.

Key insights

Video-LLMs often fail true character tracking, relying on superficial cues despite benchmark scores.

Principles

• Benchmark scores can mask fundamental model failures.
• Models exploit coarse gender cues over specific identity.
• More video input doesn't resolve core reasoning flaws.

Method

A nine-condition diagnostic protocol, including character name swaps and open-ended questioning, was applied to Video-LLMs to expose tracking failures.

In practice

• Audit Video-LLM benchmarks using diagnostic toolkits.
• Test models with name-swapped questions for tracking.
• Employ open-ended questions to reveal shallow understanding.

Topics

• Video-LLMs
• Character Tracking
• Long-Form Video
• Model Evaluation
• Diagnostic Protocol
• Benchmark Scores

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video — Video-LLMs often fail true character tracking, relying on shallow cues despite benchmark scores.
• Can you *really* train AI to "get" videos just by showing it a million of them? — Current video models excel at generation but lack systematic evaluation for genuine spatiotemporal reasoning.
• Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur? — Video LLMs struggle with real-time mistake correction due to data scarcity, which Ego-CoMist addresses by generating synthetic training examples for proactive interventions.
• A Four-Condition Diagnostic Protocol for Evidence Utilization in Long-Context and Retrieval-Augmented Language Models — Traditional metrics are insufficient; a four-condition protocol is crucial for diagnosing true evidence utilization in LLMs.
• Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory — LLM reality monitoring, distinguishing self-generated from user input, depends critically on conversational memory structure.
• Did the Model See the Benchmark During Training? Detecting LLM Contamination — A lightweight method estimates LLM benchmark contamination by analyzing model behavior when training data is unknown.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.