MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

A new inference-time framework addresses the unreliability of timestamp predictions in Multimodal Large Language Models (MLLMs) for Video Temporal Grounding (VTG). Researchers found that MLLMs often identify the correct event interval during the prefill stage, where query tokens attend to video content, but lose this temporal signal during autoregressive decoding as answer tokens shift attention to irrelevant segments. The framework identifies "Temporal Grounding Heads" (TG-Heads) through an attention knockout study, which are sparse attention heads crucial for temporal localization. It then extracts a debiased frame-level relevance signal from these TG-Heads' prefill attention. This signal guides a "read-then-regenerate" process, where the MLLM is re-invoked with visual context restricted to the detected high-attention interval via video cropping or attention masking. This method consistently improved MiMo-VL-7B, Qwen3-VL-8B, and TimeLens-8B on Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens benchmarks, achieving gains up to +3.5 mIoU without model parameter updates.

Key takeaway

For AI Engineers deploying MLLMs for video temporal grounding, you can significantly improve timestamp prediction accuracy without costly retraining. Implement the "read-then-regenerate" framework by identifying Temporal Grounding Heads and using their prefill attention to restrict visual context during decoding. This approach surfaces latent temporal grounding capabilities, yielding gains up to +3.5 mIoU on models like Qwen3-VL-8B, making your MLLM outputs more reliable for precise event localization.

Key insights

MLLMs internally know event timings during prefill, but often lose this signal during answer generation, a gap recoverable via attention-guided re-inference.

Principles

• MLLM attention heads specialize functionally.
• A perception-generation gap exists.
• Latent capabilities are recoverable.

Method

Identify Temporal Grounding Heads (TG-Heads) via attention knockout. Extract debiased prefill attention to detect a high-attention interval. Re-invoke the MLLM with visual context restricted to this interval via cropping or attention masking.

In practice

• Identify specialized attention heads.
• Apply a two-stage inference framework.
• Restrict visual context for regeneration.

Topics

• Multimodal Large Language Models
• Video Temporal Grounding
• Attention Mechanisms
• Model Interpretability
• Inference Optimization
• Qwen3-VL
• TimeLens-Bench

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs — VIGIL mitigates MLLM visual laziness and hallucinations by causally grounding responses to visual input, penalizing "blind confidence" via counterfactual alignment.
• Understanding Data Temporality Impact on Large Language Models Pre-training — Temporally ordering pre-training data significantly improves LLM factual freshness and temporal precision without sacrificing general knowledge.
• Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads — MLLMs exhibit functional sparsity via Context-aware Retrieval (CoRe) heads, crucial for efficient cross-modal information extraction.
• Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models — VLMs struggle with genuine chronological reasoning, often relying on superficial visual shortcuts.
• MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue — Decoupling linguistic disambiguation from visual localization significantly improves multimodal grounding in dynamic dialogue.
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.