video-SALMONN-R$^3$: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

video-SALMONN-R3 is an end-to-end video large language model designed for efficient video understanding by overcoming computational and memory constraints that typically force reduced frame rates. Developed by Tsinghua University, ByteDance, and the University of Cambridge, this model employs a two-stage "re-watch" paradigm, first performing coarse video understanding to localize relevant segments, then re-watching them at higher fidelity. Crucially, video-SALMONN-R3 acquires this re-watch capability through reinforcement learning, specifically dynamic sampling policy optimization (DAPO), without relying on expensive chain-of-thought (CoT) cold-start annotations or supervised fine-tuning. It integrates a "re-answer" strategy to refine initial predictions after re-watching and a "re-ask" mechanism to re-inject queries for improved question adherence. Experimental results demonstrate that video-SALMONN-R3 consistently outperforms base models and prior re-watch approaches with significantly lower computational cost across six benchmarks.

Key takeaway

For Machine Learning Engineers developing video LLMs for complex question answering, video-SALMONN-R3 demonstrates a critical shift. You should consider adopting reinforcement learning for adaptive re-watch mechanisms, as it achieves superior fine-grained understanding and lower computational costs compared to CoT-dependent or single-pass methods. This approach allows you to preserve base model capabilities while integrating efficient temporal localization.

Key insights

video-SALMONN-R3 efficiently understands video by using reinforcement learning to re-watch relevant segments at higher fidelity, bypassing CoT.

Principles

• RL for re-watch avoids CoT cold-start.
• Re-answer anchors new ability to prior.
• Re-ask strengthens query-video interaction.

Method

video-SALMONN-R3 uses a two-pass workflow: first, a low-fidelity pass generates an initial answer and localizes a segment. Second, a high-fidelity re-watch of the segment, with re-injected query, refines the answer via RL-optimized re-answer and re-ask mechanisms.

Topics

• video-SALMONN-R3
• Video Large Language Models
• Reinforcement Learning
• Temporal Localization
• Question Answering
• Computational Efficiency

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• video-SALMONN-R$^3$: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding — Video-SALMONN-R$^3$ uses RL-driven re-watching, re-asking, and re-answering to efficiently improve video LLM question answering without CoT.
• Temporal-Aware Reasoning Optimization for Video Temporal Grounding — TaRO improves MLLM video temporal grounding by optimizing time-aware reasoning through guided exploration and quality-focused rewards.
• ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding — Optimizing visual input selection via reinforcement learning significantly enhances video-LLM contextual understanding and reasoning.
• VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation — Multi-turn RL with calibrated segmentation feedback enables adaptive, precise video object segmentation.
• EvoVid: Temporal-Centric Self-Evolution for Video Large Language Models — EvoVid enables Video-LLMs to self-evolve temporal reasoning from raw video, eliminating human annotation dependency.
• Rethinking RAG in Long Videos: What to Retrieve and How to Use It? — VideoRAG performance improves by using chunk-adaptive retrieval and reranking, addressing limitations of existing benchmarks and single-configuration approaches.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.