ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QA

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

ReQuest is an uncertainty-driven, question-adaptive keyframe selection pipeline designed to improve long-form video Question Answering (QA) for Multimodal Large Language Models (MLLMs) operating under fixed input token budgets. It addresses the inefficiency of uniform sampling for evidence localization in long videos. The pipeline integrates a lightweight question-aware selector, distilled from MLLM-generated supervision, a Re-thinking Routing mechanism that triggers additional inference only when the model is uncertain, and uncertainty-guided adaptive non-maximum suppression for selecting temporally diverse frames. As a plug-and-play method, ReQuest enhances long-video QA without requiring modification or fine-tuning of the underlying MLLM. Experiments on Video-MME, MLVU, and LongVideoBench show consistent accuracy gains with competitive computational cost, particularly for medium and long video regimes.

Key takeaway

For Machine Learning Engineers or AI Scientists working with Multimodal Large Language Models on video QA, especially with long-form content and constrained token budgets, you should consider integrating ReQuest. This method offers a proven way to achieve consistent accuracy gains and improve efficiency without the need to fine-tune your existing MLLMs. Evaluate its plug-and-play capability to enhance performance on medium and long video datasets.

Key insights

ReQuest improves long-form video QA for MLLMs via uncertainty-driven, question-adaptive keyframe selection.

Principles

• Uncertainty-driven selection optimizes MLLM video processing.
• Question-aware frame selection enhances relevance for long videos.
• Adaptive non-maximum suppression improves temporal diversity.

Method

ReQuest integrates a question-aware selector, Re-thinking Routing (triggers inference on uncertainty), and uncertainty-guided adaptive non-maximum suppression (selects diverse frames, adjusts spacing).

In practice

• Apply ReQuest as a plug-and-play method.
• Improve long-video QA without MLLM fine-tuning.

Topics

• Multimodal Large Language Models
• Video QA
• Keyframe Selection
• Long-form Video
• Uncertainty-driven AI
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• Q-Fold: Query-Aware Focus-Context Spatio-Temporal Folding for Long Video Understanding — Q-Fold improves long-video understanding by creating query-guided Focus-Context representations from temporal segments, balancing fidelity and coverage.
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs — The human-view perspective unifies MLLM video understanding through watching, remembering, and reasoning capabilities.
• Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding — Long video understanding agents should actively seek query-relevant evidence through iterative, goal-directed observation.
• How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding — Calibrating LLM first-token uncertainty with label priors enables efficient, adaptive prompt optimization for classification.
• TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Living — TimeProVe efficiently reasons in long videos by proposing hypotheses with lightweight LLMs and verifying them with targeted VLM calls.
• EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering — EASE-TTT improves long-context QA by aligning test-time attention adaptation with retrieved evidence chunks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.