Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

ToolMerge is a novel keyframe retrieval method for long-video question answering (QA) that employs an LLM-based planner to decompose queries into tool calls and merge their per-tool rankings using boolean operators. Developed by researchers at the University of Illinois at Urbana-Champaign, ToolMerge utilizes lightweight visual tools like SigLIP-2 for scene-level evidence and T-REN for localized entity search, alongside OCR. To directly evaluate retrieval, the team constructed Molmo-2 Moments (M2M), a new benchmark where each question is anchored to a specific time interval. ToolMerge demonstrates competitive performance against prior keyframe selectors across QA, question retrieval, and caption retrieval tasks, notably outperforming other methods by 5% on caption retrieval.

Key takeaway

For machine learning engineers developing long-video QA systems, ToolMerge offers a robust approach to keyframe selection. You should consider implementing a planner-based decomposition strategy with specialized visual tools and rank-based merging. This method, especially when combined with reinforcement learning for planner improvement, can enhance retrieval accuracy and reduce inference costs compared to dense captioning, making your systems more efficient and verifiable.

Key insights

Decomposing video queries into tool calls and merging their ranked outputs improves keyframe retrieval.

Principles

• LLM planners can orchestrate lightweight visual tools.
• Boolean operators effectively merge per-tool frame rankings.
• Retrieval benchmarks need questions anchored to specific time intervals.

Method

An LLM planner decomposes queries into SigLIP-2, T-REN, and OCR calls, specifying boolean merging. Tools score frames, ranks are combined (AND=max rank, OR=min rank), and top-k frames are selected with greedy NMS.

In practice

• Use SigLIP-2 for scene-level queries.
• Employ T-REN for specific object/entity detection.
• Integrate OCR for on-screen text evidence.

Topics

• Keyframe Retrieval
• Long-Video QA
• Large Language Models
• Visual Tools
• Molmo-2 Moments
• Reinforcement Learning

Code references

• michalsr/ToolMerge
• huggingface/trl

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Q-Fold: Query-Aware Focus-Context Spatio-Temporal Folding for Long Video Understanding — Q-Fold improves long-video understanding by creating query-guided Focus-Context representations from temporal segments, balancing fidelity and coverage.
• Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language — SpotVMR efficiently trims query-relevant video clips for VMR, overcoming fixed-length downsampling issues and improving retrieval performance.
• MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism — MemDreamer decouples perception and reasoning for long video understanding using hierarchical graph memory and agentic retrieval.
• VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding — Integrating extrinsic query relevance with intrinsic video structure enhances long video understanding for MLLMs.
• Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly — Current LVLMs significantly struggle with fine-grained spatio-temporal reasoning and object interaction in complex video scenarios.
• GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval — Decoupling feature representation from metric optimization using gradient-isolated RL resolves VMR's optimization bottleneck.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.