MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

2026-07-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

MM-ToolSandBox is a new benchmark and evaluation framework designed for visually grounded tool-calling agents. It features a stateful execution environment supporting over 500 tools across 16 application domains, enabling multi-image, multi-turn tasks that require agents to interpret visual inputs and execute tool calls while handling conversational elements like goal revisions and error corrections. The framework includes an automated scenario generation pipeline that creates diverse, visually grounded scenarios, resulting in 258 human-verified nominal scenarios and 50 variants for interactive UI applications. Evaluation of 12 state-of-the-art models, ranging from 4B open-weight to frontier proprietary systems, revealed that current models lack robust visual tool-calling capabilities, with even the best model achieving below a 50% success rate. Failure analysis indicates that visual precision, not just planning, is a significant bottleneck, accounting for 53% of failures due to incorrect information extraction from images. A "planning-to-precision crossover" suggests smaller models struggle with planning, while larger models falter in visual perception.

Key takeaway

For AI Engineers developing visual tool-calling agents, recognize that current models, even frontier systems, achieve below 50% success rates. Your development efforts should prioritize improving visual precision and information extraction from images, as this accounts for 53% of failures in capable models. If working with smaller models, focus on planning capabilities; for larger models, concentrate on enhancing visual perception to overcome the "planning-to-precision crossover" bottleneck.

Key insights

Current visual tool-calling agents struggle with visual precision more than planning, especially at larger scales.

Principles

• Visual precision is a primary bottleneck for capable visual tool-calling agents.
• Model scale shifts failure modes from planning to visual perception.

Method

MM-ToolSandBox uses information-flow-guided planning and multi-stage quality filtering to generate diverse, visually grounded scenarios for agent evaluation.

In practice

• Evaluate visual tool-calling agents using multi-image, multi-turn tasks.
• Focus research on visual precision for larger models, planning for smaller ones.

Topics

• Visual Tool-Calling Agents
• Multimodal AI Evaluation
• Benchmark Frameworks
• Computer Vision
• Agent Planning
• Visual Perception

Code references

• apple/ml-mmtoolsandbox

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents — LLM tool-selection failures stem from decision readout, not attention, despite models often attending to the correct tool.
• SceneActBench: Can Agents Act on the 3D Scenes They See? — SceneActBench evaluates VLM agents' action capabilities on complex 3D scenes, revealing current models struggle with consistent performance.
• Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains — Tool-augmented multimodal agents may learn tool-calling patterns more reliably than actual tool-contributed capabilities.
• Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning — Tool-augmented visual agents, trained with observation-relaxed policy optimization, significantly enhance spatial reasoning in VLMs.
• AI Agent Tool Design: What Works and What Doesn’t — Effective AI agent performance hinges on robust tool design, not just advanced model capabilities.
• LAI #131: A Tool Call Can Succeed and Still Be the Wrong Tool — Data transparency and rigorous data curation are crucial for trustworthy AI model development.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.