RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models

2026-04-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

RaTA-Tool is a new framework designed for open-world multimodal tool selection, addressing limitations in existing tool-use methods that are primarily text-only and struggle with unseen tools. This approach enables a Multimodal Large Language Model (MLLM) to convert a multimodal user query into a structured task description. It then retrieves the most suitable external tool by matching this description against semantically rich, machine-readable tool descriptions. This retrieval-based formulation allows for natural extensibility to new tools without requiring retraining. The framework also integrates a preference-based optimization stage using Direct Preference Optimization (DPO) to enhance alignment between task descriptions and tool selection. To facilitate further research, the authors introduce the first dataset for open-world multimodal tool use, which includes standardized tool descriptions derived from Hugging Face model cards. Experiments show RaTA-Tool significantly improves tool-selection performance, especially in open-world, multimodal contexts.

Key takeaway

For Research Scientists developing AI systems with tool-use capabilities, RaTA-Tool offers a robust method to overcome limitations of text-only and closed-world approaches. You should consider adopting a retrieval-based framework to enable your MLLMs to generalize to unseen tools and interpret complex multimodal instructions. This approach can significantly enhance the adaptability and real-world applicability of your AI agents, particularly when integrating diverse external resources.

Key insights

RaTA-Tool uses retrieval and MLLMs for open-world multimodal tool selection, improving generalization to unseen tools.

Principles

• Convert multimodal queries to structured task descriptions.
• Match task descriptions against semantic tool descriptions.
• Optimize alignment using preference-based learning.

Method

RaTA-Tool converts multimodal queries into structured task descriptions, then retrieves tools by matching these descriptions against machine-readable tool descriptions, enhanced by DPO for alignment.

In practice

• Utilize Hugging Face model cards for tool descriptions.
• Apply DPO for fine-tuning tool selection.
• Extend AI systems with external APIs via retrieval.

Topics

• RaTA-Tool
• Multimodal Large Language Models
• Tool Selection
• Retrieval-based AI
• Direct Preference Optimization

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning — LLM agents can achieve dynamic tool selection and superior generalization by optimizing tool preferences via Plackett–Luce ranking.
• META’s New SIRA: Superintelligence RAG — SIRA uses an LLM's parametric memory and BM25 for deterministic, single-shot information retrieval.
• MAviS: A Multimodal Conversational Assistant For Avian Species — Domain-adaptive multimodal LLMs are crucial for fine-grained understanding in specialized ecological applications like avian species.
• ChinAI #340: Year-end Reflections - Chinese Researchers on AI in Science — AI tools are transforming research workflows and societal interactions in China, highlighting linguistic adaptation and ethical challenges.
• GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models — GRIP uses LMM feedback to learn effective in-context example retrieval, outperforming similarity-based methods and transferring across models.
• A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding — Explicit reasoning plans improve multimodal art retrieval, enabling grounded, interpretable artwork understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.