Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents

2026-06-29 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

Dynamo is a novel training-free framework designed to enhance vision-language models (VLMs) in visual reasoning tasks without requiring any weight updates or retraining. Operating on a small labeled training subset, Dynamo enables an agent to analyze its own successful and unsuccessful attempts, thereby evolving two distinct capabilities: reusable reasoning skills to address cognitive challenges and executable visual tools for perceptual limitations. Each generated tool is intrinsically linked with a specific skill dictating its invocation, and both skill and tool types are stored in a persistent library. The framework demonstrates significant improvements, boosting direct inference by an average of +5.6% accuracy across all 20 model-benchmark configurations, spanning four visual reasoning benchmarks and five VLM backbones. Furthermore, when provided with a pre-defined tool set, Dynamo effectively learns optimal tool invocation strategies, enhancing per-step tool choice on every tested backbone. It also closes 65-99% of the performance gap compared to task-specific reinforcement learning methods like VTool-R1 and DeepEyes, using considerably less compute, and can augment RL approaches.

Key takeaway

For ML Engineers and AI Scientists aiming to boost vision-language model performance on visual reasoning tasks, you should consider Dynamo as a training-free adaptation framework. It allows you to achieve an average of +5.6% accuracy improvement across diverse settings and efficiently close significant gaps against task-specific reinforcement learning, all without costly model retraining. Integrate this self-evolving skill-tool approach to adapt your frozen VLMs, reducing compute and development overhead while enhancing task-specific capabilities.

Key insights

Dynamo dynamically adapts frozen VLMs for visual reasoning by evolving reusable skills and visual tools through self-inspection.

Principles

• Self-inspection on attempts drives capability evolution.
• Skills and tools address cognitive and perceptual bottlenecks.
• Persistent libraries enable capability accumulation.

Method

An agent inspects its own correct and incorrect attempts on a small labeled subset to generate reusable reasoning skills for cognitive bottlenecks and executable visual tools for perceptual ones, pairing each tool with an invocation skill.

In practice

• Enhance VLM visual reasoning without model retraining.
• Adapt existing frozen VLMs to new tasks.
• Integrate with RL for combined performance gains.

Topics

• Vision-Language Models
• Agent Systems
• Skill Evolution
• Tool Learning
• Visual Reasoning
• Training-Free Adaptation

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning — MLLM agents improve 3D spatial reasoning by dynamically learning and applying scene-aware tool-use skills.
• Streaming Tokens and Tools: Multi-Turn Agentic Harness Support in NVIDIA Dynamo — Optimizing agentic inference requires precise prompt stability, correct reasoning/tool parsing, and efficient streaming.
• Paper Digest: ICLR 2026 Papers & Highlights — The ICLR 2026 highlights reveal a strong focus on enhancing LLM reasoning, multi-modal capabilities, and robust AI agent development.
• Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents — Dynamically synthesizing task-specific skills from retrieved trajectories improves LLM agent performance without parameter updates.
• PERL: Parameter Efficient Reasoning in CLIP Latent Space — Iterative latent reasoning offers a parameter-efficient adaptation mechanism for discriminative vision-language models like CLIP.
• The Hidden Evolution of Disguised Visual Context inside the VLM — Visual token integration paradigms in VLMs dictate internal representation evolution, affecting feature utilization and task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.