DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

2026-06-10 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

DIRECT is a novel routing framework designed for embodied planners that addresses the inefficiencies of uniformly scaling test-time compute in Vision-Language Models (VLMs). While increasing compute can improve VLM capabilities for embodied agents, it often leads to higher latency, token usage, and FLOPs, with inconsistent success gains. DIRECT mitigates this by using multimodal scene context to dynamically allocate compute per prompt, optimizing the success-cost Pareto frontier compared to fixed model selection. Experiments on VLABench and RoboMME, exploring chain-of-thought depth, model size, and memory history, reveal that compute scaling is not a uniform lever, with each axis offering distinct capability improvements. Validated on a physical Franka arm in a DROID setup for zero-shot manipulation and long-horizon chaining, DIRECT achieved success rates comparable to or better than a stronger model, while reducing average latency by up to 65%. This demonstrates that DIRECT enables frontier-level embodied planning more cost-effectively than naive scaling.

Key takeaway

For MLOps Engineers or Robotics Engineers deploying Vision-Language Models in embodied agents, you should reconsider static compute allocation strategies. Naively scaling test-time compute is wasteful and inefficient; instead, implement dynamic routing frameworks like DIRECT. This approach allows your systems to achieve frontier-level planning performance with up to 65% lower average latency, significantly reducing operational costs and expanding deployment possibilities for your robotic applications. Evaluate how different scaling axes impact your specific tasks.

Key insights

Dynamically allocating test-time compute based on context significantly improves embodied VLM planning efficiency and performance.

Principles

• Test-time compute scaling is not uniformly effective.
• Multimodal context enables efficient compute allocation.
• Different scaling axes yield distinct capability gains.

Method

DIRECT uses multimodal scene context to route compute per prompt, optimizing across chain-of-thought depth, model size, and memory history for embodied VLM planners.

In practice

• Implement dynamic compute routing for VLMs.
• Evaluate compute allocation across scaling axes.
• Apply DIRECT to robotic manipulation tasks.

Topics

• Embodied AI
• Vision-Language Models
• Dynamic Compute Allocation
• Robotic Manipulation
• DIRECT Framework
• Resource Optimization

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Robotics Engineer, MLOps Engineer

Related on AIssential

• Agentic Test-Time Scaling for WebAgents — Dynamic compute allocation based on decision uncertainty improves multi-step agent performance and efficiency.
• Visually-grounded Humanoid Agents — A two-layer paradigm enables digital human agents to perceive, reason, and act autonomously in novel 3D environments using visual input.
• A benchmarking framework for embodied neuromorphic agents — A new framework standardizes benchmarking for neuromorphic control of soft robots in dynamic, real-world environments.
• this new benchmark is next-level insane — Real-world AI testing reveals critical limitations in autonomy, consistency, and long-term coherence not apparent in digital benchmarks.
• ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints — Embodied agents require dynamic affordance reasoning to adapt to unspecified, changing real-world object constraints.
• RoboLayout: Differentiable 3D Scene Generation for Embodied Agents — RoboLayout generates agent-navigable 3D scenes by integrating explicit reachability constraints and local refinement into differentiable optimization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.