APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

APIVOT is a VLM-based planner designed for long-horizon robot planning, adaptively interleaving language and visual thoughts. It addresses the challenge of jointly reasoning over semantic task structure and geometric feasibility. Trained with a three-stage supervised fine-tuning curriculum, APIVOT learns to use language for semantic reasoning and visual thoughts as imagined future states for internal geometric verification. Evaluated on long-horizon kitchen tasks in the KitchenWorlds simulator, APIVOT achieved an average task success rate of 0.419, an 8.1 percentage point improvement over the top-performing VLM baseline, Gemini-ER-1.5. It demonstrated significant gains in spatially constrained settings and reduced token usage by 39% compared to always-image reasoning, showcasing efficient, meaningful modality selection.

Key takeaway

For Robotics Engineers developing long-horizon planning systems, consider integrating adaptive vision-language reasoning to overcome geometric feasibility challenges. APIVOT's approach of interleaving visual thoughts for spatial verification, especially in constrained environments, significantly improves task success and reasoning efficiency. You should explore structured fine-tuning curricula to teach VLMs to selectively apply visual reasoning, reducing computational cost while maintaining high performance in complex manipulation tasks.

Key insights

Adaptive interleaving of vision and language thoughts significantly improves long-horizon robot planning success and efficiency.

Principles

• Geometric feasibility requires visual reasoning.
• Language excels at semantic task decomposition.
• Adaptive modality selection boosts efficiency.

Method

APIVOT uses a three-stage SFT curriculum: 1) comprehend ground-truth visual thoughts, 2) generate visual thoughts, and 3) adaptively select modalities based on geometric heuristics. Visual thoughts are aligned via cosine similarity loss.

In practice

• Train VLMs with visual thoughts for geometric tasks.
• Implement a curriculum for complex VLM capabilities.
• Use occupancy ratio to measure geometric complexity.

Topics

• Robot Planning
• Vision-Language Models
• Multimodal Reasoning
• Adaptive Planning
• Geometric Feasibility
• Supervised Fine-tuning
• KitchenWorlds Simulator

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts — APIVOT adaptively interleaves language for semantic reasoning and visual thoughts for geometric feasibility in long-horizon robot planning.
• A better method for planning complex visual tasks — A dual-VLM framework combines visual understanding with formal planning for robust, generalizable long-term task execution.
• Continuous Reasoning for Vision-Language-Action — Continuous reasoning, shareable and verifiable, bridges the granularity gap between language and continuous robot control.
• PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving — PixelPilot's decoupled 2D planning and 3D lifting paradigm enhances VLA scalability and visual reasoning for autonomous driving.
• GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning — GeneralVLA-2 enhances robot planning through geometry-aware 3D reconstruction and a governed, quality-controlled long-term memory system.
• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies — Visual intermediate reasoning, not textual chain-of-thought, enables effective and low-latency vision-language-action policies.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.