CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

CLAP (Causal Language-Action Prediction) is a novel method designed for directly converting pretrained Vision-Language Models (VLMs) into Vision-Language-Action models (VLAs) with minimal architectural modifications. This approach aims to provide a clearer understanding of how VLM capabilities transfer to control tasks, addressing the challenge of output-distribution mismatch when VLMs predict bare numeric action sequences. CLAP resolves this by causally conditioning precise action-token prediction on a natural-language action description that prepends each numeric sequence, without altering the VLM's backbone. Through single-epoch fine-tuning, a 2B CLAP model achieved 90.8% on LIBERO, marking a 14.9 percentage point improvement over VLA-0, and demonstrated enhanced robustness on LIBERO-PRO against language, object, and spatial perturbations. CLAP will be released as an open-weight, multi-scale compact VLA family (0.8B, 2B, and 4B) from a single VLM lineage.

Key takeaway

For AI Scientists and Robotics Engineers evaluating VLM-to-VLA adaptation strategies, you should consider CLAP's language-action grounding approach. This method offers a transparent path to understanding capability transfer and achieves significant performance gains (90.8% on LIBERO) with minimal architectural changes. It enables efficient development of robust robotic control systems, particularly for tasks requiring resilience to language, object, and spatial perturbations.

Key insights

CLAP directly adapts VLMs to VLAs by grounding numeric actions in natural language, improving performance and transparency.

Principles

• Direct VLM-to-VLA adaptation is possible.
• Language-action grounding improves VLA performance.
• Minimal architectural changes preserve VLM capabilities.

Method

CLAP prepends numeric action sequences with natural-language descriptions, causally conditioning action-token prediction on this language-action plan without modifying the VLM backbone.

In practice

• Single-epoch fine-tuning for VLM-to-VLA.
• Develop robust VLAs for robotic control.
• Analyze VLM capability transfer across scales.

Topics

• Vision-Language Models
• Vision-Language-Action Models
• Language-Action Grounding
• Robotic Control
• Model Adaptation
• Open-weight Models

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• Vision Language Action (VLA) Models for Unmanned Aerial Robotics and Bimanual Manipulation: A Review — VLAs unify perception, language, and action, with bimanual and UAV coordination strategies being transferable.
• Robotic Policy Adaptation via Weight-Space Meta-Learning — WIZARD enables robotic VLA policy adaptation via weight-space meta-learning, generating LoRA parameters from video and language without fine-tuning.
• Let It Be Simple: One-Step Action Generation for Vision-Language-Action Models — VLA action generation's unique structure allows effective one-step prediction by biasing diffusion training towards high-noise states.
• Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs — Decoupling physical motor skill learning from language grounding enables VLA models to scale with less expert data.
• Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time — Adapting Vision-Language-Action models to new tasks can be achieved via demonstration retrieval, not costly per-task fine-tuning.
• Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models — DeepVision-VLA enhances robotic manipulation by integrating multi-level visual features and pruning irrelevant visual tokens.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.