Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving

2026-07-17 · Source: Artificial Intelligence · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel "fast-slow" architecture addresses the high inference latency of large language models (LLMs) in end-to-end driving, which conflicts with vehicle control rates. It uses a frozen 7B vision-language backbone as a slow component, processing visual history and instructions at low frequency. This component exposes its per-layer key-value cache. A lightweight action expert acts as the fast component, attending to this cache and the current camera frame at each simulation tick. This expert then regresses waypoints. The expert is trained with randomized staleness for asynchronous execution. On LangAuto-Short routes in CARLA, this architecture achieves 94.0% route completion, up from 37.0% for a frame-skipping baseline. It also reduces red-light violations by a third. The system demonstrates zero-shot transferability to unseen towns, maintaining 84-94% completion. It reduces open-loop waypoint error by nearly a factor of four with a 32 ms per-tick cost on a single consumer GPU.

Key takeaway

For Robotics Engineers developing autonomous driving systems, if you face LLM inference latency, implement a fast-slow asynchronous architecture. This method enables powerful 7B vision-language models for scene reasoning while maintaining high-frequency control. You can achieve 94.0% route completion and reduce red-light violations. Train your fast action expert with randomized staleness to ensure robust performance despite cache lag.

Key insights

Asynchronous fast-slow inference decouples LLM reasoning from high-frequency control, improving driving performance and real-time responsiveness.

Principles

• Decouple high-latency reasoning from real-time control.
• Use a standing representation for scene context.
• Train fast systems with randomized staleness.

Method

A frozen 7B vision-language backbone provides a low-frequency scene representation via its key-value cache. A lightweight action expert then attends to this cache and current frames at high frequency to regress waypoints.

In practice

• Implement 7B LLMs for driving scene reasoning.
• Achieve 20 Hz control rates with LLM backbones.
• Improve route completion in CARLA simulations.

Topics

• Autonomous Driving
• Large Language Models
• Vision-Language Models
• Asynchronous Inference
• Real-time Control
• CARLA Simulation
• Robotics

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving — CLEAR achieves efficient, multi-modal autonomous driving by combining single-step generative planning with LLM-driven cognitive reasoning.
• Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference — Jetson-PI enables real-time VLA model control on edge devices by predicting future states and optimizing asynchronous inference.
• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies — Visual intermediate reasoning, not textual chain-of-thought, enables effective and low-latency vision-language-action policies.
• DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model — Decoupling temporal processing for vision, language, and action modalities significantly enhances VLA model performance and control robustness.
• FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation — Decoupling high-level semantic reasoning from low-latency action generation enables stable, efficient long-horizon aerial navigation.
• Long-term Traffic Simulation via Structured Autoregressive Modeling — LLM architectural biases and statistical priors enable robust long-term traffic simulation for autonomous driving.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.