Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

LaMem-VLA, a novel latent-memory-native framework, addresses the limitations of mainstream Vision-Language-Action (VLA) models in handling long-horizon, temporally dependent robotic manipulation tasks. Traditional VLAs often struggle due to their Markovian assumption and by keeping memory external to their latent embedding space. LaMem-VLA overcomes this by reconstructing historical experience into latent memory tokens, which are then directly interwoven with VLA reasoning. The framework integrates four coordinated components: a curator for organizing short-term and long-term memory vaults, a seeker for querying these vaults with multimodal cognition, a condenser for reconstructing retrieved evidence into compact latent memory tokens, and a weaver for injecting these tokens with current observations and instructions. This approach allows historical experience to directly guide VLA reasoning and action generation within a continuous latent space, demonstrating superior performance in extensive experiments on SimplerEnv and LIBERO. The paper was published on 2026-07-08.

Key takeaway

For Robotics Engineers developing Vision-Language-Action models that struggle with long-horizon, temporally dependent tasks, you should consider integrating historical experience directly into your VLA's latent embedding space. Adopting a framework like LaMem-VLA, which uses dual latent memory tokens, can significantly improve action generation by allowing memory to fluidly participate in reasoning. This approach offers a path to more robust and capable robotic manipulation systems.

Key insights

Integrating historical experience directly into a VLA model's native latent embedding space enhances long-horizon task performance.

Principles

• Memory integration within the VLA's latent space is crucial for fluid reasoning.
• Organize historical experience into complementary short-term and long-term memory vaults.
• Reconstruct retrieved evidence into compact latent memory tokens for direct VLA consumption.

Method

LaMem-VLA employs a curator to organize dual memory vaults, a seeker to query them, a condenser to reconstruct evidence into latent tokens, and a weaver to inject these tokens into the VLA's continuous embedding sequence.

In practice

• Apply latent memory integration to VLA models for complex, long-horizon robotic tasks.
• Utilize dual memory vaults to manage diverse historical experience effectively.

Topics

• Vision-Language-Action Models
• Robotic Manipulation
• Latent Memory
• Memory-Augmented VLAs
• SimplerEnv
• LIBERO

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• LARA: Latent Action Representation Alignment for Vision-Language-Action Models — Jointly optimizing Latent Action Models and Vision-Language-Action models via representation alignment improves robot action prediction.
• EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies — EventVLA uses foresight-driven sparse visual evidence memory to overcome memory bottlenecks in long-horizon robotic manipulation.
• Physical Intelligence Team Unveils MEM for Robots: A Multi-Scale Memory System Giving Gemma 3-4B VLAs 15-Minute Context for Complex Tasks — MEM provides VLA models with dual-track memory for long-horizon robotic tasks, enabling adaptation and real-time performance.
• LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies — LaWAM enables efficient, dynamics-aware robot control by using compact latent visual subgoals instead of computationally expensive pixel-level video predictions.
• Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time — Adapting Vision-Language-Action models to new tasks can be achieved via demonstration retrieval, not costly per-task fine-tuning.
• $μ$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models — Isolating minimal recurrence in VLA models significantly enhances performance in partially observable robotic manipulation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.