Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

2026-06-30 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Delta-JEPA introduces an end-to-end reconstruction-free world model designed to overcome the challenge of action-insensitive latent representations in planning. This model augments latent forward prediction with a Latent Difference Action Decoder (LDAD), which reconstructs the executed action directly from the latent displacement between consecutive observations. This displacement-level supervision regularizes transition geometry, ensuring adjacent embeddings retain action information and different actions induce distinguishable latent changes crucial for rollout-based planning. Delta-JEPA avoids pixel reconstruction and distribution-matching regularizers, relying solely on latent prediction and action reconstruction. Benchmarked across four visual continuous-control tasks, Delta-JEPA consistently improves planning performance over JEPA-based and other representation-learning world model baselines, demonstrating superior action-conditioned latent responses.

Key takeaway

For Machine Learning Engineers developing visual world models for planning, Delta-JEPA offers a robust approach to overcome action-insensitive representations. By supervising latent differences with a Latent Difference Action Decoder, you can achieve collapse-resistant and action-sensitive models without relying on pixel reconstruction. Consider integrating displacement-level action supervision into your world model architectures to improve planning performance in continuous-control environments.

Key insights

Supervising latent differences in world models prevents collapse and enhances action sensitivity for planning.

Principles

• Latent displacement supervision regularizes transition geometry.
• Adjacent embeddings must retain action information.
• Distinguishable latent changes aid rollout planning.

Method

Delta-JEPA augments latent forward prediction with a Latent Difference Action Decoder (LDAD) to reconstruct actions from latent displacement, avoiding pixel reconstruction and distribution-matching regularizers.

In practice

• Use LDAD for action reconstruction.
• Apply displacement-based action decoding.
• Improve planning in visual control tasks.

Topics

• World Models
• Latent Dynamics
• Action Decoding
• Joint-Embedding Predictive Architectures
• Continuous Control
• Reinforcement Learning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems — The WAV model uses latent space inference for implicit planning in VLA systems, improving long-horizon decision-making.
• A Close Look At World Model Recovery In Supervised Fine-Tuned LLM Planners — SFT enables LLMs to internally represent planning world models, with broader training data improving recovery.
• SPLICE: Latent Diffusion over JEPA Embeddings for Conformal Time-Series Inpainting — SPLICE offers reliable time-series imputation with uncertainty quantification for critical infrastructure like power grids.
• Beyond Tokens: How JEPA Is Quietly Teaching AI to Understand the World — JEPA models learn world understanding by predicting abstract data representations, not raw pixels or tokens.
• Sub-JEPA: a simple fix to LeCun group's LeWorldModel that consistently improves performance [P] — Relaxing global latent space constraints with subspace regularization improves world model performance.
• AdaJEPA: An Adaptive Latent World Model — Adaptive latent world models enhance planning robustness by continuously recalibrating during execution.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.