Three ways AI is learning to understand the physical world

2026-03-20 · Source: VentureBeat · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, long

Summary

Large language models (LLMs) are encountering limitations in applications requiring physical world understanding, such as robotics and autonomous driving, due to their lack of grounding in physical causality. This constraint is driving significant investment into "world models," with AMI Labs raising $1.03 billion and World Labs securing $1 billion in seed funding. World models act as internal simulators, allowing AI systems to test hypotheses before physical action. Three distinct architectural approaches are emerging: Joint Embedding Predictive Architecture (JEPA) for real-time, efficient latent representation learning; Gaussian splats for generating complete, interactive 3D spatial environments; and end-to-end generative models like Google DeepMind's Genie 3 and Nvidia's Cosmos, which continuously generate scenes and physical dynamics on the fly. Hybrid architectures are also beginning to appear, combining strengths from different approaches.

Key takeaway

For AI Scientists developing systems for physical interaction, understanding the three world model architectures is crucial. Your choice between JEPA for real-time efficiency, Gaussian splats for spatial environment creation, or end-to-end generation for synthetic data and complex physics will dictate system capabilities. Consider hybrid approaches to combine strengths, ensuring your AI can reliably operate in dynamic, real-world scenarios.

Key insights

World models address LLM limitations in physical causality by providing AI with internal simulation capabilities.

Principles

• AI needs physical grounding beyond abstract knowledge.
• Efficiency is key for real-time physical AI applications.
• Spatial intelligence is critical for real-world interaction.

Method

World models employ JEPA for latent feature learning, Gaussian splats for 3D environment generation, or end-to-end generative models for continuous scene and physics simulation.

In practice

• Use JEPA for real-time robotics and autonomous driving.
• Employ Gaussian splats for interactive 3D environment creation.
• Utilize end-to-end models for synthetic data generation.

Topics

• World Models
• Joint Embedding Predictive Architecture
• Gaussian Splats
• End-to-End Generative Models
• Robotics

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, AI Researcher, Machine Learning Engineer, AI Architect

Related on AIssential

• The Sequence Knowledge #804: The Dreamer Trilogy: Inside Some of the Most Influential Papers in AI World Models — World Models address Model-Free RL's limitations in 3D environments by enabling agents to predict future states.
• What if World Models and Quantum Computing complemented LLMs? — World models offer a path to AI with real-world understanding and causal reasoning beyond current LLM limitations.
• Turing Winner LeCun’s New ‘World Model’ AI Lab Raises $1B In Europe’s Largest Seed Round Ever — World models aim to enable AI to genuinely understand and interact with continuous, high-dimensional physical reality.
• Physical AI: What It Is and What It Is Not — Physical AI closes the perception-action loop in the real world, unlike related concepts focused on prediction or representation.
• Beyond Tokens: How JEPA Is Quietly Teaching AI to Understand the World — JEPA models learn world understanding by predicting abstract data representations, not raw pixels or tokens.
• Yann LeCun Said LLMs Will Never Understand the World. His Lab Just Proved It | Front Page — Le World Model solves representation collapse, enabling efficient, faster world modeling for physical environments.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by VentureBeat.