Waymo leverages Genie 3 to create a world model for self-driving cars

2026-02-06 · Source: AI - Ars Technica · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Intermediate, quick

Summary

Waymo and DeepMind have developed the Waymo World Model, an AI system designed to generate both 2D video and 3D lidar outputs from driving scenes. This model, which is not a direct port of Genie 3, uses a specialized post-training process to integrate critical depth information from lidar with visual details from cameras. It enables "driving action control" by allowing Waymo to alter vehicle routes in simulations, providing more realistic and consistent outcomes than previous methods. The model can also generate multimodal sensor data for existing dashcam videos, showing how a self-driving AI would perceive a situation. Furthermore, it can "mutate" real video conditions, such as changing weather, time of day, or adding objects like new signage or unexpected obstacles, to help Waymo vehicles adapt to diverse driving environments, including new markets like Boston and Washington, D.C.

Key takeaway

For AI Scientists developing self-driving systems, the Waymo World Model demonstrates a powerful approach to synthetic data generation. Your teams should explore multimodal simulation techniques that combine visual and depth data to enhance training realism and accelerate adaptation to diverse, challenging environments. This can significantly reduce reliance on purely real-world data collection for edge cases.

Key insights

Waymo's World Model generates multimodal driving simulations for enhanced self-driving AI training and environmental adaptation.

Principles

• Lidar adds critical depth to visual data.
• Simulations improve AI adaptation to varied conditions.

Method

The Waymo World Model uses a specialized post-training process to generate synchronized 2D video and 3D lidar outputs, enabling driving action control and mutation of real-world video conditions for AI training.

In practice

• Generate synthetic sensor data from dashcam videos.
• Simulate varied weather and time-of-day conditions.
• Introduce novel objects into driving scenarios.

Topics

• Waymo World Model
• Self-driving AI
• Lidar Technology
• AI Simulation
• DeepMind

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• The System Behind Self-Driving: Waymo’s Dmitri Dolgov — Full autonomy demands a comprehensive AI ecosystem, integrating advanced sensing, simulation, and specialized models for robust, scalable deployment.
• Google pairs its Genie world model with Street View to create explorable AI worlds based on real places — Google's Genie model uses Street View data to create interactive AI worlds for agent training and simulation.
• Waymo taps Google Deepmind's Genie 3 to simulate driving scenarios its cars have never seen — Waymo's new generative simulation model, based on Genie 3, creates rare driving scenarios for autonomous vehicle training.
• Xiaomi EV’s Next Self-Driving AI Explained — Autonomous driving models can achieve superior performance by shifting from linguistic reasoning to a latent spatial-temporal mathematical representation.
• OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation — A multi-agent LLM world model unifies diverse driving controls and multi-view geometry through a shared latent token interlingua for video generation.
• World Models in Plain English — World models aim to spatially compress and predict real-world environments, moving beyond text-based learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI - Ars Technica.