D4RT: Teaching AI to see the world in four dimensions

2026-01-22 · Source: Google DeepMind News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, medium

Summary

D4RT (Dynamic 4D Reconstruction and Tracking) is a new unified AI model introduced on January 22, 2026, designed for 4D scene reconstruction and tracking across space and time from 2D video input. It addresses the challenge of understanding dynamic scenes by tracking every pixel's movement through three spatial dimensions and the fourth dimension of time, while disentangling camera motion and maintaining coherence during occlusions. D4RT utilizes a simplified encoder-decoder Transformer architecture with a novel query mechanism, making it up to 300x more efficient than previous methods. This efficiency allows for real-time applications and enables tasks such as point tracking, point cloud reconstruction, and camera pose estimation. The model demonstrates superior fidelity on benchmarks like MPI Sintel and achieves top-tier performance on Aria Digital Twin and RE10k datasets.

Key takeaway

For AI scientists and robotics engineers developing spatial computing applications, D4RT offers a significant leap in real-time 4D scene understanding. Its 18x to 300x efficiency improvement over prior methods, coupled with high accuracy, means you can deploy robust 4D perception in systems like AR glasses or autonomous robots without compromising performance. Consider integrating D4RT for dynamic environment navigation or low-latency scene geometry understanding.

Key insights

D4RT unifies 4D scene reconstruction and tracking into a single, efficient AI model using a query-based Transformer architecture.

Principles

• Unify dynamic scene reconstruction into one framework.
• Query-based processing enables efficiency and scalability.
• Disentangle object, camera, and static geometry motion.

Method

D4RT employs an encoder-decoder Transformer. The encoder compresses video into a geometry/motion representation, which a lightweight decoder queries to answer "Where is a given pixel in 3D space at an arbitrary time, from a chosen camera?" queries in parallel.

In practice

• Track 3D trajectories of points across time.
• Generate complete 3D scene structures from video.
• Recover camera trajectories from different viewpoints.

Topics

• D4RT
• 4D Scene Reconstruction
• Dynamic Object Tracking
• Transformer Architecture
• Real-time Perception

Best for: Machine Learning Engineer, AI Scientist, Research Scientist, AI Engineer, Computer Vision Engineer, Robotics Engineer

Related on AIssential

• How DeepMind’s New AI Predicts What It Cannot See — D4RT offers rapid, unified 4D scene reconstruction from video, outperforming prior multi-model approaches in speed and occlusion handling.
• Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs — 4D human-scene understanding is achievable using only wearable IMUs, bypassing visual perception limitations.
• RCGDet3D: Rethinking 4D Radar-Camera Fusion-based 3D Object Detection with Enhanced Radar Feature Encoding — Optimizing 4D radar feature extraction significantly boosts 3D object detection accuracy and speed, outperforming complex fusion methods.
• MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer — MoRe efficiently reconstructs dynamic 4D scenes from monocular video by disentangling motion from static structure.
• How Does AI Learn to See in 3D and Understand Space? — Bridging 2D AI predictions with 3D geometry via geometric fusion enables scalable, semantic 3D scene understanding.
• UFO-4D: Unposed Feedforward 4D Reconstruction from Two Images — UFO-4D reconstructs dense 4D representations from two unposed images using dynamic 3D Gaussian Splats.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Google DeepMind News.