DriveCtrl: Conditioned Sim-to-Real Driving Video Generation

2026-05-14 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Expert, quick

Summary

DriveCtrl is a novel depth-conditioned controllable sim-to-real video generation framework designed to synthesize realistic driving videos for autonomous driving systems. It addresses the significant domain gap between simulated and real-world driving data, which typically limits the utility of large-scale, fully annotated simulation data. Built on a pretrained video foundation model, DriveCtrl incorporates a structure-aware adapter that uses depth guidance to preserve scene layout and motion patterns from source simulations, ensuring temporal coherence. The framework also includes a scalable data generation pipeline that transforms simulator videos to match the visual style of a target real-world dataset, supporting structural depth, reference-dataset style, and text prompts as conditioning signals. DriveCtrl preserves frame-level annotations for downstream perception tasks and introduces a new evaluation metric, the Driving Video Realism Score (DVRS), to assess video realism. Experiments show DriveCtrl outperforms existing methods in realism, temporal quality, and perception task performance.

Key takeaway

For research scientists developing autonomous driving systems, DriveCtrl offers a robust solution to the sim-to-real domain gap. You should consider integrating depth-conditioned video generation and style transfer techniques to create more realistic training data from simulations, thereby improving the performance of downstream perception tasks. This approach can significantly reduce the need for costly real-world data collection.

Key insights

DriveCtrl bridges the sim-to-real gap for autonomous driving by generating realistic, temporally consistent, and annotation-preserving videos.

Principles

• Depth conditioning preserves scene structure.
• Style transfer enhances visual realism.
• Annotation preservation is critical for utility.

Method

DriveCtrl uses a structure-aware adapter on a pretrained video foundation model, guided by depth, reference style, and text prompts, to transform simulated driving videos into realistic footage while preserving annotations.

In practice

• Generate synthetic data with real-world visual styles.
• Preserve annotations during video style transfer.
• Evaluate driving video realism using DVRS.

Topics

• DriveCtrl
• Sim-to-Real Transfer
• Driving Video Generation
• Depth-Conditioned Generation
• Autonomous Driving

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos — RealityBridge uses multimodal controls and specialized training to enhance realism and temporal consistency in edited 3DGS driving simulations.
• CityGen: Structure-Guided City-Style Synthesis for Cross-City Autonomous Driving — CityGen enables zero-label cross-city adaptation for autonomous driving via HD-map-conditioned diffusion synthesis, improving robustness.
• PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation — PersonaDrive enables diverse, human-style driving agents in simulations by conditioning VLAs on retrieved style-instructed human demonstrations.
• CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL — CreFlow enhances embodied video generation by localizing RL updates with LTL-based compositional rewards and corrective reflow.
• A Generative Model for Closed-Loop Microsimulation of Signalized Intersections — Actor-centric generative model Enactor enables stable, realistic closed-loop microsimulation of signalized intersections by learning from its own predictions.
• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation — WorldComposer generates diverse, high-fidelity simulation environments from real-world panoramas for robust robot learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.