From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation

2026-01-25 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

WorldComposer is a generative framework that creates high-fidelity, interactive simulation environments for robot learning directly from real-world panoramas. It establishes a real-to-sim mapping to reconstruct "Digital Twins" and then synthesizes diverse "Digital Cousins" through semantic and geometric editing, significantly expanding data diversity. The framework supports multi-room stitching for large-scale navigation environments and integrates high-quality physics engines with a comprehensive asset library for rigid, articulated, and deformable objects. Experiments demonstrate a strong sim-to-real correlation (Pearson r=0.91) and show that policies trained with WorldComposer's augmented data achieve significantly better generalization to unseen scenes and objects, with success rates increasing from 10% to 85% with more generated data.

Key takeaway

For research scientists developing generalizable robot policies, WorldComposer offers a robust platform to overcome real-world data scarcity. You should consider integrating this framework to generate diverse "Digital Cousins" for training, as it significantly boosts generalization to unseen scenarios and provides a reliable testbed for evaluating policy performance before real-world deployment.

Key insights

WorldComposer generates diverse, high-fidelity simulation environments from real-world panoramas for robust robot learning.

Principles

• Diverse data improves robot policy generalization.
• Simulation fidelity correlates with real-world performance.
• Generative models bridge real-to-sim gaps efficiently.

Method

WorldComposer converts panoramas to editable 3D Gaussian Splatting and collision meshes, stitches multiple rooms, and populates scenes with physics-enabled assets, generating diverse "Digital Cousins" via prompt-driven editing.

In practice

• Use prompt-driven editing for scene variations.
• Integrate 3DGS with physics engines for interactive simulations.
• Employ multi-room stitching for large-scale navigation tasks.

Topics

• Generative Simulation
• Digital Cousins
• Robot Learning
• Sim-to-Real Transfer
• 3D Gaussian Splatting

Code references

• isaac-sim/IsaacSim

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation — Digital Cousins generates diverse, high-fidelity simulations from real-world panoramas for robust robot learning.
• Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Network — Factored Diffusion Policies use a single diffusion network with additive score decomposition for compositionally generalized robot control.
• A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies — Co-training effectiveness in robotics stems from structured representation alignment and importance reweighting.
• RoboDream: Compositional World Models for Scalable Robot Data Synthesis — A generalizable embodiment-centric world model synthesizes photorealistic robot demonstrations by decoupling motion from environment synthesis.
• Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation — Unifying embodied world modeling via language-conditioned video generation improves generalization and control across diverse robotic tasks.
• Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation — Decoupling video generation from action prediction in WAMs, using a Mixture-of-Transformers and asymmetric attention, improves efficiency and generalization for autonomous navigation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.