Robot-Factored World Models via Robot Rendering

2026-07-27 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Robot-factored world models are proposed to enhance action-conditioned video prediction in robotics by externalizing robot-specific factors. This method first translates action commands into a "nominal trajectory" via the robot's controller and kinematics, which is then rendered as explicit robot geometry, including URDF mesh RGB and end-effector depth. This rendered geometry, combined with camera-aware static RGB/depth context, forms a consistent visual interface for the world model, allowing it to focus solely on scene response to robot motion. Evaluated on 41,642 DROID clips and 9,380 RoboCasa-GR1 clips, the rendered interface quantitatively outperforms vector-conditioned baselines like Ctrl-World and AdaLN state vector in PSNR, SSIM, and LPIPS. The approach also demonstrates generalization to unseen robot embodiments and facilitates generating robot manipulation videos from retargeted human demonstrations using datasets like DexYCB.

Key takeaway

For robotics engineers developing video world models, consider adopting a robot-factored visual interface. This approach, which renders nominal robot trajectories as explicit geometry and depth, significantly improves model performance and generalization across embodiments. You should integrate URDF-based rendering and end-effector depth to provide direct pixel-space action evidence, enabling your models to better predict scene interactions and adapt to new robot configurations or human demonstrations.

Key insights

Factoring robot-specific action realization and rendering into a visual interface improves robot world model performance and generalization.

Principles

• Decouple action realization from scene-response prediction.
• Represent actions as visible robot geometry, not control signals.
• Augment visual interfaces with depth for geometric cues.

Method

Actions are rolled into a nominal trajectory via robot kinematics, then rendered as URDF mesh RGB and end-effector depth. These, with static scene RGB/depth, condition a latent video diffusion model to predict scene response.

In practice

• Use nominal trajectories for action conditioning.
• Render robot URDF geometry for visual prompts.
• Incorporate end-effector and scene depth for contact cues.

Topics

• Robot World Models
• Action-Conditioned Video
• URDF Rendering
• Embodiment Generalization
• Nominal Trajectories
• Latent Diffusion Models

Code references

• aigc-apps/VideoX-Fun
• ModelTC/lightx2v
• robocasa/robocasa-gr1-tabletop-tasks

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Robot-Factored World Models via Robot Rendering — Robot-factored world models improve action-conditioned video prediction by explicitly rendering robot motion and geometry, separating robot mechanics from scene response.
• URDF Synthesis from RGB-D Sequences via Differentiable Joint Inference and Energy-Consistent Verification — KinemaForge jointly infers articulated object kinematics and geometry, ensuring energy consistency for accurate simulation.
• GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation — Distilling 4D geometry foundation model features into video world models ensures correspondence-consistent generation for robot manipulation.
• From Web Video to Real-World Robots — Web-scale video pre-training combined with minimal teleoperation data enables highly efficient robot foundation models.
• FlowWAM: Optical Flow as a Unified Action Representation for World Action Models — FlowWAM unifies action representation using optical flow within a dual-stream diffusion framework for world modeling.
• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models — Aligning visual and action representations in World Action Models improves robot manipulation performance and robustness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.