SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects

2026-05-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, extended

Summary

SceneCode is a framework that generates physically interactable indoor scenes by compiling natural language prompts into executable, code-driven worlds, rather than static meshes. Developed by researchers from The Chinese University of Hong Kong, Shanghai Jiao Tong University, Shanghai AI Laboratory, Microsoft, and the University of Oxford, it addresses limitations in object-level controllability and on-demand asset production. The system uses a room-level agent to plan layouts and emit AssetRequests, which are then routed to one of five code-generation strategies to synthesize part-wise Blender Python programs. These programs are validated through an execution-guided repair-and-refine loop, compiled into simulation-ready SDF assets, and registered in a persistent scene-state registry. SceneCode improves prompt-faithful scene generation, produces assets with cleaner mesh structures, and includes simulator-loadable articulation metadata, outperforming baselines like SceneSmith, HSM, and LayoutVLM in semantic fidelity and physical usability.

Key takeaway

For Robotics Engineers or Embodied AI developers building simulation environments, SceneCode offers a significant advancement by enabling the programmatic generation of physically interactable and editable indoor scenes. You can now create custom articulated objects with specific attributes on demand, overcoming limitations of fixed asset libraries. This allows for more diverse and realistic training environments, where objects retain explicit part structure and simulation-ready articulation, directly supporting complex robot manipulation tasks and policy evaluation.

Key insights

SceneCode generates editable, interactable indoor scenes as executable code, not static meshes.

Principles

• Programmatic generation enables on-demand interactable assets.
• Execution-guided validation improves code reliability.
• Code representation couples visual, semantic, and simulation data.

Method

SceneCode converts natural language prompts into structured house layouts and AssetRequests via a planner–designer–critic loop. Requests are routed to VLM-based strategies, generating part-wise Blender Python programs validated by an execution-guided repair-and-refine loop, then compiled into simulation-ready SDF assets.

In practice

• Generate custom articulated objects with specific materials.
• Edit object parameters (e.g., leaf count) in Blender Python.
• Integrate generated assets directly into MuJoCo for robot interaction.

Topics

• Indoor Scene Synthesis
• Programmatic World Generation
• Blender Python
• Articulated Objects
• Embodied AI
• Robotic Manipulation
• Simulation Environments

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects — SceneCode generates editable, physically interactable indoor scenes from natural language using executable code.
• HomeWorld: A Unified Floorplan-to-Furnished Framework for Generating Controllable, Densely Interactive Whole-Home Scenes — HomeWorld unifies floorplan-to-furnished scene generation using a hierarchical AI framework, enhancing realism and interactivity for embodied AI.
• RoboLayout: Differentiable 3D Scene Generation for Embodied Agents — RoboLayout generates agent-navigable 3D scenes by integrating explicit reachability constraints and local refinement into differentiable optimization.
• The Sequence Knowledge #842: Everything You Need to Know About World Models — World models represent the shift from text-based AI to physical simulation, enabling AI to understand and operate within dynamic reality.
• Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation — Unifying embodied world modeling via language-conditioned video generation improves generalization and control across diverse robotic tasks.
• TAI #190: Genie 3 World Model Goes Public — Genie 3 enables real-time interactive world generation, advancing both game development prototyping and embodied AI research.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.