GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

GARDEN, an RGB-only framework developed by Zhejiang University and Ant Group, reformulates multi-view RGB reconstruction into physically-grounded scene factorization, outputting a structured hybrid scene representation. It leverages gravity as a universal physical prior, aligning reconstructions to a unified Gravity-View frame to resolve global orientation ambiguity. The pipeline recovers object-centric rigid meshes with accurate 6-DoF placement and removes duplicate object geometry from the background via conditional 3D point classification. This results in explicit rigid bodies decoupled from a clean background, enabling direct physics simulation while preserving visual realism. Experiments show GARDEN reduces processing time from 4330s (LiteReality) to 560s and improves object placement reliability, disentanglement quality, and rendering-simulation efficiency compared to retrieval-based baselines.

Key takeaway

For Robotics Engineers or ML Engineers developing embodied AI, GARDEN offers a robust way to create simulation-ready 3D environments. You can achieve high-fidelity, physically-consistent scene factorization directly from multi-view RGB images, eliminating the need for external CAD retrieval. This approach significantly reduces processing time and improves object interaction stability, enabling more realistic and efficient simulation development for your projects.

Key insights

Gravity provides a universal physical prior for robust, physically-grounded 3D scene reconstruction from RGB images.

Principles

• Gravity-View alignment resolves global orientation ambiguity.
• Decoupling objects from background prevents entanglement.
• Hybrid representation supports both simulation and rendering.

Method

Align multi-view RGB reconstruction to a Gravity-View frame, generate target objects with refined 6-DoF poses, then use conditional 3D point classification to remove object geometry from the background.

In practice

• Use multi-view foundation models for initial scene structure.
• Employ SAM-3D and FoundationPose for object reconstruction.
• Train point classifiers with artifact-oriented augmentations.

Topics

• 3D Reconstruction
• Scene Factorization
• Gravity Alignment
• Embodied AI
• Physics Simulation
• Multi-view RGB

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Towards Learning a Generalizable 3D Scene Representation from 2D Observations — A Generalizable Neural Radiance Field predicts 3D workspace occupancy from robot observations in a global frame.
• FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelity — FlowObject bridges generative priors and observational fidelity for robust 3D object reconstruction from sparse views.
• PRISM: Feed-Forward Single-Image 3D Reconstruction via Geometric Warp-Residual Modeling — PRISM enables fast, feed-forward single-image 3D reconstruction by correcting geometric warps with a learned residual.
• Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments — SPG-Layout generates physically plausible 3D indoor scenes in complex non-Manhattan environments using statistical priors and hierarchical object placement.
• DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interaction — DeSeG decouples semantic intent from geometric constraints to synthesize physically plausible human-scene interactions, preventing shortcut learning and penetrations.
• Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models — Rein3D integrates 3D Gaussian Splatting with panoramic video diffusion to restore and refine 3D indoor scenes from sparse inputs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.