Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, extended

Summary

Image2Sim is a generative neural simulator designed to overcome data limitations in embodied navigation by creating scalable, high-fidelity, and physically grounded interactive environments from posed RGB-D image sequences. It decouples 3D spatial anchoring from photorealistic observation synthesis, utilizing a feed-forward feature Gaussian model for scene construction and a Geometry-Aware One-Step Pixel Flow model for panoramic RGB-D rendering at approximately 40 FPS on an RTX 4090 GPU. This framework functions as an automated data engine, converting large video and image collections into nearly 20,000 interactive scenes and synthesizing over 10 million navigation training samples. Navigation models, such as Image2Nav, trained exclusively in these neural environments, demonstrate significant performance improvements on benchmarks like R2R-CE and RxR-CE, and transfer effectively to real-world zero-shot settings, improving path-following SR from 8/20 to 11/20.

Key takeaway

For AI Scientists and Machine Learning Engineers developing embodied navigation agents, Image2Sim offers a practical solution to the data scarcity problem. You should consider integrating neural simulation frameworks that convert existing visual data into scalable, physically grounded, and interactive training environments. This approach enables the generation of millions of diverse vision-language-action samples, significantly improving cross-simulator performance and real-world zero-shot generalization for your navigation models.

Key insights

Neural simulation from real-world captures can scale embodied navigation data and improve real-world transfer.

Principles

• Decouple 3D anchoring from visual synthesis.
• Combine explicit 3D grounding with generative completion.
• Scaling data yields consistent navigation gains.

Method

Image2Sim lifts RGB-D to 3D feature Gaussians, renders panoramic RGB-D via a geometry-aware pixel flow model, then generates collision-aware trajectories and VLM-annotated instructions.

In practice

• Convert video/image collections into interactive 3D scenes.
• Generate 10M+ vision-language-action samples.
• Train navigation policies for zero-shot real-world transfer.

Topics

• Embodied Navigation
• Neural Simulation
• 3D Gaussian Splatting
• Generative Models
• Vision-Language Navigation
• Sim-to-Real Transfer

Code references

• MrZihan/Image2Sim

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation — Digital Cousins generates diverse, high-fidelity simulations from real-world panoramas for robust robot learning.
• Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation — Grounding waypoints in executable trajectories resolves planning-control inconsistencies and improves reachability in Vision-Language Navigation.
• GEM: Generative Supervision Helps Embodied Intelligence — Integrating depth map generation into VLM pre-training significantly enhances embodied intelligence by bridging semantic and physical knowledge gaps.
• MagicSim: A Unified Infrastructure for Executable Embodied Interaction — MagicSim unifies embodied interaction simulation by integrating control, skills, and planning within a deterministic, batched runtime and shared MDP.
• SPEAR: A Simulator for Photorealistic Embodied AI Research — SPEAR offers a high-performance, highly programmable Unreal Engine simulator for embodied AI with novel ground truth data.
• MatterDoor: Sampling Zero-shot Spatio-semantic Priors using Generative Models — Generative models can sample spatio-semantic priors for robotic planning in partially observed 3D environments.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.