Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective

2026-04-16 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

This survey provides a problem-driven review of feed-forward 3D reconstruction, a paradigm that efficiently generates 3D representations from 2D inputs in a single forward pass, overcoming the limitations of slow per-scene optimization. It introduces a novel taxonomy focusing on model design strategies, independent of output formats like NeRF, 3DGS, or Pointmap. The taxonomy organizes research into five key areas: feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temporal-aware models for 4D reconstruction. The survey also reclassifies benchmarks into geometry-oriented and visual-oriented categories, discusses real-world applications in autonomous driving, robotics, and scene understanding, and outlines future directions including rigorous benchmarks, scalable representations, and deeper integration with generative and semantic models. This work aims to guide future research toward more robust and scalable 3D reconstruction systems.

Key takeaway

For research scientists developing 3D reconstruction systems, focusing on feed-forward architectures is crucial for achieving real-time performance and scalability. You should prioritize developing models that are robust to sparse inputs and capable of cross-scene generalization, potentially by integrating visual foundation models and exploring novel, inherently scalable 3D representations. Consider contributing to standardized benchmarks that rigorously evaluate both geometric accuracy and perceptual fidelity to advance the field transparently.

Key insights

Feed-forward 3D reconstruction offers efficient, generalizable scene modeling by directly mapping 2D inputs to 3D representations.

Principles

• Model design should be agnostic to output representation.
• Efficiency and generalization are paramount for practical deployment.
• Geometric priors enhance reconstruction from sparse inputs.

Method

Feed-forward models use an encoder-decoder architecture, mapping input images to 3D representations in a single pass, optimized via multi-scene training with geometric, photometric, and regularization losses.

In practice

• Use pre-trained 2D models for geometric priors.
• Employ hierarchical Gaussian generation for detail.
• Integrate diffusion models for visual realism.

Topics

• Feed-Forward 3D Reconstruction
• Neural Radiance Fields
• 3D Gaussian Splatting (3DGS)
• Pointmap Representation
• Geometry Awareness

Code references

• ziplab/Awesome-Feed-Forward-3D

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Robotics Engineer

Related on AIssential

• Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective — Feed-forward 3D reconstruction offers efficient, generalizable scene modeling by mapping 2D inputs directly to 3D.
• WildSplatter: Feed-forward 3D Gaussian Splatting with Appearance Control from Unconstrained Images — WildSplatter enables fast, pose-free 3DGS from unconstrained images by disentangling geometry and appearance via learned embeddings.
• GGPT: Geometry Grounded Point Transformer — GGPT refines dense 3D reconstructions by fusing feed-forward predictions with accurate sparse geometric guidance from an improved SfM pipeline.
• Hand-4DGS: Feed-Forward 3D Gaussian Splatting for 4D Hand Reconstruction from Egocentric Videos — Hand-4DGS is the first feed-forward 3D Gaussian Splatting framework for dynamic 4D hand reconstruction from egocentric videos.
• JointEdit3D: Feed-Forward 3D Scene Editing in a Unified Latent Space — JointEdit3D enables feed-forward 3D scene editing by unifying RGB-geometry reconstruction and generation in a single latent space.
• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation — IVGT implicitly models continuous 3D geometry from unposed multi-view images using a neural scene representation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.