LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

The Large Sparse Reconstruction Model (LSRM) significantly enhances feed-forward 3D object reconstruction and inverse rendering by scaling transformer context windows. It adapts native sparse attention (NSA) to process 20x more object tokens and >2x more image tokens than previous methods. LSRM utilizes an efficient coarse-to-fine pipeline, a 3D-aware spatial routing mechanism that employs explicit geometric distances, and a custom block-aware sequence parallelism strategy for balanced GPU workloads. Evaluations on novel-view synthesis benchmarks demonstrate substantial gains, including >2.4 dB higher PSNR and >40% lower LPIPS on the GSO dataset. For inverse rendering, LSRM achieves an LPIPS that matches state-of-the-art dense-view optimization methods on StanfordORB.

Key takeaway

For Machine Learning Engineers developing high-fidelity 3D reconstruction or inverse rendering models, consider adopting LSRM's approach. Expanding transformer context windows with sparse attention and a coarse-to-fine pipeline significantly boosts texture and geometry detail. Implement 3D-aware block routing and block-aware sequence parallelism to manage computational demands and achieve state-of-the-art performance, even with fewer input views than typically required.

Key insights

Scaling transformer context windows markedly improves feed-forward 3D reconstruction and inverse rendering fidelity.

Principles

• Expand context windows to narrow fidelity gaps.
• Focus computation on informative regions.
• Use geometric distances for 2D-3D correspondences.

Method

A two-stage coarse-to-fine pipeline initializes with dense reconstruction, then refines with sparse reconstruction predicting high-resolution residuals using NSA, 3D-aware routing, and block-aware sequence parallelism.

In practice

• Adapt native sparse attention for 3D tasks.
• Implement coarse-to-fine pipelines for high-resolution output.
• Employ 3D-aware routing for sharper textures.

Topics

• 3D Reconstruction
• Inverse Rendering
• Sparse Attention
• Transformer Architectures
• Context Windows
• Novel View Synthesis
• LSRM

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• VGGT-$Ω$ — Feed-forward reconstruction models scale predictably with data and model size, offering geometry-aware features.
• Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups — A transformer-based inpainting method enhances real-time 3D streaming quality in sparse multi-camera AR/VR setups.
• Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective — Feed-forward 3D reconstruction offers efficient, generalizable scene modeling by directly mapping 2D inputs to 3D representations.
• MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer — MoRe efficiently reconstructs dynamic 4D scenes from monocular video by disentangling motion from static structure.
• Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering — Training-free geometric and temporal correspondence improves autoregressive generative rendering consistency during camera revisits.
• Scaling View Synthesis Transformers — Encoder-decoder architectures can be compute-optimal for view synthesis transformers with proper design and training strategies.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.