Déjà View: Looping Transformers for Multi-View 3D Reconstruction

2026-05-28 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

DéjàView, a novel 3D reconstruction model, applies a single looped transformer block recurrently for K refinement steps, challenging the trend of increasing model capacity in computer vision. It posits that traditional feed-forward transformer depth often inefficiently simulates iteration, which DéjàView makes explicit in its architecture. Trained once, DéjàView exposes K as an inference-time compute knob, matching or outperforming substantially larger feed-forward baselines across five reconstruction benchmarks, including indoor, outdoor, object-centric, and driving scenes. It achieves this while using a fraction of their parameters and comparable or lower compute, suggesting explicit iteration provides a stronger inductive bias for multi-view 3D reconstruction.

Key takeaway

For Machine Learning Engineers optimizing multi-view 3D reconstruction models, you should consider recurrent architectures like DéjàView over deep feed-forward designs. This approach allows you to achieve comparable or superior performance with significantly fewer parameters and adjustable inference compute. Explore explicit iteration in your model designs to potentially gain a stronger inductive bias and improve efficiency for complex 3D tasks.

Key insights

DéjàView uses a single looped transformer block recurrently for efficient multi-view 3D reconstruction, outperforming larger feed-forward models.

Principles

• Model depth can inefficiently simulate iteration in feed-forward transformers.
• Explicit iteration provides a stronger inductive bias for multi-view 3D reconstruction.

Method

DéjàView applies a single transformer block recurrently to per-view features for K refinement steps, exposing K as an inference-time compute knob.

In practice

• Achieve competitive 3D reconstruction with fewer model parameters.
• Dynamically adjust inference compute via K refinement steps at runtime.

Topics

• Multi-View 3D Reconstruction
• Looping Transformers
• Model Efficiency
• Recurrent Neural Networks
• Computer Vision
• Inductive Bias

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer — MoRe efficiently reconstructs dynamic 4D scenes from monocular video by disentangling motion from static structure.
• DVSM: Decoder-only View Synthesis Model Done Right — Decoder-only view synthesis models, like DVSM, can outperform encoder-decoder architectures with fewer parameters and aligned features.
• VGGT-$Ω$ — Feed-forward reconstruction models scale predictably with data and model size, offering geometry-aware features.
• Stream3D: Sequential Multi-View 3D Generation via Evidential Memory — Stream3D uses an evidential memory and evidence score mechanism to enable consistent 3D generation from sequential multi-view inputs without retraining existing models.
• Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective — Feed-forward 3D reconstruction offers efficient, generalizable scene modeling by mapping 2D inputs directly to 3D.
• Scaling View Synthesis Transformers — Encoder-decoder architectures can be compute-optimal for view synthesis transformers with proper design and training strategies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.