DVSM: Decoder-only View Synthesis Model Done Right

2026-05-28 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

DVSM, a novel decoder-only view synthesis model, re-examines the common encoder-decoder architecture in Large View Synthesis Models (LVSMs). Through controlled experiments, DVSM demonstrates that a decoder-only design, which implicitly represents scenes as a KV-cache, outperforms encoder-decoder variants. This is achieved with fewer parameters and identical rendering complexity. Further analysis reveals that sharing weights between the color-input reconstruction network and the camera-only rendering network improves feature alignment, facilitating image synthesis. DVSM integrates foundation model priors and stage-wise patch sizing to enhance its efficiency-quality tradeoff, establishing a new state of the art for novel-view synthesis across multiple benchmarks, even surpassing per-scene-optimized 3DGS under dense input views.

Key takeaway

For computer vision engineers designing novel-view synthesis models, you should consider the DVSM's decoder-only architecture. This approach offers superior efficiency and quality compared to traditional encoder-decoder designs. Evaluate integrating shared weights and foundation model priors into your models to achieve state-of-the-art performance, potentially outperforming even 3DGS in dense input view scenarios.

Key insights

Decoder-only view synthesis models, like DVSM, can outperform encoder-decoder architectures with fewer parameters and aligned features.

Principles

• Decoder-only architectures can be more efficient for view synthesis.
• Sharing weights between networks aligns features for improved image synthesis.

Method

DVSM employs a decoder-only architecture representing scenes as a KV-cache, incorporating foundation model priors and stage-wise patch sizing for efficiency and quality.

In practice

• Consider decoder-only designs for novel-view synthesis tasks.
• Explore weight sharing in reconstruction and rendering networks.

Topics

• Novel View Synthesis
• Decoder-only Models
• DVSM
• Encoder-Decoder Architectures
• Foundation Models
• 3D Gaussian Splatting

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• Scaling View Synthesis Transformers — Encoder-decoder architectures can be compute-optimal for view synthesis transformers with proper design and training strategies.
• DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis — DF3DV-1K provides a large-scale, diverse dataset and benchmark for distractor-free novel view synthesis.
• Novel View Synthesis as Video Completion — Video diffusion models can be adapted for sparse novel view synthesis by treating it as low frame-rate video completion.
• 346 Blog Posts To Learn About Computer Vision — Computer Vision leverages AI to interpret visual data, enabling diverse applications from image generation to real-world object detection.
• Déjà View: Looping Transformers for Multi-View 3D Reconstruction — DéjàView uses a single looped transformer block recurrently for efficient multi-view 3D reconstruction, outperforming larger feed-forward models.
• Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson — Computer vision, though trailing language models, is approaching its "ChatGPT moment" through specialized, efficient, and edge-deployable models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.