Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

A novel panoptic segmentation pipeline extends large view synthesis models, such as Less3Depend, beyond appearance rendering to 3D scene understanding. The research demonstrates that these models, trained solely for RGB image reconstruction, learn implicit cross-view spatial correspondence that effectively propagates non-photorealistic signals like binary-encoded panoptic labels to novel views. This method reuses a frozen view synthesis model, eliminating the need for explicit 3D reconstruction or segmentation-specific training. On the ScanNet dataset, the approach achieves 33.56 PSNR and 0.5949 mIoU, surpassing Gaussian-based methods like SIU3R (25.88 PSNR, 0.5894 mIoU) in rendering quality by over 7 dB. It also transfers labels across datasets like Replica without fine-tuning, outperforming SIU3R significantly (23.52 vs. 14.05 dB PSNR, 0.394 vs. 0.186 PQ, 0.454 vs. 0.171 mIoU). The decoupled design preserves rendering fidelity and shows robust performance even in low-overlap scenarios.

Key takeaway

For Machine Learning Engineers developing 3D scene understanding systems, you should consider utilizing frozen large view synthesis models for novel-view panoptic segmentation. This approach allows you to preserve high rendering quality (e.g., 33.56 dB PSNR) while achieving competitive segmentation (0.5949 mIoU) without explicit 3D reconstruction or segmentation-specific training. Your pipeline can become more modular, enabling independent upgrades of rendering and segmentation components and improving cross-dataset transferability.

Key insights

Large view synthesis models implicitly learn geometric correspondence, enabling panoptic label propagation without explicit 3D reconstruction or segmentation training.

Principles

• NVS models learn input-agnostic geometric correspondence.
• View-independent signals transfer via frozen NVS models.
• Decoupling rendering and segmentation preserves quality.

Method

Segment input views with a shared query decoder, encode panoptic labels into 3-bit binary channels, then propagate through a frozen RGB-pretrained view synthesis model, and finally decode the output into a panoptic map.

In practice

• Encode instance IDs using 3-bit binary channels.
• Use multi-pass decoding for scenes with >8 instances.
• Swap source-view segmenters without NVS model retraining.

Topics

• Large View Synthesis Models
• Panoptic Segmentation
• 3D Scene Understanding
• Novel View Synthesis
• Implicit Correspondence
• ScanNet
• Less3Depend

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation — Large view synthesis models' learned spatial correspondence propagates non-photorealistic labels, enabling multi-view panoptic segmentation without 3D reconstruction.
• Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling — Decoupling semantic and spatial representations in NVS transformers improves rendering fidelity by preventing spatial bias.
• Harnessing Generative Image Models for Training-Free Primitive Shape Abstraction — Training-free 3D shape abstraction leverages generative image models for semantic part segmentation and classical optimization for primitive fitting.
• NeoMap: Training-free Novel-View Synthesis from Single Images and Videos — NeoMap finds high-fidelity novel view solutions by optimizing initial noise through convergent manifold alternating projection within pre-trained video models' data manifold.
• DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis — DF3DV-1K provides a large-scale, diverse dataset and benchmark for distractor-free novel view synthesis.
• PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting — Panoramic scene completion with layout-anchored attention steering significantly improves sparse-view 3D reconstruction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.