JointEdit3D: Feed-Forward 3D Scene Editing in a Unified Latent Space

2026-06-11 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

JointEdit3D is a novel framework designed for feed-forward 3D scene editing, addressing limitations of existing methods like high test-time costs and structural inconsistencies. This system leverages a unified RGB-geometry reconstruction-generation latent space to couple appearance synthesis and geometry prediction. JointEdit3D operates by performing asymmetric latent inpainting, observing a single edited RGB reference latent to generate additional RGB views and an edited geometry latent, all while maintaining source-scene anchoring. It incorporates a dedicated SceneAnchor Branch to inject structural information without direct copying and uses edit/background-aware losses for balanced fidelity and preservation. To facilitate evaluation, the authors introduce SceneEdit3D-15K, a dataset with 15,000 paired editing samples, and SceneEdit3D-Bench, a 100-sample benchmark. Experiments demonstrate JointEdit3D's superior edited-region quality and 3D structural completeness compared to prior baselines, alongside competitive background preservation.

Key takeaway

For Computer Vision Engineers developing 3D scene editing applications, JointEdit3D offers a significant advancement over traditional per-scene optimization. You should consider integrating this feed-forward, unified latent space approach to achieve faster test-time performance and superior 3D structural consistency in your edited scenes. Utilize the provided SceneEdit3D-15K dataset and SceneEdit3D-Bench for robust evaluation of your own or competing methods, ensuring high-quality results and efficient workflows.

Key insights

JointEdit3D enables feed-forward 3D scene editing by unifying RGB-geometry reconstruction and generation in a single latent space.

Principles

• Unify RGB-geometry in a single latent space for consistent 3D editing.
• Employ asymmetric latent inpainting from a single edited RGB view.
• Anchor edits to source scene structure without direct copying.

Method

Adapt a unified RGB-geometry latent space for feed-forward 3D scene editing. Perform asymmetric latent inpainting from a single edited RGB reference, generating remaining views and geometry latent with source-scene anchoring via a SceneAnchor Branch and specific losses.

In practice

• Utilize SceneEdit3D-15K and SceneEdit3D-Bench for 3D scene editing evaluation.
• Achieve higher edited-region quality and 3D structural completeness.

Topics

• 3D Scene Editing
• Latent Space Models
• Feed-Forward Networks
• RGB-Geometry Synthesis
• SceneEdit3D Dataset
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective — Feed-forward 3D reconstruction offers efficient, generalizable scene modeling by directly mapping 2D inputs to 3D representations.
• View-Consistent 3D Scene Editing via Dual-Path Structural Correspondense and Semantic Continuity — Achieving view-consistent 3D scene editing requires joint distribution modeling across viewpoints.
• Native3D: End-to-End 3D Scene Generation via Unified Mesh-Texture Modeling and Semantic Alignment — Native3D generates 3D scenes end-to-end by unifying mesh-texture modeling and semantic alignment, avoiding 2D conversion issues.
• EditSSC: Toward Editable Semantic Occupancy Scenes with Unconditional Diffusion Models — EditSSC repurposes 2D latent diffusion for 3D semantic scene generation, enabling training-free editing capabilities.
• GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers — Jointly solving 3D geometry and relighting from a single image improves physical consistency and performance.
• Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion — Real2SAM2Real uses editable 3D caches from lifting models to provide robust geometric scaffolds for Video Diffusion Models, improving control and consistency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.