PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generation

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

PUF is a Plug-and-play, Uncertainty-aware, and training-free Fusion framework designed for online 3D scene graph generation. This framework incrementally builds a persistent, structured 3D scene representation by fusing 2D observations, addressing three overlooked uncertainty sources: observation, 2D model, and 3D representation. PUF reformulates scene graph node association as a probabilistic likelihood based on semantic and spatial factors, replacing deterministic accept/reject gates. It employs Dirichlet evidence accumulation to distribute class and relationship evidence across plausible candidates, proportional to their association likelihood. An optional class-conditional prior completes edges for sparsely or unobserved object pairs. The framework demonstrates consistent improvements when instantiated with both 3D Gaussian and 3D voxel backends, proving its generalizability. Experiments on the 3DSSG and ReplicaSSG benchmarks show PUF substantially outperforms existing methods while maintaining real-time latency. Source code is available at https://github.com/yyyyangyi/PUF.

Key takeaway

For Computer Vision Engineers developing online 3D scene understanding systems, you should integrate uncertainty-aware fusion to enhance robustness. By reformulating node association probabilistically and distributing evidence with Dirichlet accumulation, your systems can achieve substantial performance gains. Consider PUF's plug-and-play framework to address observation, 2D model, and 3D representation uncertainties, ensuring more accurate and persistent scene graph generation. This approach maintains real-time latency while outperforming deterministic methods.

Key insights

Uncertainty-aware fusion is a principled paradigm for robust online 3D scene graph generation.

Principles

• Reformulate node association as probabilistic likelihood.
• Distribute evidence using Dirichlet accumulation.
• Account for observation, model, and representation uncertainty.

Method

PUF reformulates node association probabilistically, uses Dirichlet evidence accumulation for class/relationship evidence, and applies an optional class-conditional prior for sparse edges.

In practice

• Apply probabilistic association in 3D scene graphs.
• Integrate uncertainty handling into fusion pipelines.
• Use Dirichlet accumulation for evidence distribution.

Topics

• 3D Scene Graph Generation
• Uncertainty Quantification
• Probabilistic Fusion
• Computer Vision
• Real-time Systems
• Dirichlet Evidence Accumulation

Code references

• yyyyangyi/PUF

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• Bayesian 3D Steerable CNNs: Enabling Equivariance and Uncertainty Quantification Simultaneously — Bayesian Steerable-CNNs unify SE(3)-equivariance with uncertainty quantification via stochastic kernels and variational inference.
• Breaking the Communication-Accuracy Trade-off: A Sparsified Information Diffusion Framework for Multi-Agent Collaborative Perception — EDC-CIF balances multi-agent tracking accuracy, response speed, and communication efficiency by optimizing local and global filtering.
• An Information Theoretic Framework for Graph Novelty Generation via Latent Mixture Modeling — Generate novel graphs by modeling latent distributions and applying information-theoretic novelty and reliability conditions.
• Noise2Params: Unification and Parameter Determination from Noise via a Probabilistic Event Camera Model — A unified probabilistic model for event cameras enables accurate parameter determination from static scene noise.
• Uncertainty-Aware Pedestrian Attribute Recognition via Evidential Deep Learning — UAPAR uses Evidential Deep Learning to quantify prediction uncertainty in pedestrian attribute recognition, enhancing robustness.
• Architecture-Adaptive Uncertainty Fusion for Deepfake Detection — Correlation-Optimized Fusion (COF) adaptively combines multiple uncertainty sources to predict deepfake detection errors, improving cross-domain reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.