PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generation

Source: arXiv:2607.07170 · Published 2026-07-08 · By Yi Yang, Myrna Castillo, Bodo Rosenhahn, Michael Ying Yang

TL;DR

This paper addresses the problem of online 3D scene graph generation, where a persistent 3D graph representation of a scene is incrementally constructed from streaming RGB-D input and 2D scene graph predictions. Existing online methods treat fusion as deterministic, ignoring multiple inherent sources of uncertainty arising from partial observations, 2D model soft predictions, and approximate 3D representations. The authors propose PUF, a plug-and-play, training-free uncertainty-aware fusion framework that jointly models semantic and spatial uncertainties during node association and evidence accumulation. PUF replaces binary accept/reject association gates with a probabilistic likelihood formulation combining semantic similarity and spatial overlap, propagates full softmax distributions via Dirichlet evidence accumulation for both nodes and edges, and optionally supplements edge evidence with a class-conditional relationship prior to complete sparsely observed relations. PUF is demonstrated on two backends (3D Gaussian and 3D voxel) and evaluated on the 3DSSG and ReplicaSSG benchmarks. Experimental results show consistent, large improvements over state-of-the-art online methods such as FROSS, with relationship recall gains of 18.1 points on 3DSSG and real-time latency of ~15 ms per frame. The method generalizes across different 3D representations without additional training, establishing uncertainty-aware fusion as an effective paradigm for real-time 3D scene understanding.

Key findings

• PUF-Gaussian improves relationship Recall@1 from 27.9% (FROSS) to 46.0% on 3DSSG benchmark, a +18.1 point gain at 15 ms/frame latency.
• PUF outperforms FROSS on ReplicaSSG benchmark without relationship prior, raising relationship Recall@1 from 22.5% to 25.3%.
• Dirichlet node representation alone improves relationship recall by 3.4 points over FROSS under Gaussian backend.
• Adding Dirichlet edge representation and probabilistic association further improves relationship recall to 33.9% (Gaussian).
• Incorporating the relationship prior on 3DSSG lifts relationship recall to 46.0%, indicating strong benefit for sparsely observed edges.
• The voxel backend variant shows consistent gains over FROSS, demonstrating representation-agnosticism.
• Semantic bandwidth σ_se=0.3 and birth density λ_birth=0.4 are optimal hyperparameters robust across datasets.
• Relationship prior especially benefits edges with few or no co-observations, completing missing relational evidence.

Threat model

n/a — The paper addresses online 3D scene graph generation in a robotics/perception context rather than a security or adversarial scenario. It assumes streaming RGB-D data and pretrained models producing probabilistic outputs but does not consider active adversarial manipulation or attacks on the perception pipeline.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model is not explicitly discussed as this is a perception and fusion technique, not a security paper. The method assumes streaming RGB-D frames with known camera poses, a pretrained 2D scene graph generation (SGG) model that outputs soft semantic and relational class probability distributions, and seeks to fuse these incrementally into a global 3D SG without full offline reconstruction.

2. Dataset & Preprocessing: Experiments use 3DSSG (1,482 indoor RGB-D scans with 20 object classes and 7 predicate types) and ReplicaSSG (18 photorealistic indoor scenes with 33 objects and 9 relations). 3DSSG training splits enable relationship prior computation; ReplicaSSG does not have training data, so no prior is used there. Objects detected below 0.7 confidence threshold are discarded; top 10 relation scores per frame are kept.

3. Architecture & Algorithm Components: PUF is a training-free, plug-and-play framework that wraps around any pretrained 2D SGG model producing softmax distributions. It incrementally constructs a global directed 3D graph G=(V,E) with nodes representing object entities with semantic label distributions (modeled as Dirichlet distributions) and edges encoding relationship labels also as Dirichlet distributions.

• 3D Representation: Objects from 2D detection bounding boxes are lifted to 3D either as Gaussian distributions (mean and covariance from backprojected depth points) or voxel sets (quantized 3D points within bounding boxes).

• Probabilistic Node Association: Association likelihood between an observation node and existing global nodes is computed as the product of spatial overlap (Bhattacharyya coefficient for Gaussian or voxel containment) and semantic similarity measured by exponentiated negative Jensen-Shannon divergence over Dirichlet mean class probabilities. A birth probability models the chance of spawning a new node.

• Evidence Accumulation: Dirichlet concentration parameters representing accumulated evidence for each class or relation are updated incrementally weighted by association likelihoods, redistributing semantic and relational evidence softly among candidate nodes rather than hard merging or rejecting.

• Relationship Prior: A class-conditional prior over edges is computed offline based on training data co-occurrence counts with Laplace smoothing, modulated by spatial distance and existence frequency, and combined with observed evidence via Bayesian posterior update to better estimate sparsely or never co-observed relationships.

4. Training Regime: No additional training is performed in PUF itself; the 2D SGG models are pretrained separately.

5. Evaluation Protocol: Recall, predicate recall, relationship recall, and mean recall metrics are computed on standard test splits for 3DSSG and ReplicaSSG. Latency is measured end-to-end on CPU/GPU hardware. Ablations examine the effect of Dirichlet node/edge modeling and the relationship prior independently. Hyperparameters are selected by grid search on the validation splits.

6. Reproducibility: Source code is publicly available. The paper uses off-the-shelf pretrained RT-DETR and EGTR 2D SGG models for fair comparison with FROSS. The datasets 3DSSG and ReplicaSSG are public or derived from public sources. Detailed parameter settings and implementation details are provided.

Example Walkthrough: For each RGB-D frame, 2D SGG generates soft label distributions for objects and relations. Objects are lifted into 3D Gaussians or voxels with uncertainty. Probabilistic association computes likelihoods of matching these observations to nodes in the global graph, soft evidence is accumulated into Dirichlet parameters across candidate matches rather than hard decisions. Relationship edges between nodes are updated similarly and complemented by a class-conditional prior for poorly observed edges. This process repeats in real time, incrementally refining the 3D scene graph.

Technical innovations

• Introduction of a training-free, plug-and-play uncertainty-aware fusion framework that propagates 2D soft class and relation distributions into 3D scene graph nodes and edges via Dirichlet evidence accumulation.
• Probabilistic node association replacing traditional deterministic hard gating with a joint spatial-semantic likelihood model that integrates 3D geometric uncertainty and semantic softmax similarity.
• Use of Dirichlet distributions to jointly model and incrementally accumulate uncertainty over both node semantics and edge relationships, enabling soft evidence redistribution across multiple plausible global graph candidates.
• Incorporation of a class-conditional, spatially modulated relationship prior derived from training data to complete and regularize relations between sparsely or never co-observed object pairs.

Datasets

• 3DSSG — 1,482 RGB-D indoor scans with 21,974 object instances and 16,324 relationships — public
• ReplicaSSG — 18 photorealistic indoor scenes with 33 object and 9 relation categories — public (Replica dataset extension)

Baselines vs proposed

• FROSS (3D Gaussian): Relationship Recall@1 = 27.9% vs PUF-Gaussian: 46.0% on 3DSSG
• FROSS (3D Gaussian): Latency = 13 ms vs PUF-Gaussian: 15 ms on 3DSSG
• FROSS (3D Gaussian): Relationship Recall@1 = 22.5% vs PUF-Gaussian: 25.3% on ReplicaSSG
• FROSS (3D Voxel): Recall@1 relationship = 23.7% vs PUF-Voxel: 40.3% on 3DSSG
• Ablation of Dirichlet Node modeling alone improves relation recall by 3.4 points vs FROSS (Gaussian backend)
• Adding Dirichlet Edge and probabilistic association improves relation recall to 33.9% (Gaussian)
• Adding relationship prior boosts relation recall further to 46.0% on 3DSSG

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.07170.

Fig 1: Our method is aware of 3 types of uncertainty which are overlooked in existing

Fig 2: Overview of our proposed PUF. Stage 1: An RGB-D frame is processed by

Fig 3: Relationship recall@1 under different 2D observation counts on 3DSSG test

Fig 4: Qualitative between our PUF and FROSS [12] on 3DSSG dataset with Gaussian

Fig 5: compares PUF’s qualitative outputs under the Gaussian and voxel

Fig 6: visualizes PUF’s predictions on two ReplicaSSG scenes (Office0 and

Limitations

• No explicit adversarial or attack evaluation; uncertainty modeling focuses on perceptual noise, not malicious manipulation.
• Relationship prior relies on training data statistics and is unavailable or less useful in datasets without training splits (e.g., ReplicaSSG).
• Evaluation mostly on indoor RGB-D datasets; generalization to outdoor or more diverse scenes untested.
• The method assumes accurate camera pose information; performance under pose noise or dynamic scenes not explored.
• Fusion depends on quality of initial 2D SGG model; intrinsic errors from that stage propagate into 3D graph.
• Computational overhead of probabilistic association grows with number of global nodes, potentially limiting scalability in very large scenes.

Open questions / follow-ons

• How does PUF perform under dynamic scene conditions where objects move or appear/disappear frequently?
• Can the probabilistic association and Dirichlet accumulation be extended or adapted for real-time large-scale outdoor scenes?
• What is the impact of camera pose uncertainty or SLAM errors on the fusion performance and uncertainty estimates?
• How might an end-to-end trainable approach incorporating PUF principles improve fusion in 3D scene graph learning?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners interested in bot detection through environmental context or 3D scene understanding, PUF introduces principled uncertainty modeling for online 3D scene graph fusion from partial and noisy 2D inputs. The probabilistic node association and Dirichlet evidence accumulation mechanisms demonstrate a robust way to aggregate uncertain observations incrementally without hard decisions, which could inspire more uncertainty-aware fusion strategies in other domains requiring real-time multi-modal aggregation under noise and partial observability. Although not directly applicable to CAPTCHA challenges, the methodology highlights how propagating and accumulating uncertainty through fusion steps can improve the reliability and interpretability of complex scene representations—concepts that could transfer to techniques verifying human presence or detecting bots via environmental reasoning.

Cite

@article{arxiv2607_07170,
  title={ PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generation },
  author={ Yi Yang and Myrna Castillo and Bodo Rosenhahn and Michael Ying Yang },
  journal={arXiv preprint arXiv:2607.07170},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.07170}
}

Read the full paper

• arXiv:2607.07170 (abstract)
• arXiv:2607.07170 (PDF)