Relation-Centric Open-Vocabulary 3D Gaussian Segmentation

Source: arXiv:2607.01140 · Published 2026-07-01 · By Eunsung Cha, Hyunjoon Lee, Jaesik Park

TL;DR

The paper addresses the challenge of open-vocabulary 3D Gaussian segmentation, which requires both rich language understanding for diverse text queries and precise geometric separation of Gaussian primitives at object boundaries. Prior approaches either rely on embedding language features directly into each Gaussian or optimize per-Gaussian instance features using costly per-scene 2D mask supervision. These methods suffer from noisy segmentations or inefficiency. This work introduces PairGS, a novel training-free framework that reframes segmentation as estimating pairwise relations between Gaussian nodes. By leveraging intrinsic signals such as view contribution weights and multi-view mask evidence, PairGS efficiently constructs a sparse relation graph without heavy optimization. This graph is clustered hierarchically using a novel TreeDBSCAN method, supporting multi-granular queries and clean instance boundaries even for thin or touching parts. PairGS achieves state-of-the-art segmentation accuracy on benchmarks like LERF-OVS and ScanNet while being significantly faster (up to 50×) than optimization-based instance-feature methods.

The key innovation is the relation-centric formulation that exploits 3D Gaussian rendering properties to explicitly measure affinity between Gaussian pairs. The two-stage graph construction first proposes sparse edge candidates via low-dimensional semantic and positional descriptors, then computes precise affinity only on those edges using robust co-occurrence metrics across views. The hierarchical clustering organizes these affinities into a meaningful cluster tree, enabling flexible, fine-grained segmentation. Extensive experiments demonstrate superior mIoU and accuracy compared to both language-based and instance-feature methods, as well as strong semantic segmentation and class-agnostic instance segmentation. Ablations show each component, such as positional scaling, directed graph connectivity, and view-aware affinity, contributes substantially.

Key findings

• PairGS achieves mean mIoU of 60.4% and mean accuracy of 68.2% on the LERF-OVS open-vocabulary object selection benchmark, outperforming LaGa [6] which scores 52.2% mIoU and 55.1% accuracy.
• PairGS-Fast runs in 0:58 (min:sec) on LERF, outperforming THGS [12] which takes 1:59 with lower mIoU (42.5%) and accuracy (63.9%).
• On ScanNet semantic segmentation, PairGS achieves up to 40.7% mIoU and 54.3% accuracy on 19 classes, beating prior methods including InstanceGaussian [29] with 39.5% mIoU and 53.9% accuracy.
• PairGS obtains 51.9% mIoU and 56.1% accuracy on class-agnostic instance segmentation in ScanNet, outperforming InstanceGaussian [29] with 49.9% mIoU and 53.9% accuracy.
• Hierarchical clustering via TreeDBSCAN avoids redundant and spurious clusters better than flat DBSCAN runs, enabling meaningful multi-granular segmentation trees.
• Positional scaling factor in node descriptors improves segmentation quality by balancing semantic and geometric cues (Tab. 4 ablations).
• Using directed k-NN graph rather than undirected reduces performance degradation, preserving local connectivity critical for clustering.
• Gaussian-aware pairwise affinity formulation leveraging view contribution weights and dual mask sampling (center- and max-based indices) improves discrimination near boundaries.

Threat model

n/a - The paper focuses on semantic segmentation of 3D Gaussian scenes using multi-view mask evidence and language queries without a security or attacker model.

Methodology — deep read

Threat Model & Assumptions: The setting is open-vocabulary 3D Gaussian segmentation, aiming to segment 3D scenes represented by Gaussian primitives given diverse natural language queries. The adversary is not explicitly modeled as this is not a security paper. The task assumes access to multi-view images, corresponding 2D masks from foundation models (e.g., SAM), and pretrained 3D Gaussian splatting (3DGS) models. The goal is to produce coherent semantic and instance segmentations without requiring per-scene optimization.

Data: Experiments use the LERF-OVS dataset for open-vocabulary object selection with annotated ground-truth masks, and the ScanNet RGB-D dataset for semantic and instance segmentation evaluation. The LERF-OVS dataset has multi-view images and 3D Gaussian reconstructions with mask annotations obtained from foundation models like SAM. ScanNet provides dense 3D reconstructions and semantic labels. The dataset splits and detailed preprocessing follow the baselines, using pretrained 3DGS models for initialization.

Architecture / Algorithm: PairGS differs fundamentally by shifting from per-Gaussian features to explicitly modeling pairwise relations between Gaussians as a directed sparse graph. Each Gaussian node has a lightweight descriptor combining a 6D PCA projection of semantic features (from masked CLIP features lifted via contribution weighting) and a scaled 3D position vector. Edge candidates are proposed by k-nearest neighbor search in this descriptor space, yielding a sparse digraph where each node connects to its k nearest neighbors.

Pairwise affinities for each edge are computed robustly by aggregating evidence across views. This evidence combines two factors: 1) whether the two Gaussians share the same mask index in a given view, where mask indices are derived by sampling both the Gaussian’s projected center and the pixel location of maximal view contribution to resolve boundary ambiguity; and 2) the product of their view-dependent contribution weights (a natural AND gate filtering weak/noisy signals). This yields an affinity score in [0,1] representing same-instance likelihood.

For hierarchical segmentation, the method introduces TreeDBSCAN, a modified density-based clustering adapted to directed affinity graphs. TreeDBSCAN clusters nodes at increasing affinity thresholds hierarchically, enforcing parent-child containment relationships and pruning redundant or spurious splits based on cluster size and erosion rate metrics. This results in a cohesive cluster tree supporting multi-granular queries from coarse to fine levels.

Training regime: PairGS is training-free for segmentation — it uses pretrained feature lifting from frozen models SAM and CLIP and PCA projection to generate node descriptors. No per-scene optimization or backpropagation is performed. Two variants are released: PairGS (using contribution-weighted lifting) and PairGS-Fast (using center-based lifting) differing only in initialization method.

Evaluation protocol uses metrics like mean Intersection over Union (mIoU) and accuracy (mAcc) for semantic and instance segmentation on held-out multi-view scenes. Efficiency is measured by runtime on a single NVIDIA RTX 6000 Ada GPU. Baselines include language-based methods (LangSplat, Occam’s LGS, Dr.Splat), instance-feature based approaches (OpenGaussian, InstanceGaussian, COS3D, LaGa), and a point-based baseline (THGS). Ablation studies evaluate contributions of positional scaling, graph directionality, use of view contribution, mask index sampling, and hierarchical clustering.

One example pipeline: Given a pretrained 3DGS model of a scene, PairGS lifts masked CLIP features via 2D views using SAM masks, projects them with PCA into 6D semantic features per Gaussian, concatenates scaled 3D position, then computes k-NN edges in this 9D space to form a sparse digraph. For each edge, multi-view masks and view contribution weights compute an affinity score. TreeDBSCAN then clusters Gaussians into hierarchical groups. For a given query (text label), clusters are refined semantically by associating aggregated cluster features with the query, enabling multi-granular segmentation without explicit retraining or optimization.

Reproducibility: The paper mentions using standard datasets and pre-trained components (SAM, CLIP, 3DGS). Detailed code release or frozen weights are not explicitly referenced, so full reproduction may require implementing the described pipeline. The methodology appears deterministic and training-free beyond feature lifting, aiding reproducibility.

Technical innovations

• Reframing 3D Gaussian segmentation as explicit pairwise relation estimation leveraging intrinsic view-dependent contribution and mask evidence signals instead of per-Gaussian features or instance optimization.
• Two-stage efficient graph construction using low-dimensional PCA-based semantic + scaled position descriptors for sparse k-NN edge proposal, followed by accurate affinity computation only on candidates.
• TreeDBSCAN, a novel hierarchical clustering algorithm for directed sparse affinity graphs that enforces parent-child containment and prunes redundant/spurious clusters, producing a single coherent hierarchy.
• Robust mask index retrieval per Gaussian using dual sampling (center and maximal contribution pixel) to detect boundary ambiguity and assign noise labels for cleaner affinity computation.

Datasets

• LERF-OVS — size unspecified — open-vocabulary object selection dataset with multi-view images and masks
• ScanNet — 1,513 RGB-D scans — public indoor scene dataset with semantic and instance segmentation labels

Baselines vs proposed

• LangSplat [39]: mIoU = 27.4% vs PairGS: 60.4% on LERF
• Occam’s LGS [9]: mIoU = 46.4% vs PairGS: 60.4% on LERF
• Dr.Splat [18]: mIoU = 46.8% vs PairGS: 60.4% on LERF
• OpenGaussian [48]: mIoU = 42.8% vs PairGS: 60.4% on LERF
• InstanceGaussian [29]: mIoU = 43.8% vs PairGS: 60.4% on LERF
• THGS [12]: mIoU = 42.5% vs PairGS-Fast: 56.8% on LERF
• ScanNet Semantic Segmentation (19 classes): InstanceGaussian [29] mIoU = 39.5%, PairGS mIoU = 40.7%
• ScanNet Instance Segmentation: InstanceGaussian [29] mIoU = 49.9%, PairGS mIoU = 51.9%
• Runtime: OpenGaussian [48] 48:42 vs PairGS-Fast 0:58 (min:sec) on LERF

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.01140.

Fig 1: We propose PairGS, a training-free method that estimates pairwise relations between Gaussians to enable efficient and fine-grained

Fig 2: Overview of PairGS. Our method begins with a pretrained 3DGS [19] and initializes its features by lifting SAM [22] masked

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Relies on availability and quality of multi-view 2D masks for instance evidence; small objects never separated in masks cannot be clustered.
• Semantic retrieval depends on SAM-masked CLIP features; semantically ambiguous clusters may cause query failures despite correct geometry.
• Does not incorporate learned instance features or fine-tuning, which could improve boundary precision in highly cluttered scenes.
• Evaluation limited to indoor scenes and object selection; robustness to outdoor or large-scale scenes not demonstrated.
• Hierarchical clustering thresholds are fixed and shared, which may limit adaption to diverse scene densities or object scales.
• No explicit adversarial robustness or handling of corrupted/missing views tested.

Open questions / follow-ons

• Can the relation-centric formulation be combined with trainable instance features or end-to-end learning to further improve boundary resolution?
• How robust is PairGS segmentation under occlusions, sparse views, or noisy/missing mask inputs in real-world scenarios?
• Could adaptive hierarchical thresholds or cluster validity metrics improve the TreeDBSCAN clustering for heterogeneous scenes?
• How well does the method generalize beyond indoor or synthetic scenes, e.g., to outdoor Lidar point clouds or large urban reconstructions?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, PairGS offers an efficient and interpretable mechanism for 3D scene semantic segmentation from multi-view inputs, utilizing intrinsic relations rather than requiring heavy optimization or monolithic embeddings. This relation-centric approach could inform novel defenses that rely on fine-grained spatial coherence and multi-view consistency to detect automated or spoofed queries. Moreover, the hierarchical clustering offers multi-granular object decomposition, potentially useful for adaptive challenge generation or verification at varying abstraction levels. The training-free nature and speed gains (50× faster than prior optimization-heavy methods) enable real-time or on-the-fly segmentation in live bot-defense pipelines, reducing latency and resource costs. However, limitations in semantic disambiguation and reliance on quality multi-view masks suggest that robust adversarial scenarios may need complementary mechanisms. Overall, this work demonstrates how explicit modeling of geometric and appearance relations can improve both segmentation quality and computational efficiency in 3D language grounding tasks relevant to security contexts.

Cite

@article{arxiv2607_01140,
  title={ Relation-Centric Open-Vocabulary 3D Gaussian Segmentation },
  author={ Eunsung Cha and Hyunjoon Lee and Jaesik Park },
  journal={arXiv preprint arXiv:2607.01140},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.01140}
}

Read the full paper

• arXiv:2607.01140 (abstract)
• arXiv:2607.01140 (PDF)