StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views

Source: arXiv:2606.28321 · Published 2026-06-26 · By Jia-Chen Zhao, Beiqi Chen, Xinyang Chen, Guangcong Wang, Liqiang Nie

TL;DR

StructSplat addresses the challenging problem of generalizable 3D scene reconstruction from uncalibrated, sparse multi-view images without requiring any camera intrinsics or extrinsics. Unlike prior methods that rely on per-scene optimization or known camera parameters, StructSplat adopts a structured feed-forward neural framework that separately models geometry, semantic, and texture cues with explicit functional roles. This design overcomes the entanglement of appearance and geometry common in previous unified backbones, which often limits reconstruction fidelity and generalization.

The core innovation includes a pixel-aligned feature injection mechanism for accurate texture modeling, a semantic encoder to provide high-level scene priors enhancing global consistency, and a camera alignment strategy to prevent target-view information leakage during training and improve generalization. StructSplat achieves state-of-the-art results on major benchmarks, surpassing the leading camera-parameter-free method AnySplat by +5.67 dB PSNR on DL3DV and +1.94 dB and +1.72 dB on cross-dataset evaluations of ACID and RealEstate10K, respectively. Qualitative results demonstrate superior detail preservation, sharper edges, and fewer artifacts in novel view synthesis. Ablations confirm each component’s utility including semantic features, texture features, and camera alignment.

Key findings

• StructSplat achieves 28.045 PSNR on DL3DV, outperforming AnySplat's 22.377 by +5.67 dB, with SSIM improved from 0.716 to 0.888 and LPIPS reduced from 0.150 to 0.091.
• Cross-dataset evaluation shows a +1.94 dB PSNR gain on ACID (24.372 vs 22.433) and +1.72 dB on RealEstate10K (22.240 vs 20.521) over AnySplat without relying on camera parameters.
• Ablation adding semantic encoder features boosts PSNR from 20.610 to 26.236 on DL3DV, showing semantic cues resolve structural ambiguities and improve global consistency.
• Adding pixel-aligned texture features further raises PSNR from 26.236 to 28.045, proving critical for recovering fine-grained textures and colors.
• Camera alignment strategy reduces Absolute Trajectory Error for rotation from 0.0866 (AnySplat) to 0.0016 and for translation from 0.1003 to 0.0093, enhancing spatial consistency.
• Removing camera alignment causes PSNR to drop from 28.045 to 27.338 on DL3DV, indicating its importance in preventing information leakage and preserving reconstruction quality.
• StructSplat consistently outperforms baselines across different numbers of source views (2, 4, 6) on DL3DV with PSNR rising up to 31.137 for 6 views.
• Compared with pose-dependent but intrinsics-based methods (e.g., PF3plat's 22.579 PSNR), StructSplat achieves competitive or better results despite requiring no camera info.

Threat model

The adversary is an uncalibrated multi-view capture scenario posing the challenge of reconstructing 3D scenes from sparse, camera parameter-free RGB images. The threat model assumes no access to camera intrinsic or extrinsic information, making prior geometry or pose knowledge unavailable. The model aims to prevent information leakage from target views during training to avoid implicit supervision and maintain fair generalization. Adversaries cannot manipulate camera poses or corrupt the multi-view inputs; the threat is real-world operational complexity rather than malicious attacks.

Methodology — deep read

1. Threat Model & Assumptions: The approach assumes an adversary or user provides uncalibrated sparse multi-view RGB images without any camera intrinsics or extrinsics. The method must reconstruct a 3D scene representation generalizable across unseen scenes. The adversary cannot tamper with the input beyond standard camera capture variability, and the system avoids target-view information leakage during training to ensure fair generalization.

2. Data: Experiments use three datasets—DL3DV, RealEstate10K, and ACID. DL3DV is used for training and validation with official splits; RealEstate10K and ACID are held out for cross-dataset evaluation to examine generalization. Input images are preprocessed by resizing and center-cropping to 256×256. Ground truth novel views are used for evaluation but camera parameters are not provided to the model.

3. Architecture / Algorithm: StructSplat uses a structured multi-encoder design. A VGGT-based geometry encoder extracts 3D scene structure through attention mechanisms across source views, producing tokenized geometry features and depth predictions. A pre-trained vision foundation model (DINOv3) serves as a semantic encoder extracting view-invariant 2D semantic features providing global scene priors. A lightweight texture encoder extracts high-resolution pixel-aligned convolutional features to preserve fine color and detail. These three feature streams feed into a Gaussian decoder with modified DPT heads using cascaded Reassembling and Fusion blocks that spatially align and fuse geometric and semantic features. At a late stage, pixel-aligned texture features are injected explicitly. The decoder outputs camera-centric Gaussian primitives parameterized by 3D position, opacity, color, scale, and rotation quaternion. In parallel, a camera decoder estimates source and target camera parameters from mixed and source-only views. A camera alignment module then aligns target camera poses to source-only coordinate system via constrained optimization minimizing rotation and translation differences to enforce world consistency and avoid cross-view leakage.

4. Training Regime: Training uses BF16 mixed precision with DeepSpeed for efficient scaling and a Warmup-Stable-Decay learning rate scheduler to ensure stable convergence. Training jointly renders source and target views for multi-view consistency with supervision against target ground truths. The camera alignment module ensures the input to Gaussian decoder comes solely from source-only features, separating target info. Hyperparameters like batch size and epochs are unspecified in detail but training uses standard deep learning practice on GPUs.

5. Evaluation Protocol: Metrics include PSNR, SSIM, and LPIPS computed on synthesized target views. Baselines include state-of-the-art camera-parameter-free and pose-free methods such as AnySplat, RayZer, Depth Anything 3, PF3plat, and NoPoSplat. Cross-dataset evaluation by training on DL3DV and testing on ACID and RealEstate10K assesses robustness to distribution shift. Ablations sequentially remove semantic and texture features and the camera alignment module to measure their impact. Absolute Trajectory Error evaluates pose alignment accuracy.

6. Reproducibility: The authors provide code and a project page publicly (https://github.com/J-C-Zhao/StructSplat, https://structsplat.github.io). Datasets are public benchmarks. Exact hyperparameter details and random seeds are not explicitly stated but the released code presumably facilitates reproducibility. Camera alignment involves a closed-form quaternion optimization and least-squares translation alignment.

Example pipeline: Given N uncalibrated images, features are extracted separately by geometry, semantic, and texture encoders. The Gaussian decoder predicts per-pixel Gaussian parameters in local camera coordinates. The camera decoder predicts extrinsics for source and target views from two parallel streams. Target camera poses are then aligned to source-only coordinate system to avoid leakage. Finally, camera-centric Gaussians are transformed into a global coordinate frame and rendered via differentiable Gaussian splatting to novel viewpoints for supervision and evaluation.

Technical innovations

• A structured multi-encoder representation that disentangles geometry, semantic, and texture cues with dedicated pathways to improve reconstruction fidelity and generalization.
• A pixel-aligned feature injection mechanism that injects high-resolution 2D texture features late in the Gaussian decoding pipeline to accurately recover fine-grained, view-dependent appearance details.
• A semantic encoder leveraging pretrained foundation models (DINOv3) to incorporate high-level scene priors that improve global consistency and structural ambiguity resolution under sparse uncalibrated views.
• A camera alignment strategy that estimates and aligns source and target camera parameters from separate feature streams to enforce a unified coordinate system, preventing target-view information leakage and enhancing fair training and generalization.

Datasets

• DL3DV — official benchmark split — public dataset
• RealEstate10K — cross-dataset evaluation — public video dataset with diverse real-world indoor/outdoor scenes
• ACID — cross-dataset evaluation — public dataset with complex camera motions and challenging layouts

Baselines vs proposed

• AnySplat: PSNR = 22.377 on DL3DV vs StructSplat: 28.045 (+5.67 dB)
• AnySplat: PSNR = 22.433 on ACID vs StructSplat: 24.372 (+1.94 dB)
• AnySplat: PSNR = 20.521 on RealEstate10K vs StructSplat: 22.240 (+1.72 dB)
• NoPoSplat: PSNR = 25.592 on DL3DV vs StructSplat: 28.045 (+2.45 dB) — Note NoPoSplat uses intrinsics
• PF3plat (intrinsics known): PSNR = 22.579 on DL3DV vs StructSplat (no camera parameters): 28.045
• Removing semantic features: PSNR drops from 28.045 to 20.610 on DL3DV
• Removing texture features: PSNR drops from 28.045 to 26.236 on DL3DV
• Without camera alignment: PSNR drops from 28.045 to 27.338 on DL3DV

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28321.

Fig 1: Qualitative comparison for novel-view synthesis from uncalibrated

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The method requires training on fairly large datasets with ground truth multi-view images and relies on large pretrained models (e.g., DINOv3) for semantics.
• While the approach removes dependency on explicit camera parameters, the camera alignment step adds complexity and depends on accurate pose estimation from learned features.
• No reported evaluation under strong adversarial attacks or corrupted input images that could destabilize camera pose estimation or Gaussian decoding.
• Cross-dataset tests exist but broader generalization to unconstrained outdoor scenes or drastically different domains is unreported.
• Training details such as epoch counts, exact hyperparameters, and random seeds are not exhaustively specified, which may impact reproducibility.
• Methods building on attention-based encoders may incur higher memory costs and latency compared to per-scene optimization approaches.

Open questions / follow-ons

• Can the structured representation and camera alignment strategy be extended to handle dynamic scenes or non-rigid objects?
• How robust is the method to extreme sparse view counts (e.g., 1-2 images) or very wide-angle cameras with severe distortion?
• Can unsupervised or self-supervised variants be developed to eliminate the need for ground truth novel views during training?
• What are the trade-offs between feed-forward generalization and traditional per-scene optimization in terms of runtime and reconstruction fidelity in diverse environments?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, StructSplat presents a cutting-edge example of leveraging advanced 3D reconstruction without reliance on explicit camera calibration. While not directly a bot-defense tool, the method's capability to reconstruct faithfully from uncalibrated inputs could inform future challenges assessing well-formed human-like visual data versus synthetic or spoofed inputs. For CAPTCHA systems using novel-view synthesis or 3D scene understanding as proofs of humanness, StructSplat-style structured representations and leakage-preventing alignment strategies provide valuable insights. Understanding how to disentangle semantic, geometric, and texture features robustly could help detect synthetic views generated by bots or aid in designing CAPTCHAs requiring 3D spatial reasoning from minimal visual cues.

Cite

@article{arxiv2606_28321,
  title={ StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views },
  author={ Jia-Chen Zhao and Beiqi Chen and Xinyang Chen and Guangcong Wang and Liqiang Nie },
  journal={arXiv preprint arXiv:2606.28321},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28321}
}

Read the full paper

• arXiv:2606.28321 (abstract)
• arXiv:2606.28321 (PDF)