FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation

Source: arXiv:2606.24876 · Published 2026-06-23 · By Orest Kupyn, Goutam Bhat, Philipp Henzler, Fabian Manhardt, Christian Rupprecht, Federico Tombari

TL;DR

This paper addresses the problem of generating explorable, geometrically accurate 3D scenes from a single input image by leveraging latent representations from pretrained video diffusion models. Existing feedforward latent scene decoders produce volumetric 3D Gaussians which lack clear surfaces, limiting their application in real-time rendering or graphics pipelines. The authors propose FLAT, a novel feedforward decoder that directly maps compressed video diffusion latents to explicit surface primitives in the form of soft triangle splats in a single forward pass. This approach overcomes challenges related to instability and poor gradient flow in predicting flat triangle primitives by introducing a ray-centered rotation parameterization and an improved differentiable window function for rendering.

FLAT achieves significantly better geometric accuracy than state-of-the-art volumetric Gaussian splatting baselines across standard datasets while maintaining comparable visual quality. The paper also introduces a lightweight test-time refinement step to convert the semi-transparent triangle representation into an opaque, mesh-compatible format suitable for real-time rendering in graphics engines. The authors provide the first systematic evaluation comparing 3D Gaussian Splatting, 2D Gaussian Splatting, and triangle splatting approaches under identical training regimes, demonstrating clear tradeoffs between rendering fidelity, geometric accuracy, and downstream usability. Overall, FLAT demonstrates a practical, feedforward pathway from single images to high-quality, game-engine-ready 3D scenes that balances generative priors from large video models with explicit surface representation.

Key findings

• FLAT’s triangle splat representation achieves a cosine similarity of 0.853 for normal map accuracy on RealEstate10K and DL3DV datasets, surpassing 2DGS’s 0.587 and 3DGS’s near random normals.
• FLAT maintains competitive novel view synthesis quality with PSNR of 21.45 and SSIM of 0.710 on RealEstate10K, close to volumetric 3DGS (22.39 PSNR, 0.762 SSIM).
• Opaque mesh conversion via lightweight test-time refinement yields over 7 dB PSNR improvement compared to 3DGS mesh extraction on RealEstate10K (21.23 vs 14.18).
• Direct world-space rotation prediction leads to training divergence; FLAT’s ray-centered residual rotation parameterization stabilizes training, enabling feedforward triangle decoding.
• The novel product window function for differentiable triangle rendering improves gradient flow across boundaries compared to previous sigmoid or max distance functions.
• Ablations show that each key component (ray-centered parameterization, window function, architecture) is essential for geometric quality and training stability.
• Training solely the feedforward decoder atop a frozen video latent VAE enables generation without costly scene optimization steps.
• FLAT supports arbitrary novel camera trajectories without post-optimization, unlike some prior generate-then-optimize pipelines.

Methodology — deep read

1. Threat Model & Assumptions: The work assumes an input single RGB image and a known camera trajectory for novel view synthesis of the scene. The adversary model is not explicitly defined as this is not a security paper, but the challenge is to infer accurate 3D scene geometry and appearance solely from 2D input and camera parameters, accounting for occlusions and ambiguous depth.

2. Data: The training dataset comprises a mixture of real and synthetic video sequences with associated depth and normal maps. RealEstate10K and DL3DV provide realistic scenes and camera trajectories; synthetic data comes from object-centric S3OD images rendered into videos by the Uni3C video diffusion model, expanding scene diversity. Camera parameters are refined using MapAnything for metric scale consistency. Ground-truth or pseudo ground-truth normals are generated by NormalCrafter.

3. Architecture / Algorithm: FLAT builds on a frozen pretrained video diffusion latent VAE backbone (Wan-2.1), encoding camera-conditioned video latents. A feedforward scene decoder modifies the VAE's RGB decoder by removing the last upsampling stage and adding lightweight output heads that regress triangle splat parameters (position, shape, orientation, color, opacity, sharpness) at a 2x2 pixel spatial resolution per token. Each triangle is parameterized via a canonical equilateral triangle transformed by a constrained Cholesky-style 2x2 matrix ensuring positive area avoidance of degenerate triangles, and oriented using a stable ray-centered frame with residual tilt and spin angles. Camera conditioning includes encoding dense per-pixel ray embeddings with the RPPC parameterization and injection into zero-initialized fusion blocks.

A novel product window function provides a differentiable softened triangle boundary for rasterization, improving gradient flow by summing contributions from all edges instead of a max operation. Rendering follows standard alpha compositing from triangles projected by pinhole camera equations.

4. Training Regime: Only the scene decoder is trained, atop frozen video latents. Training uses AdamW at 1e-4 learning rate over 200,000 iterations on 8 Nvidia H100 GPUs. The process is staged progressively from 320p to 768p resolution. Losses include L2 pixel loss, LPIPS perceptual loss, scale-invariant disparity (depth) loss, normal map cosine loss, and opacity regularization. Triangles with opacity below 40% are pruned during high-res training.

5. Evaluation Protocol: Evaluated on RealEstate10K and DL3DV single-image to 3D scene generation, compared with SOTA feedforward (Lyra, Bolt3D) and optimization-based baselines (Wonderland) all relying on 3D Gaussian splatting (3DGS). Metrics include PSNR, SSIM, LPIPS for novel view synthesis quality, and cosine similarity and L1 error for geometric normal accuracy using Metric3D-v2 to reduce bias. Ablations test rotation parameterization, window function, architecture variations. Opaque mesh conversion quality is measured after postprocessing.

6. Reproducibility: The paper mentions a project webpage with code. The video diffusion backbone is Uni3C (Wan-2.1 VAE) pretrained. Datasets used include public RealEstate10K, DL3DV, and S3OD synthetic data. Some real camera pose labels come from RealCam-Vid. Frozen VAE encoders mean only the scene decoder is trained, simplifying replication. Exact seeds/hyperparams beyond the learning rate and iteration count are not detailed.

Technical innovations

• A novel ray-centered triangle parameterization employing a constrained Cholesky-style 2x2 shape matrix and residual rotations around a ray-aligned frame for stable feedforward triangle regression.
• A new product-based smooth window function for differentiable triangle splatting that improves gradient flow across edges compared to prior sigmoid or max-based window approximations.
• A single-pass feedforward decoder architecture built on frozen video latent diffusion models that maps compressed latents directly to explicit semi-transparent triangle splats without expensive per-scene optimization.
• A lightweight post-processing refinement turning predicted semi-transparent triangle soups into coherent, opaque meshes compatible with real-time graphics engines and game pipelines.
• The first systematic feedforward comparison of 3D Gaussian Splatting, 2D Gaussian Splatting, and triangle splatting under identical training setups, isolating representation tradeoffs.

Datasets

• RealEstate10K — ~10k videos — public dataset of real camera movements in indoor/outdoor scenes
• DL3DV — size not specified — real multi-view dataset with reliable camera poses
• S3OD synthetic — 25,000 images synthesized into videos using Uni3C to expand training variety with known camera metadata

Baselines vs proposed

• Lyra 3DGS: PSNR=21.79 vs FLAT 3DGS: PSNR=22.39; FLAT improves by 0.6 dB
• Lyra 3DGS: LPIPS=0.219 vs FLAT 3DGS: LPIPS=0.203; FLAT reduces perceptual error
• 2DGS baseline: cosine similarity (normal accuracy) = 0.587 vs FLAT triangles: 0.853; FLAT improves normal accuracy by ~45%
• 3DGS baseline geometries yield near-random normals (not reported quantitatively), while FLAT triangle splats provide structured surfaces
• Opaque mesh conversion PSNR on RealEstate10K: 3DGS GS2Mesh=14.18 vs FLAT opaque mesh=21.23; >7 dB improvement
• Ablation: Direct world-space rotation regression causes failure vs FLAT ray-centered residual rotations yield stable convergence with PSNR ~21.45

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.24876.

Fig 1: FLAT regress soft triangles directly from video diffusion latent, enabling geometrically

Fig 2: Pipeline: Starting from a single image, we construct a point-cloud-based control video by

Fig 3: Window Function: Comparison of sigmoid-based window function [26, 14], max edge

Fig 4 (page 4).

Fig 5 (page 4).

Fig 6 (page 4).

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• The approach relies on high-quality pretrained video diffusion latent models and frozen encoders, limiting adaptability to new domains without retraining the backbone.
• Evaluation focuses on typical indoor/outdoor datasets but does not include adversarial or extreme lighting/viewpoint conditions impacting robustness.
• The synthetic data distribution and camera trajectories are approximated rather than real, potentially limiting generalization to complex real-world motion.
• Postprocessing for mesh conversion, though lightweight, still introduces additional optimization and heuristics rather than producing fully end-to-end trainable meshes.
• Opacity pruning thresholding and the choice of window function hyperparameters may impact final visual fidelity and require dataset-specific tuning.
• No detailed timing or real-time throughput benchmarks reported, leaving practical runtime performance and scalability unclear.

Open questions / follow-ons

• Can the ray-centered parameterization and window function be extended to other explicit surface primitives beyond triangles, e.g., quads or higher-order patches?
• How does FLAT perform under more challenging occlusions, dynamic scenes, or unknown camera trajectories not provided at inference?
• Could end-to-end training of the whole video diffusion latent encoder and the triangle decoder improve fidelity or robustness?
• What are the tradeoffs in compression or storage size when using triangle splatting vs Gaussian splatting for large-scale scene datasets?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, FLAT presents a significant advance in the generation of explicit, geometrically accurate 3D scene representations from limited input (single images), without expensive optimization. This is relevant when designing or analyzing systems that attempt to either synthesize or verify 3D scenes for authenticity or human verification. Its method of reliably decoding explicit surface primitives directly from compressed latent representations could inspire new approaches to efficiently generate realistic yet verifiable 3D content for interactive challenges. The focus on feedforward, single-pass generation with explicit surface control aligns well with requirements for scalable, real-time deception or detection systems, as explicit surfaces facilitate physical simulation or rendering properties that volumetric alternatives lack. Additionally, the improved geometric fidelity compared to Gaussian splatting suggests opportunities to raise the bar on both generation realism and on adversarial detection relying on geometric consistency.

Cite

@article{arxiv2606_24876,
  title={ FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation },
  author={ Orest Kupyn and Goutam Bhat and Philipp Henzler and Fabian Manhardt and Christian Rupprecht and Federico Tombari },
  journal={arXiv preprint arXiv:2606.24876},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.24876}
}

Read the full paper

• arXiv:2606.24876 (abstract)
• arXiv:2606.24876 (PDF)