Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

Source: arXiv:2606.23688 · Published 2026-06-22 · By Yehonathan Litman, Xiaoxuan Ma, Manan Shah, Nicolas Ugrinovic, Kris Kitani, Fernando De la Torre et al.

TL;DR

Lift4D addresses the challenge of reconstructing dynamic, non-rigid 4D (3D + time) representations of unknown objects from monocular in-the-wild videos, where data is scarce and occlusions plus large deformations are common. Unlike prior methods that either directly predict 4D shapes limited by synthetic training data or optimize deformable static models without continuous data-driven priors, Lift4D combines both by leveraging an adapted single-view 3D reconstruction backbone with causal latent conditioning to produce temporally consistent per-frame geometry. This serves as a coherent initialization for a deformable 3D Gaussian Splatting representation that is further refined through occlusion-aware test-time optimization aided by view-conditioned diffusion priors for unobserved regions. This method harmonizes geometry and appearance priors effectively and produces superior 4D reconstructions in terms of structure, motion accuracy, and novel view semantic fidelity.

Key findings

• Causal latent propagation in the single-view 3D backbone reduces temporal jitter compared to independent per-frame reconstructions, enabling temporally consistent geometry (Sec. 3.1, Fig. 2).
• Lift4D achieves 0.072 End Point Error (EPE) on in-the-wild Pexels videos, outperforming the closest baseline PAD3R at 0.119 (Tab. 1).
• On the synthetic Consistent4D benchmark, Lift4D improves LPIPS to 0.116 versus PAD3R's 0.137 and V2M4's 0.192, while also achieving the best CLIP semantic score of 0.950 (Tab. 2).
• Occlusion-aware rendering supervision using depth-based occlusion masks and color-corrected per-frame reconstructions improves robustness to partial visibility and occlusion (Sec. 3.3, Fig. 5).
• The use of a view-conditioned 2D image diffusion prior (LSDS) during appearance reconstruction significantly enhances completion of unobserved and occluded regions; ablating LSDS causes LPIPS to degrade from 0.116 to 0.170 and CLIP from 0.950 to 0.848 (Tab. 3).
• Introducing motion regularization loss (Lreg) stabilizes temporal coherence; removing it increases FVD from 592.44 to 794.82 (Tab. 3).
• Lift4D reconstructs fully consistent 4D geometry and appearance without multi-view input or category-specific templates, enabling robust generalization to diverse in-the-wild sequences with large non-rigid deformations.
• Ablating causal latent conditioning by inferring per-frame reconstructions independently degrades geometric quality and temporal fidelity (Tab. 3).

Threat model

The adversary is the intrinsic ambiguity and incompleteness inherent in monocular video capturing dynamic, deforming objects with occlusions under uncontrolled lighting and viewpoints. The system does not assume access to multi-view data or category-specific templates that would disambiguate geometry or motion. The adversary cannot manipulate input segmentation masks or depth estimates, and cannot provide multi-view or ground-truth 3D supervision during test-time optimization.

Methodology — deep read

1. Threat model & assumptions: The adversary is effectively the intrinsic ambiguity and incompleteness arising from monocular video input of dynamic, non-rigid objects in uncontrolled real-world scenes. The method assumes only monocular RGB video plus object segmentation masks and monocular depth estimates. No multi-view or template shape priors are assumed. The goal is to reconstruct temporally consistent 4D geometry and appearance despite occlusion, large deformation, and sparse viewpoints.

2. Data: Lift4D is tested on 10 in-the-wild internet videos (Pexels dataset) showing deformable and rigid objects with occlusions, plus 8 real-world videos from DAVIS. Videos range from 77 to 100 frames. Synthetic evaluation is done on Consistent4D, a collection of 7 synthetic object videos with ground-truth novel views. No training dataset is needed for Lift4D itself—reconstruction is test-time optimization.

3. Architecture / algorithm: A pretrained image-to-3D model (based on flow-matching diffusion, specifically SAM3D) produces per-frame 3D Gaussian Splatting geometry reconstructions as latent codes Z_i. Temporal consistency is enforced by causal latent conditioning: for frame i, its latent Z_i,t0 at intermediate diffusion timestep t_0 is linearly interpolated between fresh noise and the prior frame's denoised latent Z_{i-1}. This adapts the single-frame diffusion backbone into a weak 4D prior without retraining.

The per-frame consistent Gaussian Splat reconstructions G_i are distilled into a canonical deformable 3D Gaussian representation G*, parameterized by sparse control nodes {p_k}. Deformations over time are predicted by an MLP ψ that outputs SE(3) transformations per node per frame, using linear blend skinning to deform all Gaussians.

The deformation is optimized to minimize a Chamfer distance alignment loss and a multi-view rendering loss (L1 + DSSIM) between renderings of deformed G* and the per-frame G_i, with a learnable per-frame alignment transform T_align to absorb drift.

Appearance reconstruction is performed after freezing ψ to preserve coarse motion. A denser set of control nodes with separate per-frame SE(3) parameters deform color attributes. This fine appearance refinement is supervised by a new occlusion-aware rendering loss, which composites detected occluded pixels with color-matched renderings of G_i to produce a clean reference view.

Finally, a novel view-conditioned score-distillation sampling (SDS) loss leverages a pretrained 2D image diffusion prior conditioned on the occlusion-completed reference to hallucinate plausible appearance in unobserved and occluded regions.

The overall loss combines the reconstruction geometry loss, appearance loss, and a motion regularization penalty.

4. Training regime: Lift4D does not train a model from scratch but conducts test-time optimization per video, running 20,000 iterations divided evenly between geometry and appearance phases using AdamW optimizer. Optimization uses sparse control nodes and rendered losses on H200 GPU hardware, completing a 32-frame video in ~30 minutes. Causal conditioning is performed on the fly during each optimization iteration.

5. Evaluation protocol: Evaluation uses multiple metrics. For synthetic data with ground-truth novel views: perceptual similarity (LPIPS), temporal coherence (FVD), and semantic alignment (CLIP score). For in-the-wild data without GT novel views: image and text query CLIP scores for novel view renderings, and a 3D motion accuracy metric (EPE) measuring reprojection error of tracked dense 3D points versus 2D matches predicted by CoTracker3. Multiple baselines spanning feedforward, diffusion, and test-time optimization with differing priors are benchmarked on both synthetic and real-world video datasets.

6. Reproducibility: The methodology relies on publicly available pretrained components (SAM3D, image-to-3D models, monocular depth estimators, SAM segmentation, diffusion models) with Lift4D implemented as an orchestration framework with test-time optimization. The datasets used are public or commonly available (Pexels videos, DAVIS, Consistent4D). Code and further implementation details are publicly linked, enabling reproduction of key results. Some training details (hyperparameters, regularization weights) are in the appendix.

Example walk-through: For a monocular video, the first frame latent Z_0 is fully denoised, then causal latent conditioning propagates latent information through frames to ensure temporal consistency of per-frame Gaussian splat reconstructions G_i. These G_i reconstructions are distilled into a canonical deformable 3D Gaussian set with sparse control nodes. An optimization loop minimizes alignment with G_i via Chamfer and rendering losses while learning deformable motions ψ. In parallel, an occlusion-aware rendering supervision scheme masks out occluded pixels based on depth comparisons and fuses color-matched renderings for reliable appearance supervision. On top of this, a 2D diffusion-based prior conditions on these completed renderings to regularize appearance in unobserved regions via a distillation loss. This process converges to a temporally and spatially coherent 4D dynamic reconstruction with plausible completions.

Technical innovations

• Causal latent conditioning in single-view image-to-3D diffusion models to propagate temporal consistency without retraining.
• Optimization of deformable canonical 3D Gaussian Splatting with sparse control nodes driven by enhanced temporally consistent per-frame reconstructions.
• Occlusion-aware rendering supervision combining depth-based occlusion masks and histogram color matching to improve appearance fitting in partially visible regions.
• Anchoring unobserved and occluded regions with view-conditioned 2D image diffusion priors during appearance optimization.

Datasets

• Pexels in-the-wild videos — 10 videos (77–100 frames each) — publicly sourced internet videos
• DAVIS dataset — 8 real-world videos (77–100 frames each) — publicly available video segmentation dataset
• Consistent4D synthetic benchmark — 7 synthetic object videos with 32 frames each and 4 ground-truth novel view videos — publicly released 4D evaluation dataset

Baselines vs proposed

• Pexels dataset: PAD3R baseline EPE = 0.119 vs Lift4D EPE = 0.072
• Pexels dataset: STAG4D CLIP = 0.757 vs Lift4D CLIP = 0.780
• Consistent4D dataset: PAD3R LPIPS = 0.137 vs Lift4D LPIPS = 0.116
• Consistent4D dataset: DM4D FVD = 688.84 vs Lift4D FVD = 592.44
• Consistent4D dataset: PAD3R CLIP = 0.942 vs Lift4D CLIP = 0.950

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.23688.

Fig 1: 4D Reconstruction from Monocular In-the-Wild Video. Given a video of a dynamic scene, Lift4D recovers the full geometry,

Fig 2: Causal Single-view Reconstruction. Given a video input, we obtain per-frame 3D reconstructions Gi with an image-to-3D

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Dependence on quality of per-frame initial SAM3D reconstructions; errors propagate without mechanisms to refine or reject flawed priors.
• Optimization is computationally intensive (~30 minutes per 32-frame video on high-end GPU), limiting real-time applicability.
• No explicit handling of multi-object inter-occlusions or complex articulated human interactions, such as grasping or object-object contact.
• Limited evaluation on extreme viewpoint extrapolation where monocular cues alone are insufficient; heavy occlusions challenge completion fidelity.
• Ablations suggest motion regularization and diffusion-based appearance priors are critical, indicating fragility without these components.

Open questions / follow-ons

• How to improve the geometry initialization stage, e.g., by integrating more robust or learned multi-frame priors to reduce error propagation from single-view latent conditioning.
• Extending the framework for interacting multiple objects with mutual occlusions or contact dynamics in dynamic scenes.
• Incorporating explicit semantic or physical constraints to better regularize large deformations and rare poses beyond data-driven priors.
• Reducing computational load and speeding up test-time optimization to enable near real-time 4D reconstruction from monocular video.

Why it matters for bot defense

For bot defense engineers designing CAPTCHA or bot detection mechanisms utilizing 3D reconstructions or dynamic scene analysis, Lift4D’s methodology demonstrates a novel way to combine strong priors with test-time optimization to recover consistent 4D object representations from monocular video in unconstrained real-world settings. The causal latent conditioning approach ensures temporal coherence essential for reliable tracking or interaction modeling across frames, while the occlusion-aware supervision and diffusion priors enable plausible completion of partially observed content, common in security scenarios involving adversarial occlusions or spoofing attempts. Lift4D’s balance between data-driven priors and optimization-based refinement could inform robust bot-detection systems that rely on appearance and motion coherence cues, particularly when multi-view data is unavailable. However, computational cost and dependency on quality initial reconstructions may pose practical challenges for real-time deployment in CAPTCHA pipelines, suggesting this is more applicable to offline forensic analysis or high-assurance verification contexts. Overall, Lift4D provides an advanced framework in the limited-data monocular input regime relevant for scenarios where adversaries attempt to fool dynamic object recognition or pose estimation.

Cite

@article{arxiv2606_23688,
  title={ Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild },
  author={ Yehonathan Litman and Xiaoxuan Ma and Manan Shah and Nicolas Ugrinovic and Kris Kitani and Fernando De la Torre and Shubham Tulsiani },
  journal={arXiv preprint arXiv:2606.23688},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.23688}
}

Read the full paper

• arXiv:2606.23688 (abstract)
• arXiv:2606.23688 (PDF)