TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy

Source: arXiv:2606.26092 · Published 2026-06-24 · By Hao Sun, Hao Yan, Mengting Chen, Quanjian Song, Yu Li, Juan Cao et al.

TL;DR

This paper addresses a key limitation in Video Virtual Try-on (VVT) systems: their inherent dependency on fixed source camera trajectories, which restricts user interaction and viewpoint exploration. To overcome this, the authors propose a new research task called Camera-controllable Video Virtual Try-on (CaM-VVT), which requires synthesizing realistic try-on videos under arbitrary, user-controlled camera movements while preserving structural consistency of non-rigid human motion and background alignment. The main contribution is TryOnCrafter, a unified end-to-end framework built around a novel Renderable 4D Try-on Proxy. This proxy explicitly decomposes the dynamic scene into a clothed 3D Gaussian Splatting (3DGS)-based human avatar animated by metric-aligned SMPL-X sequences, coupled with a reconstructed background point cloud. It serves as a robust geometric anchor guiding a Proxy-Anchored Video Diffusion Transformer (DiT) that synthesizes photorealistic videos from prescribed camera trajectories. Extensive experiments on both conventional VVT benchmarks and a newly introduced CaM-VVTBench demonstrate that TryOnCrafter improves spatio-temporal consistency, garment fidelity, and viewpoint-agnostic synthesis compared to state-of-the-art baselines. Additionally, the approach enables downstream applications like human relocalization, 360-degree orbital viewing, and bullet-time effects in virtual try-on.

Key findings

• TryOnCrafter achieves VFIDpI = 9.61 and VFIDpR = 0.18 on ViViD paired evaluation, outperforming previous methods (DreamVVT best prior: 11.01 and 0.25 respectively).
• In unpaired ViViD evaluation, TryOnCrafter attains VFIDuI = 10.76 and VFIDuR = 0.21, surpassing Magic-Tryon (17.57 and 0.51).
• On CaM-VVTBench, TryOnCrafter attains an Overall Score of 75.47, beating two-stage baselines like Magic-Tryon+TrajectoryCrafter at 71.64 and Magic-Tryon+ReCaMaster at 61.92.
• Ablation removing the 4D proxy rendered prior increases VFIDpI from 9.61 to 16.46, indicating the proxy is critical for structural integrity.
• Removing multi-modal semantic garment cues degrades unpaired VFIDuI from 10.76 to 18.51, showing semantic conditioning aids generalization.
• Anchor-based alignment of SMPL-X pose to world reconstructed point cloud reduces spatial misalignments, critical under aggressive camera movement.
• CRA (Cross-view Reference Adapter) module helps preserve identity and appearance details by fusing latent features across views with minimal computation.
• Rendered 4D Try-on Proxy enables physically plausible cloth deformation across radical novel camera viewpoints unlike previous point cloud based methods.

Threat model

n/a — The work focuses on synthesizing camera-controllable video virtual try-on without a direct security adversary; the challenges are technical rather than adversarial, concerning accurate geometry and texture preservation under novel camera viewpoints.

Methodology — deep read

TryOnCrafter tackles the newly defined Camera-controllable Video Virtual Try-on (CaM-VVT) task, which requires synthesizing realistic virtual try-on videos under arbitrary user-prescribed camera trajectories.

1. Threat Model: The task assumes a monocular source video of a person wearing base garments, and the adversarial challenge is to generate photorealistic videos with novel camera movements never seen in the input. No explicit security adversary, but challenges arise from viewpoint-agnostic texture synthesis and geometry-motion disambiguation.

2. Data: The framework is trained primarily on the ViViD dataset (7,759 training videos) and tested on the ViViD-S subset and a newly curated CaM-VVTBench dataset comprising ~60K clips for training and 96 test samples with predefined camera motion patterns (tilt, zoom, orbit). Data includes dense monocular depth, point clouds, SMPL-X pose estimations, and target garment overlays.

3. Architecture:

• The system first reconstructs a 4D Try-on Proxy: a combination of a clothed 3D Gaussian Splatting (3DGS)-based canonical avatar and a metric-aligned background point cloud, dynamically animated by pose sequences parameterized via SMPL-X.
• Anchor-Based Alignment applies a weighted similarity transform (scale, rotation, translation) to map SMPL-X pose vertices from camera space to a global world space point cloud representation, minimizing noisy mismatches using confidence weights.
• The canonical 3DGS avatar is created by distilling high-fidelity 2D try-on reference images into a 3D Gaussian primitives set linked to a posed SMPL-X mesh, enabling view-consistent geometry and texture.
• The avatar is dynamically deformed with Linear Blend Skinning (LBS) using the aligned SMPL-X pose sequence, with transformations propagated into world space ensuring temporal and spatial coherence.
• The Proxy-Anchored Video Diffusion Transformer (DiT) synthesizes final photorealistic try-on videos guided by rendered proxy videos acting as structural priors.
• A Cross-view Reference Adapter (CRA) module fuses DiT features with source domain reference features for fine-grained identity preservation using shared key/value projections and distinct query/output weights.
• Multi-modal semantic conditioning injects garment texture cues (CLIP image embeddings) and attribute semantics (UmT5 text embeddings) via cross-attention to improve global consistency.

4. Training:

• Progressive two-stage training based on Wan2.1-I2V-14B pretrained weights.
• Conducted on 16 NVIDIA A100 GPUs with 20 denoising steps at inference and class-guidance scale 6.0.
• Hyperparameters like mask usage, semantic tokens, and reference image injection are ablated.

5. Evaluation:

• Quantitative metrics include VFID (video FID) with I3D and ResNext backbones, SSIM, LPIPS, and VBench metrics specifically created for CaM-VVT.
• Baselines compared include GAN/diffusion video try-on methods (Magic-Tryon, CatV2TON, DreamVVT) and camera control video-to-video approaches (TrajectoryCrafter, ReCaMaster).
• Also tested ablations removing major components such as the avatar, point cloud, anchor alignment, rendered prior, semantic cues, and CRA module to measure structural degradation.

6. Reproducibility:

• The new CaM-VVTBench dataset and evaluation protocol are introduced but it is unclear if code/models are publicly released.
• Supplementary material details sampling and training specifics, but no explicit link to open-source code.

Example: For a given source video, dense depth is reconstructed per frame to generate a point cloud separated into human and background. An optimal keyframe is selected to synthesize a 2D target try-on reference image, from which a canonical 3D Gaussian avatar is distilled. SMPL-X pose sequences extracted from each frame are aligned via a confidence-weighted similarity transform to the global point cloud to ensure metric consistency. The avatar is animated via Linear Blend Skinning and combined with the background to render a 4D proxy video along a desired camera trajectory. This rendered sequence is fed along with semantic and cross-view reference features into the Proxy-Anchored Video Diffusion Transformer, which generates the final video preserving fine-grained texture and motion aligned with the new viewpoint.

Technical innovations

• Renderable 4D Try-on Proxy combining a canonical 3D Gaussian Splatting avatar with metric-aligned SMPL-X animated pose sequences explicitly decouples human and background for robust geometric anchoring.
• Anchor-based alignment algorithm bridges SMPL-X poses from camera coordinates to world space point clouds using confidence-weighted similarity transforms to resolve scale and coordinate discrepancies.
• Proxy-Anchored Video Diffusion Transformer conditions generation on rendered 4D proxy priors and cross-view reference features via a novel Cross-view Reference Adapter module integrating dual-domain attention.
• Multi-modal semantic conditioning injecting CLIP image and UmT5 text embeddings enhances texture fidelity and global consistency across viewpoint changes.
• Unified end-to-end framework for CaM-VVT synthesizes realistic try-on videos with arbitrary camera trajectories, outperforming cascaded two-stage pipelines that accumulate errors.

Datasets

• ViViD — 7,759 training samples + 180 test samples (paired) — public benchmark for Video Virtual Try-On
• CaM-VVTBench — approximately 60K training video clips and 96 test samples with six standardized camera motions — collected from online sources, details not stated if public

Baselines vs proposed

• DreamVVT: VFIDpI = 11.018 vs TryOnCrafter: 9.6085
• Magic-Tryon: VFIDuI = 17.571 vs TryOnCrafter: 10.756
• Magic-Tryon+TrajectoryCrafter Overall Score = 71.64 vs TryOnCrafter Overall Score = 75.47 on CaM-VVTBench
• TryOnCrafter w/o rendered proxy: VFIDpI = 16.46 vs full TryOnCrafter = 9.61 (ablation)
• TryOnCrafter w/o garment semantics: VFIDuI = 18.51 vs full TryOnCrafter = 10.76 (ablation)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26092.

Fig 1: Examples synthesized by TryOnCrafter. We introduce a Renderable 4D Try-on

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The method relies on accurate monocular depth and SMPL-X pose estimation, which can be noisy and cause misalignments; failure cases occur when perspective-induced parallax or pose inaccuracies happen.
• CaM-VVTBench is relatively small (96 test samples) for rigorous generalization tests, and it is unclear if broader diverse real-world videos are fully represented.
• The framework’s heavy computational requirements and reliance on multi-stage complex processing may hinder real-time application deployment.
• No explicit adversarial or robustness evaluation to perturbations or occlusions is provided.
• The paper does not clarify code or training data public release, which could limit reproducibility.
• The approach implicitly assumes static background reconstruction, potentially limiting dynamic environments.

Open questions / follow-ons

• How to improve robustness to inaccuracies in monocular depth and parametric pose estimation, especially under challenging occlusions or complex scenes?
• Can the proxy and diffusion model be extended to handle dynamic backgrounds and non-rigid environmental elements for fully general scenes?
• What are the efficiency optimizations possible to enable real-time or interactive camera control during video try-on?
• How does the model perform under large changes in body shape or with complex layered clothing, which may challenge skinning and Gaussian splatting assumptions?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, TryOnCrafter introduces an advanced technique for synthesizing photorealistic videos conditioned on arbitrary camera trajectories via explicit 4D proxies and diffusion models. The core idea of grounding generation in robust geometric anchors could inspire new defenses against video forgery or impersonation attempts by bots generating realistic human avatars under viewpoint changes. Moreover, the paper’s emphasis on decoupling human geometry from background and enforcing multi-view semantic consistency could influence anti-bot detection methods relying on cross-view consistency checks. The comprehensive evaluation protocol CaM-VVTBench provides a template for testing video synthesis fidelity under controlled viewpoint perturbations, a useful asset when evaluating generative forgeries or CAPTCHA-resistance of video-based biometric verifications. However, this work does not directly address adversarial attacks or detection strategies but provides valuable insights into controlling and verifying complex video generation pipelines relevant for future bot defense systems involving virtual human renderings.

Cite

@article{arxiv2606_26092,
  title={ TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy },
  author={ Hao Sun and Hao Yan and Mengting Chen and Quanjian Song and Yu Li and Juan Cao and Jinsong Lan and Xiaoyong Zhu and Bo Zheng and Sheng Tang },
  journal={arXiv preprint arXiv:2606.26092},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26092}
}

Read the full paper

• arXiv:2606.26092 (abstract)
• arXiv:2606.26092 (PDF)