ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

Source: arXiv:2607.06555 · Published 2026-07-07 · By Ruihang Zhang, Felix Taubner, Pooja Ravi, Kiriakos N. Kutulakos, David B. Lindell

TL;DR

ProxyPose addresses the long-standing challenge of tracking 6-DoF pose of objects and surfaces from monocular video without relying on explicit 3D models, depth sensors, or object segmentation masks. Unlike prior methods that require additional inputs or struggle on textureless, transparent, reflective, or deformable surfaces, ProxyPose reformulates pose tracking as a video-to-video translation task. Given a monocular video and a single marked pixel in the first frame, a fine-tuned video diffusion model synthesizes a proxy video depicting a colored, rigid polyhedron undergoing the same local 3D motion as the surface region corresponding to that pixel. The proxy frames have known geometry and appearance, enabling classical pose estimation solvers to recover the full 6-DoF trajectory efficiently. This use of large-scale pre-trained video diffusion models allows ProxyPose to implicitly encode complex motion and material properties, handling occlusions and non-rigid motion without task-specific supervision or explicit 3D reconstruction.

Empirically, ProxyPose achieves state-of-the-art pose tracking accuracy on standard benchmarks including HO3D and YCBInEOAT, dramatically reducing rotation error (e.g., ARE of 5.1 deg vs 14-75 deg for baselines on HO3D) and translation error (ATE 15.8 mm vs ≥25 mm) while requiring only monocular RGB input. It outperforms prior model-based and model-free methods that depend on depth or 3D models, and is robust to challenging in-the-wild scenes such as specular, transparent, deforming, or occluded regions. ProxyPose’s design also extends naturally to face tracking, camera pose estimation, and novel imaging modalities (event/single-photon cameras), highlighting its generality as a 3D motion backbone.

Key findings

• ProxyPose (one query pixel) achieves absolute translation error (ATE) of 15.79 mm and absolute rotation error (ARE) of 5.13 degrees on HO3D, outperforming FoundationPose tracking mode (ATE 25.44 mm, ARE 15.38 deg) and other baselines.
• On YCBInEOAT, ProxyPose achieves ATE of 30.07 mm and ARE of 15.07 deg without requiring depth or 3D models, compared to >40 mm and >17 deg for most baselines.
• Multi-query bundle adjustment further improves accuracy and temporal smoothness on rigid objects by fusing poses from multiple proxy videos corresponding to several query pixels.
• ProxyPose requires only monocular RGB input and a single marked point in the first frame, whereas all baselines require additional inputs such as 3D models, depth maps, or object masks.
• Fine-tuning on a synthetic dataset of 35,000 paired videos suffices to adapt a pretrained video diffusion model, with strong performance even at 300 training samples.
• Proxy video generation using a noise schedule offset improves first-frame fidelity and overall stability compared to naive noise scheduling.
• Qualitative experiments demonstrate ProxyPose tracking non-rigid, shiny, transparent, deforming surfaces and successfully handling occlusions and rapid motion where existing methods fail.
• ProxyPose recovers temporally consistent 6-DoF trajectories suitable for downstream tasks such as camera pose estimation and face tracking in wild videos.

Threat model

ProxyPose does not explicitly consider an adversarial threat model. The system assumes a benign setting where a single monocular RGB video and a user-selected pixel in the first frame are provided. The method does not rely on secret or trusted hardware, nor does it claim security guarantees against adversarial manipulation or spoofing. It focuses on accurate 6-DoF pose tracking of real-world objects and surfaces under challenging visual conditions.

Methodology — deep read

ProxyPose is formulated as a two-stage pipeline involving video-to-video translation followed by classical pose estimation.

1. Threat Model & Assumptions: The adversary is not explicitly modeled, but the method assumes a monocular RGB video input with a single user-selected pixel in the first frame whose local 3D motion trajectory is to be tracked. The camera's focal length is known or can be approximated. ProxyPose does not require prior 3D object models or segmentation masks. It assumes local rigidity around the tracked pixel but can handle non-rigid deformations approximately.

2. Data: The training data is synthetic—35,000 paired video sequences rendered from diverse 3D assets (Trellis-500K from Objaverse). For each pair, the source video shows a cluttered scene with the object(s) undergoing 6-DoF motion; a corresponding proxy video depicts a colored cube undergoing the exact rigid motion of the tracked object region. Proxy videos have known geometry and appearance.

3. Architecture & Algorithm: ProxyPose fine-tunes a large pretrained video diffusion transformer model (Wan-14B backbone) with low-rank adaptation (LoRA) applied to attention and feedforward layers. The input is an RGB source video and a marked pixel location. The model encodes both the source video and the proxy video latent space into spatio-temporal tokens, concatenated for joint self-attention using a 4D rotary positional embedding to distinguish streams. The diffusion model is trained to translate the source video to the proxy video by denoising the corrupted proxy tokens conditioned on the clean source tokens. A noise schedule offset preserves first-frame fidelity.

During inference, given the input video and query pixel, the fine-tuned diffusion model generates a proxy video showing a colored cube undergoing the same 3D motion locally around the pixel. The proxy object has 6 distinct colored faces, edges, and known 3D model geometry.

4. Training Regime: Fine-tuning proceeds in three stages over about 100k iterations: first 80k iterations at 256x256 resolution (29 frames), then 10k at 512x512 (29 frames), then 10k at 512x512 (49 frames). The LoRA rank is 64. Optimizer hyperparameters include learning rates between 2×10⁻⁴ and 5×10⁻⁵, and a noise schedule offset of 500 steps. Training is conducted on 4 Nvidia H100 GPUs totaling ~22 GPU days.

5. Evaluation Protocol: ProxyPose is evaluated on the HO3D and YCBInEOAT benchmarks with ground-truth 6-DoF pose annotations. Metrics include absolute translation error (ATE), absolute rotation error (ARE), relative pose errors (RPE-t, RPE-r), and 2D reprojection distance (2D-dist). Baselines use the same inputs where applicable. Additionally, a synthetic holdout set and challenging in-the-wild videos test robustness. Ablations study LoRA rank, dataset size, and noise schedule offset.

6. Reproducibility: The paper states code, models, and video results are available on the project webpage. The synthetic dataset generation protocol is described but the dataset is presumably not fully public. The fine-tuning details are thorough, using publicly known assets and environments.

End to end example: Given a monocular video and a marked pixel in frame 1, ProxyPose initializes a colored proxy cube by rendering it aligned with the camera ray through that pixel, occupying a fixed fraction of the frame. The diffusion model translates the source video latents to proxy video latents, generating proxy frames. OpenCV-based segmentation of cube faces and vertex localization from the proxy frames produce 2D-3D correspondences. Solving Perspective-n-Point (PnP) yields a pose estimate for each frame, optionally refined with temporal smoothness and multi-query bundle adjustment for rigid surfaces.

Technical innovations

• Recasting 6-DoF pose tracking as a video-to-video translation task using a fine-tuned video diffusion model to generate a synthetic proxy video of a colored polyhedron matching local object motion.
• Introducing a joint self-attention architecture over concatenated source video and proxy video token streams with 4D rotary positional embeddings encoding stream identity to enable long-range correspondence learning for motion transfer.
• Applying a noise schedule offset to stabilize first-frame proxy video generation and maintain identity, scale, and orientation consistency during diffusion training.
• Fusing multiple proxy videos from several query pixels with bundle adjustment incorporating per-proxy depth scalars to enforce multi-point rigidity constraints without explicit 3D model access.

Datasets

• Synthetic paired (source, proxy) video dataset — 35,000 sequences — rendered using Blender and PyTorch3D with Trellis-500K 3D assets from Objaverse
• HO3D — 13 sequences — hand-object manipulation benchmark
• YCBInEOAT — 9 sequences — dual-arm robotic manipulation benchmark

Baselines vs proposed

• FoundationPose (track mode, 3D model + depth): HO3D ATE=25.44mm, ARE=15.38deg vs ProxyPose (one query) ATE=15.79mm, ARE=5.13deg
• BundleSDF (3D model + depth): HO3D ATE=23.24mm, ARE=14.28deg vs ProxyPose (one query) ATE=15.79mm, ARE=5.13deg
• CoTracker 3 + Depth Anything 3 (pixel level, 3D model + depth): HO3D ATE=26.62mm, ARE=17.85deg vs ProxyPose ATE=15.79mm, ARE=5.13deg
• ProxyPose (two queries) with bundle adjustment further reduces ARE to 3.94 deg on HO3D
• On YCBInEOAT proxy Pose (one query) achieves ATE=30.07mm, ARE=15.07deg < baselines which have ATE ≥40 mm and ARE ≥17 deg

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.06555.

Fig 1: Our approach enables tracking relative 6-DoF pose in diverse, highly dynamic scenes

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• ProxyPose's video diffusion backbone limits the number of frames processed in a single pass to the model's generation length, restricting long-term tracking.
• Absolute global pose cannot be recovered from monocular tracking without additional constraints; only relative motion is estimated by design.
• Fast motion can degrade proxy video quality due to VAE encoding blur, causing pose estimation errors.
• Tracking drift arises on textureless or highly reflective surfaces and for deforming fluid surfaces where local rigidity assumptions break down.
• Inference is computationally expensive, requiring multiple minutes on a NVIDIA H100 GPU, limiting real-time applicability currently.
• While proxy videos faithfully track local rigid motion, non-rigid deformations are only approximately handled and could degrade accuracy.

Open questions / follow-ons

• Can the proxy video generation approach be extended to handle absolute pose estimation by integrating additional constraints or multi-view inputs?
• How can efficient autoregressive or compressed video diffusion models be employed to enable real-time 6-DoF tracking with ProxyPose?
• What are the limits of proxy-based representations for highly non-rigid or fluid deformable surfaces in complex scenes?
• Could the framework be generalized to articulated body tracking or dense scene flow by designing proxy objects with more complex geometries or motion priors?

Why it matters for bot defense

ProxyPose introduces a novel paradigm of leveraging large-scale video generation models for pixel-level 6-DoF pose tracking purely from monocular RGB input. For bot-defense and CAPTCHA systems that aim to distinguish human vs automated interactions based on subtle physical object manipulation or scene understanding, ProxyPose offers a potential route for robust 3D tracking without complicated sensors or pre-built 3D models. Its ability to handle textureless, shiny, transparent, and deforming surfaces is particularly relevant for real-world applications where straightforward visual cues are limited.

However, the current inference latency and requirement for high-end GPUs may restrict on-device or real-time deployment in CAPTCHA contexts. Also, potential drift and failure modes on non-rigid or fast-moving targets indicate that ProxyPose would likely serve best as a backend verification module rather than a front-line bot detector. Integrating ProxyPose with other modalities or efficient models could improve its applicability to user interaction verification and manipulation behavior analysis.

Cite

@article{arxiv2607_06555,
  title={ ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation },
  author={ Ruihang Zhang and Felix Taubner and Pooja Ravi and Kiriakos N. Kutulakos and David B. Lindell },
  journal={arXiv preprint arXiv:2607.06555},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.06555}
}

Read the full paper

• arXiv:2607.06555 (abstract)
• arXiv:2607.06555 (PDF)