Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

Source: arXiv:2606.30598 · Published 2026-06-29 · By Siddhant Bansal, Zhifan Zhu, Shashank Tripathi, Jiahe Zhao, Michael J. Black, Dima Damen

TL;DR

This paper tackles the difficult problem of 3D hand-object pose estimation from challenging in-the-wild egocentric RGB videos, where the hands and objects are heavily occluded and contact regions are ambiguous. Existing methods trained on controlled lab datasets do not generalize well to such real-world settings due to limited diversity and scarce supervision. To address this, the authors contribute two key advances: (1) EPIC-Contact, the first large-scale in-the-wild egocentric dataset with 2.3K video clips (62.3K frames) densely annotated with bijective 3D hand-object contact correspondences and posed hand-object meshes. This dataset enables training on diverse, cluttered, and occluded scenes and provides a unique resource for model development and evaluation. (2) HOPformer, a novel end-to-end transformer model that jointly estimates bi-manual hand pose and object pose from a single RGB image in one forward pass. HOPformer uses a multi-layer cross-attention decoder that conditions object features on strong hand pose priors, allowing the model to reason about hand-object interaction explicitly and robustly. When evaluated on the in-lab ARCTIC dataset, HOPformer improves the success rate by 6.2 percentage points (82.4% vs prior state-of-the-art), and on the much more challenging EPIC-Contact dataset, it nearly doubles the success rate of existing methods while reducing contact deviation error by 75%. The authors release their dataset, code, and trained models to enable future research. Together, these contributions represent a significant advance towards practical, scalable, and accurate 3D hand-object pose estimation in-the-wild under real interaction scenarios.

Key findings

• EPIC-Contact dataset: 2,272 video clips, 62.3K frames, across 9 object classes, annotated with dense bijective hand-object 3D contact correspondences and posed meshes.
• Inter-annotator agreement (Fleiss' Kappa) for hand contact annotation κh=0.61 and object contact κo=0.62 on 10 video samples, indicating consistent labeling.
• HOPformer outperforms the previous best JointTransformer baseline on ARCTIC egocentric split: success rate ([email protected]) improves from 76.2% to 82.4%, contact deviation (CDev) drops from 35.0 mm to 31.9 mm, motion deviation (MDev) from 10.4 mm to 7.3 mm, and MPJPE reduces from 20.0 mm to 16.1 mm.
• On the in-the-wild EPIC-Contact set, HOPformer nearly doubles success rate [email protected] from 17.6% to 29.8%, reduces contact deviation from 30.1 mm to 20.7 mm, and motion deviation from 20.0 mm to 11.4 mm compared to JointTransformer.
• Scaling object mesh sizes with Vision Language Model (Gemini 2.5) yields 0.94 cm mean absolute error (5.9% relative error) for object dimensions compared to ground truth, validating scale estimation.
• Cross-attention decoder conditioning object features on hand priors is critical: removing it reduces [email protected] on ARCTIC from 82.4% to 29.4%.
• Removing training losses related to object pose and interaction constraints significantly degrades performance; interaction loss removal reduces success rate to 79.2%.
• HOPformer uses DINOv2 ViT with WiLoR hand features; alternative choices (e.g., using DINOv2 as hand features) reduce performance substantially.

Threat model

n/a – This paper addresses robustness and generalization challenges in in-the-wild 3D hand-object pose estimation from monocular egocentric RGB images, focusing on handling occlusion and ambiguity from natural scene distribution rather than adversarial threats.

Methodology — deep read

The authors consider the problem of estimating 3D poses of both hands (bi-manual) and a manipulated object from a single egocentric RGB image, focusing on in-the-wild scenarios characterized by occlusions, clutter, and ambiguous hand-object contact.

The threat model is not adversarial but addresses inherent challenges in naturalistic data distribution shifts. The goal is robust generalization from limited labeled data.

The core data resource is EPIC-Contact, derived from 2,272 clips in the EPIC-Kitchens dataset, manually annotated through a novel annotation pipeline: annotators paint contact vertices on a finely subdivided MANO hand mesh (3106 vertices) across stable grasp video clips, marking contact regions per hand side with temporal context. Subsequently, bijective contact correspondences on objects are transferred using a parametrized contact axis approach (2-DoF axis), requiring few clicks and preserving vertex mappings.

The object meshes are scaled per instance using Gemini 2.5 Vision Language Model prompts to estimate non-uniform scaling parameters, validated against physical measurements, achieving ~0.94 cm error.

Posed hand-object meshes are fit via an optimization pipeline (EC-fit) that initializes hand poses with WiLoR predictions, optimizes object poses to minimize contact vertex distances, and refines both hand and object poses using occlusion-aware mask losses and penetration penalties for physical plausibility. The posed meshes are manually verified and adjusted.

HOPformer architecture inputs an RGB image and extracts features via two pretrained encoders: DINOv2 ViT-G for objects (Φo) and WiLoR for hand features (Φh). These produce sequences of object and hand tokens. A multi-layer (L=12) transformer decoder with cross-attention uses hand token memory to condition object token queries progressively. The self-attention and cross-attention layers allow the refinement of object features informed by the hand context.

The decoder outputs aggregated interaction tokens passed to independent lightweight MLP heads that regress left and right hand MANO parameters (pose θ and shape β), weak perspective camera parameters, and object pose parameters (rotation in 6D, translation in 3D, articulation). An object classification head predicts object category for mesh retrieval from a model pool.

Training uses a composite loss combining supervised 3D/2D keypoint reprojection, pose parameter regression, camera parameter loss, classification loss, and an interaction contact deviation loss to encourage geometric congruity at contacts. Loss weights are tuned empirically. Training is done in two stages: first on exocentric ARCTIC views, then fine-tuned on egocentric ARCTIC and EPIC-Contact views.

Evaluation metrics include contact deviation (vertex distance), relative root position errors, motion deviation across stable grasp frames, acceleration error for smoothness, mean per-joint position error, articulation error, and success rates measuring fraction of correctly reconstructed vertices within a tolerance of object diameter.

Baselines ArcticNet-SF and JointTransformer are re-trained for fair comparison on both datasets. Ablations systematically remove losses, architectural components (e.g., cross-attention), or alternative features to validate design choices.

The entire pipeline from annotation to model training to evaluation is detailed, with code, data, and model weights released for reproducibility. Limitations include manual verification steps and some failure cases under extreme occlusion or surprising object forms.

Technical innovations

• EPIC-Contact: a large-scale, manually annotated, in-the-wild egocentric dataset with dense bijective 3D hand-object contact correspondences enabling posed mesh supervision.
• Bijective contact annotation pipeline using vertex painting on a high-resolution MANO hand mesh combined with a parametrized 2-DoF contact axis for efficient hand-object contact transfer on object meshes.
• EC-fit: an optimization pipeline utilizing contact correspondences, occlusion-aware image-mask losses, and penetration penalties to derive accurate 3D posed hand-object meshes from contact annotations.
• HOPformer: an end-to-end transformer model that uses multi-layer cross-attention decoder to condition object features on explicit hand pose priors for improved joint bi-manual hand-object pose estimation in a single forward pass.
• Incorporation of Vision Language Model (Gemini 2.5) for class-specific non-uniform object mesh scaling in the wild.

Datasets

• EPIC-Contact — 2,272 video clips, 62.3K frames — Derived and annotated from EPIC-Kitchens dataset, public release planned
• ARCTIC — 2.1M RGB frames across 9 views, 10 participants, 11 objects — Public lab dataset with MoCap ground-truth

Baselines vs proposed

• ArcticNet-SF: [email protected] = 59.0% vs HOPformer: 82.4% on ARCTIC egocentric
• JointTransformer: CDev = 35.0 mm vs HOPformer: 31.9 mm on ARCTIC egocentric
• JointTransformer: [email protected] = 17.6% vs HOPformer: 29.8% on EPIC-Contact in-the-wild
• ArcticNet-SF: CDev = 94.2 mm vs HOPformer: 20.7 mm on EPIC-Contact

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30598.

Fig 1: (Left) We introduce EPIC-Contact, an in-the-wild egocentric dataset for 3D

Fig 2: EPIC-Contact annotation process. Given a hand–object interaction clip,

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• EPIC-Contact annotations require manual verification and corrections, which may limit scalability.
• Annotation and pose fitting rely on accurate object mesh scaling via Vision Language Models; inaccuracies here degrade results.
• HOPformer requires known object categories for mesh retrieval; CAD-free objects or unknown categories remain unaddressed.
• No explicit adversarial robustness or robustness to extreme occlusion beyond dataset distribution is tested.
• Pose estimation is limited to stable grasp frames; highly dynamic or transient contacts are not comprehensively annotated.
• Failure cases remain for objects with shapes or articulations that conflict with strong priors learned in training.

Open questions / follow-ons

• How to extend joint hand-object pose estimation to unknown or deformable object categories without pre-defined meshes?
• Can richer temporal dynamics and motion cues be exploited to improve stability and accuracy for transient or highly dynamic hand-object interactions?
• How to further automate or scale the bijective contact annotation process, possibly with semi-supervised or self-supervised techniques?
• What are the limits of generalization to radically novel object categories or extreme occlusions beyond current in-the-wild datasets?

Why it matters for bot defense

This work advances accurate 3D hand-object pose estimation from challenging, occluded egocentric video in the wild, which is relevant to bot defense systems relying on in-device camera inputs to verify human manipulation and interaction plausibility. Understanding nuanced hand-object contact and pose dynamics can inform sophisticated behavioral biometrics or liveness analyses that distinguish genuine human activity from synthetic or replayed inputs. The EPIC-Contact dataset facilitates training more robust vision models under occlusion and clutter, while HOPformer's architecture highlights the benefit of conditioning object pose estimations on reliable hand priors, a technique that could inspire improvements in multi-modal fusion for CAPTCHA verification tasks involving physical interactions. However, practical deployment would need to address limitations such as known object meshes, computational complexity of transformer models, and failure modes in extreme occlusions.

Cite

@article{arxiv2606_30598,
  title={ Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation },
  author={ Siddhant Bansal and Zhifan Zhu and Shashank Tripathi and Jiahe Zhao and Michael J. Black and Dima Damen },
  journal={arXiv preprint arXiv:2606.30598},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30598}
}

Read the full paper

• arXiv:2606.30598 (abstract)
• arXiv:2606.30598 (PDF)