OASIS: Occlusion-aware Single-image Hand Avatar Reconstruction via 3D Gaussian Splatting

Source: arXiv:2607.29633 · Published 2026-07-31 · By Zhisheng Han, Shiyao Wu, Jiayan Qiu, Yakun Ju, Lu Liu, Le Zhang et al.

TL;DR

OASIS addresses the challenging problem of reconstructing animatable 3D hand avatars from a single RGB image, where severe self-occlusion and highly articulated pose-dependent deformation make the task ill-posed. Prior approaches largely rely on implicit neural volumetric (NeRF-style) representations that are computationally expensive and struggle to faithfully reconstruct fine geometric and texture details. OASIS introduces a novel pipeline based on 3D Gaussian Splatting (3DGS), which enables efficient real-time rendering and high-fidelity reconstruction. Key innovations include geometry-aligned visual evidence tokens that explicitly link 2D image features with 3D hand geometry, and a visibility-conditioned point-image attention mechanism which modulates feature fusion depending on occlusion visibility to produce occlusion-aware Gaussian features. Additionally, a Feature-on-Mesh representation is proposed to model local non-rigid deformation by encoding features at mesh-face level, better capturing surface stretching under articulation. OASIS employs a one-shot adaptation scheme that learns a shared hand prior from multi-identity training data and quickly fine-tunes it on a target image to personalize appearance and details.

Extensive quantitative and qualitative experiments on the InterHand2.6M dataset and in-the-wild images demonstrate that OASIS significantly outperforms previous single-image hand reconstruction methods in visual fidelity (PSNR, SSIM, LPIPS) while being orders of magnitude faster to adapt (∼5 mins vs. ∼1 hour for closest competitor) and providing real-time rendering at ∼390 FPS. It also shows robustness to challenging poses and occlusion, and versatility in downstream tasks such as text-to-avatar generation and texture editing.

Key findings

• OASIS achieves PSNR of 27.38, SSIM of 0.956, and LPIPS* of 11.45 on InterHand2.6M single-image reconstruction, outperforming prior one-shot methods like OHTA (PSNR 26.11, SSIM 0.864, LPIPS* 12.93).
• Without one-shot fine-tuning, OASIS prior network achieves PSNR 26.48, SSIM 0.950, showing strong prior generalization.
• One-shot adaptation converges in approximately 5 minutes (100 steps for color calibration + 400 steps for texture learning), an order of magnitude faster than OHTA (~1 hour).
• The visibility-conditioned point-image attention reliably shifts reliance from local image cues to global context for occluded points, improving robustness under severe self-occlusion.
• Feature-on-Mesh representation capturing face-level features tied to local surface deformation leads to smoother pose-dependent texture and geometry reconstruction compared to vertex-only representations.
• Rendering efficiency is greatly improved, with OASIS delivering ∼390 FPS on NVIDIA A100 GPU, enabling real-time interactive applications.
• Qualitative results show superior preservation of fine details like nails and complex textures under challenging hand poses and extreme viewpoints compared to NeRF-based methods.
• OASIS generalizes well to in-the-wild datasets such as HanCo, COCO-Hand, and WHIM, demonstrating robustness to real-world imagery.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly modeled since this is a reconstruction-oriented computer vision paper. The focus is on reconstructing high-fidelity hand avatars from single monocular RGB images given estimated 3D hand pose and shape priors (from MANO-HD). The method assumes access to an off-the-shelf hand pose estimator and camera parameters but copes with severe occlusions and ambiguous visual evidence.

2. Data: The training is performed on the InterHand2.6M dataset comprising over 21 subjects with multi-view RGB images, annotations, and high-resolution hand models. The data split follows official training and testing partitions. In-the-wild datasets HanCo, COCO-Hand, and WHIM are used for qualitative evaluation. Images are preprocessed to extract dense backbone features (DINOv2) and 3D hand parameters using established estimators.

3. Architecture / Algorithm:

• Input image features (DINOv2) are projected onto 3D hand geometry points derived from MANO-HD via linear blend skinning (LBS) based on estimated pose.
• Local image features around projected points are aggregated using spatial weighted pooling to form point-aligned local visual observations.
• A learnable tokenization module compresses redundant point features into a compact set of Visual Evidence Tokens (VETs) via cross-attention with learnable queries.
• Visibility for each 3D point is estimated by comparing its depth to the rendered z-buffer of the posed hand mesh, producing a soft continuous visibility score.
• Visibility-conditioned point-image attention selectively fuses local VETs and a global appearance token; visible points attend more to local cues, occluded points rely more on global context.
• Self-attention among occlusion-aware Gaussian features facilitates contextual propagation and coherence.
• Feature-on-Mesh representation interpolates vertex features to mesh face level using learnable barycentric coordinates, enabling Gaussian attributes to deform with surface stretching under non-rigid articulation.
• Gaussian attributes (position offset, rotation, scale, color, opacity) are decoded from these features via MLPs, defining the explicit 3D Gaussian Splatting representation.
• The 3D Gaussians are animated to target hand poses via LBS for rendering.

4. Training Regime:

• The prior network is trained on multiple identities from InterHand2.6M using Adam optimizer on two NVIDIA A100 GPUs.
• Loss terms include reconstruction, regularization, and appearance consistency with weights detailed in appendix.
• One-shot fine-tuning for a target image comprises two stages: color calibration (100 steps) and texture refinement (400 steps) using LoRA-based adaptation.

5. Evaluation Protocol:

• Metrics: PSNR, SSIM, LPIPS* (multiplied by 100) on synthesized novel views.
• Baselines: Compared quantitatively to state-of-the-art single-image and multi-image methods like HandAvatar, OHTA, LHM.
• Ablations demonstrate contributions of visibility-conditioned attention, Feature-on-Mesh, and tokenization.
• Qualitative evaluation includes occlusion-challenging scenarios and in-the-wild images.

6. Reproducibility:

• The authors plan to release code on the project page.
• Training datasets (InterHand2.6M) are public; pretrained weights and fine-tuning scripts expected.
• Some architectural details and hyperparameters are provided, but full details on random seeds or exact training schedule are deferred to appendix.

Example End-to-End: Given an input RGB image, the method estimates 3D pose parameters, projects MANO-HD geometry to image plane, samples local image features using DINOv2 backbone, aggregates these into compact visual tokens, estimates each point's visibility from z-buffer, performs visibility-conditioned cross-attention to fuse image cues, refines via self-attention, lifts features onto mesh faces, decodes Gaussian parameters, and generates a 3D Gaussian Splatting avatar that can be posed and rendered in real-time.

Technical innovations

• Geometry-aligned visual evidence tokens that explicitly aggregate local image features onto 3D hand points aligned by MANO-HD geometry, improving fine-grained appearance encoding under sparse single-view evidence.
• Visibility-conditioned point-image cross-attention mechanism that modulates attention weights between local and global image cues depending on estimated point visibility, robustly addressing severe self-occlusion.
• Feature-on-Mesh representation that lifts latent features to mesh face level via learnable barycentric interpolation, enabling Gaussian splatting parameters to deform coherently with local surface stretching in articulated hands.
• One-shot adaptation scheme with LoRA-based fine-tuning that efficiently personalizes a shared multi-identity 3D Gaussian prior model to a single input image, enabling quick convergence (~5 mins) while preserving unseen region priors.

Datasets

• InterHand2.6M — 2.6 million annotated hand images — public research dataset
• HanCo — size not specified — in-the-wild hand dataset
• COCO-Hand — unspecified subset of COCO with hand annotations
• WHIM — wild hand images dataset with diverse poses

Baselines vs proposed

• HandAvatar (single-image): PSNR=23.79, SSIM=0.820, LPIPS*=17.78 vs OASIS: PSNR=27.38, SSIM=0.956, LPIPS*=11.45
• OHTA (single-image): PSNR=26.11, SSIM=0.864, LPIPS*=12.93 vs OASIS: PSNR=27.38, SSIM=0.956, LPIPS*=11.45
• LHM (no fine-tune): PSNR=24.45, SSIM=0.901, LPIPS*=13.96 vs OASIS no-finetune: PSNR=26.48, SSIM=0.950, LPIPS*=11.74
• HandAvatar (multi-image): PSNR=28.23, SSIM=0.894, LPIPS*=10.35 vs OASIS one-shot (single-image): PSNR=27.38, SSIM=0.956, LPIPS*=11.45

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.29633.

Fig 1: We introduce OASIS, a 3DGS-based one-shot hand avatar reconstruction model with efficient adaptation (∼5 minutes)

Fig 2: Overview of the proposed OASIS. Given a single input image, we construct geometry-aligned VETs to encode image-specific appearance

Fig 3: Architecture of the proposed Visibility-conditioned Point-

Fig 4: Qualitative comparison with state-of-the-art methods on In-

Fig 5: Visual results for ablation studies of OASIS on InterHand2.6M dataset.

Fig 6: In the wild visualization from the HanCo [42] dataset, COCO-Hand [20] dataset, and WHIM [23] dataset.

Fig 7: Qualitative comparison with [41] on text-to-avatar application. The first column is the input reference images, and the rendered

Fig 8: Qualitative comparison on HanCo [42]. The white box

Limitations

• The training and evaluation rely on accurate MANO-HD estimates and camera calibration; errors there can propagate and reduce reconstruction quality.
• The proposed method requires a reasonably good initial pose and shape input from off-the-shelf estimators; fully end-to-end from image to avatar is not demonstrated.
• One-shot adaptation is efficient but still requires GPU time (~5 minutes on A100), which may be impractical on low-resource devices or real-time applications without fine-tuning.
• Evaluation mostly focuses on standard datasets and does not extensively test extreme occlusions beyond dataset distributions or adversarial occlusion scenarios.
• Some architectural design choices, such as the number of tokens or hyperparameters controlling visibility-conditioned attention, require tuning and may not generalize without retraining.
• The approach currently models hands only, with assumptions on hand topology limiting direct extension to other highly deformable objects without redesign.

Open questions / follow-ons

• Can the visibility-conditioned attention scheme be generalized to multi-object scenarios or full-body avatars with complex occlusion interplay?
• How robust is the method to inaccurate 3D prior pose estimates or camera parameters, and can these be jointly optimized within the pipeline?
• Could the one-shot adaptation be sped up to real-time speeds using more efficient optimization or distillation techniques?
• How well can the Feature-on-Mesh representation model fine-grained skin deformations or non-rigid surface details beyond mesh stretching?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners interested in bot detection via 3D hand reconstruction or biometric gesture recognition, OASIS provides a state-of-the-art methodology to generate high-fidelity animatable 3D hand avatars from a single image with real-time rendering capability. The visibility-conditioned attention and geometry-aligned visual tokens afford robustness under severe self-occlusion, a common challenge in hand gesture capture. Its one-shot fine-tuning allows rapid personalization to unknown hands, which could help enhance the realism and naturalness of simulated hand avatars used in CAPTCHA challenges or to benchmark anti-spoofing systems. However, practical deployment would require integration with fast pose estimators and possibly further optimizations for low-powered devices. The advanced geometric-feature coupling suggests new directions for improving hand pose and appearance consistency in hand interaction verification tasks.

Cite

@article{arxiv2607_29633,
  title={ OASIS: Occlusion-aware Single-image Hand Avatar Reconstruction via 3D Gaussian Splatting },
  author={ Zhisheng Han and Shiyao Wu and Jiayan Qiu and Yakun Ju and Lu Liu and Le Zhang and Pengfei Feng and Huiyu Zhou and Zheheng Jiang },
  journal={arXiv preprint arXiv:2607.29633},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.29633}
}

Read the full paper

• arXiv:2607.29633 (abstract)
• arXiv:2607.29633 (PDF)