Pose Anything Anywhere:Model-free Object Poses from Arbitrary References

Source: arXiv:2606.23634 · Published 2026-06-22 · By Hongli Xu, Jiaqi Hu, Junwen Huang, Boyang Zhong, Peter KT Yu, Nassir Navab et al.

TL;DR

This paper addresses the challenging task of estimating 6D poses of unseen objects in open-world settings without relying on CAD models or heavy onboarding. Prior model-free methods mostly rely on pairwise single-reference matching, which struggles under occlusion, wide baselines, and limited query-reference overlap. PANY is proposed as a unified model-free framework that supports both RGB and RGB-D inputs and can operate on either one or multiple sparse, pose-free reference views. PANY harnesses a multi-view transformer-based geometry backbone with geometry-aware, spatially consistent cross-view correspondences, enabling robust pose estimation under challenging conditions such as large viewpoint changes and occlusions. When available, additional unposed assist views are aggregated via pose-graph canonical registration to increase geometric coverage and refine poses further. Extensive evaluation on multiple benchmark datasets including YCB-Video, LINEMOD, LM-O, Real275, and Toyota-Light demonstrates state-of-the-art performance with substantial improvements over prior model-free methods (e.g., +12% on YCB-V and +20% on LM-O pose accuracy). The approach generalizes well across modalities and sparse image collections, showing robustness for real-world robotics and embodied perception applications.

Key findings

• PANY improves pose accuracy by +12% ADD AUC on YCB-Video dataset over prior model-free approaches.
• On the LM-O dataset, PANY achieves +20% AR (42.3%) using 1 posed anchor + 8 unposed assist views, nearly doubling accuracy of single-reference methods like One2Any (18.2%).
• Using RGB-D inputs, PANY achieves 81.8% AR and 89.3% ADD-0.1d on Real275 dataset, outperforming Oryon RGB-D baseline (46.5% AR).
• PANY consistently performs robustly under single-reference and sparse multi-reference settings across benchmarks with wide viewpoint changes and occlusion.
• Ablation shows that joint geometry and cross-view correspondence learning yields up to 17.8% ADD-0.1d improvement over geometry-only prediction.
• Pose-graph aggregation over unposed assist views increases geometric coverage and improves pose estimation when single references have limited overlap.
• Evaluations demonstrate low sensitivity to anchor view selection on LM-O dataset, confirming stable inference.
• PANY’s model trains on synthetic Omni6DPose dataset with over 2 million RGB-D samples covering 149 categories, ensuring generalization to real-world unseen objects.

Threat model

The threat model assumes the adversary cannot provide CAD models or explicit geometric onboarding data and has limited control over the reference views, which are assumed sparse, possibly unposed, and may contain occlusions or wide viewpoint shifts. The adversary cannot manipulate the pose graph construction or latent transformer representations directly. The goal is to robustly estimate 6D poses despite minimal assumptions on the availability or quality of reference images, under realistic open-world conditions.

Methodology — deep read

The threat model is an open-world scenario where the adversary has no access to CAD models or dense posed images; the system must estimate 6D object pose from sparse, arbitrary reference images, some posed (anchors) and many unposed (assist views). The adversary may attempt to supply views with minimal overlap or occlusion, but cannot provide CAD models or ground-truth poses for assist views.

Training data is the large-scale synthetic Omni6DPose dataset containing 5,000+ CAD objects from 149 categories; over 2 million RGB-D images of object-centric crops with accurate pose labels are used. References and query images are sparsely sampled from these rendered views.

The architecture uses a multi-view transformer-based backbone (adapted from VGGT) with patch tokenization of input RGB or RGB-D images, alternating intra-view attention and global inter-view cross-attention to learn consistent multi-view representations. Visual tokens are decoded with DPT-based upsampling to predict per-view dense depth and 3D point maps.

A key contribution is a cross-view 3D matching module: predicted dense point maps (coordinates) are encoded with a geometric transformer backbone using rotation-invariant positional encodings to produce geometry-aware 3D descriptors. These descriptors are fused with lifted per-pixel visual features from the visual backbone to generate rotation-invariant, geometry-aligned tokens.

The matching head is trained using a local contrastive loss on cross-view correspondences defined via nearest neighbor in predicted shared canonical coordinate frames, supervised bidirectionally for view pairs, improving correspondence robustness under large viewpoint changes and occlusion.

Training jointly optimizes geometry and correspondence objectives with balanced L1 losses on depth and point maps plus a contrastive cross-view matching loss. The backbone is frozen and finetuned with LoRA during training.

At inference, given a query image, an anchor reference (with known canonical pose), and optional unposed assist views, the model predicts dense point clouds and descriptors, constructs a pose graph connecting views with sufficient correspondences, and estimates pairwise transformations using RANSAC and Umeyama alignment.

Global canonical alignment is performed by composing transformations along graph paths to align all views into the anchor coordinate frame. Final query pose is recovered from the estimated relative transformation between registered query and anchor point clouds. RGB-only cases use PnP optimization from 3D-2D correspondences.

Evaluation is conducted on five unseen real-world benchmarks with no overlap to the training dataset, using Average Recall (AR), ADD-0.1d, and ADD AUC metrics. Baselines include recent state-of-the-art RGB and RGB-D model-free and model-based pose estimators. Ablations isolate the benefit of matching, geometric fusion, and assist view aggregation, showing consistent improvements. The model runs efficiently using AdamW optimizer on A100 GPUs over 15 epochs with batch size 16.

One example end-to-end involves estimating 6D pose on LM-O using a single posed anchor view plus multiple unposed assist views: the model predicts dense 3D point maps for each view, generates cross-view descriptors, constructs the pose graph, estimates transformations between pairs, aligns all point clouds in anchor space, and calculates the relative pose transform for the query image robustly despite severe occlusions and viewpoint gaps.

Technical innovations

• Introduction of a model-free pose estimation framework (PANY) that leverages multi-view transformer geometry backbone to achieve object-centric canonical alignment from arbitrary sparse posed and unposed reference views, without CAD models.
• A cross-view 3D matching module producing rotation-invariant, geometry-aligned descriptors from fused geometric point maps and visual embeddings, enabling robust correspondences under wide baselines and occlusion.
• A local contrastive learning objective enforcing dense 3D correspondence consistency across wide baselines, overcoming view-dependent instability common in prior model-free methods.
• Use of a pose-graph canonical registration algorithm that integrates pose-free assist views as intermediates to increase geometric coverage and resolve pose ambiguities during inference, improving robustness when no single reference overlaps well with the query.

Datasets

• Omni6DPose — 2 million+ synthetic RGB-D images — synthetic dataset for training with 5,000+ CAD objects across 149 categories
• LINEMOD — thousands of real test images — public benchmark for 6D object pose
• YCB-Video (YCB-V) — real RGB-D video frames with object pose labels
• Toyota-Light — real RGB-D test dataset
• LM-O (LineMOD Occluded) — challenging occluded real images
• Real275 — RGB-D real images for unseen object pose estimation

Baselines vs proposed

• Oryon (RGB-D) on Real275: AR = 46.5% vs PANY (RGB-D) AR = 81.8%
• One2Any (RGB-D) on LM-O: AR = 18.2% vs PANY (1 posed + 8 unposed) AR = 42.3%
• FoundPose (model-based RGB) on LINEMOD: ADD-0.1d = 48.3% vs PANY single-ref = 55.3%
• SingRef6D (RGB) on Toyota-Light: ADD-0.1d = 13.1% vs PANY (RGB) ADD-0.1d = 10.8% (note PANY outperforms in AR but this metric is per dataset)
• Any6D (RGB-D) on Real275: AR = 51.0% vs PANY AR = 81.8%
• MegaPose (model-based) on LM-O: AR = 22.9% vs PANY (multi-ref) AR = 42.3%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.23634.

Fig 1: PANY is the first model-free approach that simultaneously handles single- and

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 2: Overview of the PANY framework. (i) Given a query image and multiple ref-

Limitations

• The model relies on synthetic training data with accurate poses, which may limit generalization to highly different real-world appearances or sensor noise.
• Performance under extreme occlusions or very sparse reference views (e.g., single reference with no assist views and nearly no overlap) remains a challenging corner case.
• The method assumes availability of at least one posed anchor reference; fully unposed multi-view pose estimation is not addressed.
• Inference runtime may be relatively high compared to lightweight pairwise models due to multi-view transformer computations and pose-graph registration.
• The robustness to adversarial or maliciously crafted views aiming to confuse correspondence matching was not evaluated.
• The approach has limited evaluation on RGB-only inputs without depth; accuracy may drop without depth sensors.

Open questions / follow-ons

• Can the cross-view correspondence learning be extended to fully unposed multi-view setups without anchor poses, enabling fully unknown coordinate frame alignment?
• How does the method perform when trained and evaluated purely on real-world images without synthetic pretraining? Is domain adaptation feasible?
• What are the limits of sparse view aggregation — how few and how unstructured can assist views be while still improving pose estimates?
• Can adversarial robustness be improved by incorporating explicit uncertainty modeling or outlier rejection in matching and pose-graph registration?

Why it matters for bot defense

From a bot-defense engineering perspective, PANY's approach to robust multi-view, model-free 6D pose estimation without reliance on precise object models or dense onboarding could inspire more adaptive visual challenge designs for CAPTCHAs. Its ability to tolerate sparse and unposed reference inputs under occlusion and viewpoint changes may inform defenses leveraging spatial consistency checks or multi-view correspondences to differentiate bots from humans. The pose-graph aggregation mechanism illustrates how multiple weak or incomplete signals can be combined to improve reliability, analogous to aggregating multi-step behavioral or biometric signals in bot detection.

However, deploying similar geometry-based pose estimation techniques in CAPTCHA settings would require consideration of computational costs and latency. Additionally, adapting the system to more varied or deformable objects commonly encountered in web contexts might be non-trivial. Overall, the paper contributes valuable insights into achieving robust visual geometric reasoning when explicit 3D models and annotations are unavailable, which aligns well with challenges in scalable, generalizable CAPTCHA design.

Cite

@article{arxiv2606_23634,
  title={ Pose Anything Anywhere:Model-free Object Poses from Arbitrary References },
  author={ Hongli Xu and Jiaqi Hu and Junwen Huang and Boyang Zhong and Peter KT Yu and Nassir Navab and Benjamin Busam and Slobodan Ilic },
  journal={arXiv preprint arXiv:2606.23634},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.23634}
}

Read the full paper

• arXiv:2606.23634 (abstract)
• arXiv:2606.23634 (PDF)