VR3D: View-Robust 3D Representation Learning for Aerial-Ground Person Re-Identification

Source: arXiv:2608.02598 · Published 2026-08-03 · By Chao Ji, Shiyu Xuan, Zechao Li

TL;DR

This paper addresses the challenging problem of aerial-ground person re-identification (ReID), where large viewpoint variations between aerial and ground cameras cause severe occlusion and geometric deformation. Existing methods attempt to learn view-invariant features solely in the 2D image space, but the authors show this approach suffers from viewpoint bias since 2D images only provide partial visible appearance information tied to specific camera angles. To overcome this, the authors propose VR3D, a View-Robust 3D Representation Learning framework that lifts features from 2D images into a unified canonical 3D coordinate space, enabling direct 2D–3D feature interaction and more view-independent identity representations. The core VR3I module fuses 2D appearance with 3D human voxel representations via a novel 3D Geometry-Semantic Attention mechanism, grounded in physical 3D spatial locations. To handle sample-specific reliability variations due to extreme views or 3D reconstruction errors, a Reliability-Aware Fusion module adaptively aggregates complementary 2D and 3D representations. Extensive experiments on three aerial-ground person ReID benchmarks (CARGO, AG-ReID.v1, AG-ReID.v2) demonstrate substantial improvements over prior state-of-the-art methods, e.g., a +5.63% Rank-1 gain on CARGO under aerial-to-ground retrieval. Ablations validate the effectiveness of the 3D semantic-geometry attention and reliability-aware fusion modules. Overall, the method provides a compelling approach to break the viewpoint bias bottleneck by learning identity representations in a physically consistent 3D space rather than purely in 2D image space.

Key findings

• VR3D improves Rank-1 accuracy by 5.63% and mAP by 4.93% over the previous best method on CARGO under the aerial-to-ground (A↔G) protocol.
• On AG-ReID.v1, VR3D achieves Rank-1 / mAP of 87.39% / 79.95% for aerial-to-ground and 89.60% / 82.87% for ground-to-aerial, improving over baselines by more than 2%.
• On AG-ReID.v2, VR3D obtains the best performance across all four protocols including aerial-to-CCTV, aerial-to-wearable, etc., with mAP improvements exceeding 2.0% and up to 5.15%.
• Ablation studies show that directly adding 3D global features to 2D baseline brings limited gains, but the proposed 3D Geometry-Semantic Attention fused feature improves mAP by 3.76% and 2.42% for A→G and G→A respectively.
• Introducing multi-feature representations (2D, fused, 3D structural) jointly further improves results, showing their complementarity.
• Reliability-Aware Fusion (RAF) based on cross-attention and conditioning on view-related features improves Rank-1 by 2.7% over fixed average fusion under G→A protocol.
• Geometry-based and semantic-based attention individually improve 2D–3D interaction over coarse fusion; combining geometry and semantics yields best results, validating spatial and semantic complementarity.
• Visualization confirms that RAF adaptively weights 2D/3D/fused features according to sample conditions, while 3D Geometry-Semantic Attention grounds 2D patches to semantically consistent and spatially nearby 3D voxels.

Threat model

The adversary is an aerial or ground camera capturing monocular images of persons under arbitrary viewpoints causing severe occlusion and geometric distortion, aiming to evade identification or cause mismatches. The system assumes no adversarial perturbations like spoofing or active attacks, but the challenge is in viewpoint variation and partial observability. The model cannot access multi-view or temporal sequences, nor perfect 3D scans, thus must infer robust identity representations from single-view noisy inputs.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicitly a challenging cross-view retrieval scenario with aerial and ground camera viewpoints producing severe occlusion and geometric deformation. The model assumes single RGB images per person capture from unknown aerial or ground views, with no paired multi-view temporal sequences. No explicit adversarial attacks beyond viewpoint shifts are considered.

2. Data: Experiments use three established aerial-ground person ReID datasets: CARGO (108,563 images, 5,000 IDs, 8 ground + 5 aerial cameras), AG-ReID.v1 (21,893 images, 388 IDs, 1 aerial + 1 ground camera), and AG-ReID.v2 (100,502 images, 1,615 IDs, multiple camera types). Official train/test splits and evaluation protocols are used. Images resized to 256×128, patch size 16x16.

3. Architecture/Algorithm: The VR3D framework consists of an offline 3D human representation generation pipeline leveraging pre-trained SAM3 for person segmentation and SAM3D for voxel-wise 3D reconstruction and camera pose estimation. The coordinates of 2D image patches are transformed to a canonical 3D space using extracted camera extrinsics. The 2D image is encoded by a ViT-based 2D encoder (VDT backbone), extracting global appearance features, local patch features, and view-related features. The 3D voxel representation is encoded via a lightweight sparse 3D encoder producing local 3D features and a pooled global 3D structural feature. The core View-Robust 3D Representation Interaction (VR3I) module uses 3D Geometry-Semantic Attention (3D-GSA) to compute attention weights between local 2D patches and 3D voxels by jointly considering semantic similarity and geometric spatial proximity in 3D space. This yields a spatially grounded fused feature. The final representation concatenates the 2D, fused, and 3D structural features.

4. Training Regime: Trained with identity supervision (cross-entropy and triplet losses) applied to both 2D appearance and final fused features. View-related constraints include a view classification loss and an orthogonality loss to promote disentanglement. SGD optimizer with batch size 64 (16 IDs × 4 images), trained for 120 epochs on a single Nvidia A100 GPU. Learning rate decayed from 8e-3 to 1.6e-6 with warm-up cosine schedule.

5. Evaluation Protocol: Metrics include Rank-1 accuracy and mean Average Precision (mAP) evaluated under various aerial-ground cross-view protocols (A→G, G→A, A↔G). Baselines include SOTA methods such as VDT, SD-ReID, SeCap, ViSA, and others. Ablation studies isolate VR3I components, fusion strategies, and geometry/semantic attention roles. Cross-validation not reported. Distribution shifts tested via diverse aerial, ground, wearable, and CCTV cameras.

6. Reproducibility: Code release planned but not yet available. Uses publicly available datasets (CARGO, AG-ReID.v1/v2). Pretrained SAM3 and SAM3D models are external components for 3D priors, exact versions and weights are from cited works.

Concrete example: Given a single aerial image, the method first segments the person using SAM3, then generates a 3D voxel human model and camera parameters using SAM3D. The 2D image patches are lifted to 3D canonical coordinates. The ViT encodes 2D features, while a sparse 3D encoder processes voxels. Attention between the two modalities considering semantic and geometric similarity produces fused features. These plus the original 2D and 3D global features are reliability-weighted and fused through the RAF module, conditioned by view-related features, resulting in a final robust representation for matching gallery images from ground cameras.

Technical innovations

• Propose View-Robust 3D Representation Interaction (VR3I) that lifts 2D local appearance patches into a canonical 3D coordinate system and uses 3D Geometry-Semantic Attention to fuse 2D and 3D voxel features based on spatial and semantic consistency.
• Introduce Reliability-Aware Fusion (RAF), a view-conditioned cross-attention module that adaptively weights multiple complementary representations (2D, fused, 3D) depending on sample-specific reliability influenced by viewpoint and 3D reconstruction quality.
• Construct an offline 3D human representation generation pipeline leveraging pre-trained SAM3/SAM3D to infer accurate 3D voxel human models and camera poses from single monocular 2D images under challenging aerial-ground viewpoints.
• Combine geometry-based spatial proximity and semantic-based feature correlation within a unified multi-head attention mechanism to ground 2D image semantics structurally in 3D space, addressing viewpoint bias in aerial-ground ReID.

Datasets

• CARGO — 108,563 images — publicly available aerial-ground ReID dataset with 5,000 identities from 8 ground + 5 aerial cameras
• AG-ReID.v1 — 21,893 images — publicly available aerial-ground ReID dataset with 388 identities captured by 1 aerial and 1 ground camera
• AG-ReID.v2 — 100,502 images — publicly available aerial-ground ReID dataset with 1,615 identities and diversified viewpoints including UAVs, CCTV, and wearable devices

Baselines vs proposed

• VDT: Rank-1 = 45.00% (A↔G CARGO) vs VR3D: 59.38%
• SD-ReID: Rank-1 = 53.12% (A↔G CARGO) vs VR3D: 59.38%
• SeCap: mAP = 46.37% (A→G AG-ReID.v1) vs VR3D: 79.95%
• SVPR-ReID: mAP = 77.85% (G→A AG-ReID.v1) vs VR3D: 82.87%
• ViSA: mAP = 83.61% (A→C AG-ReID.v2) vs VR3D: 84.54%
• 3D Global Feature fusion only: mAP 76.60% (A→G AG-ReID.v1) vs VR3I fused feature: 78.20%; full VR3D with RAF: 79.95%
• RAF cross-attention fusion: Rank-1 = 87.39% (A→G AG-ReID.v1) vs average fusion: 86.44%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02598.

Fig 1: Illustration of our motivation. (a) Previous methods

Fig 2: Illustrations of the View-Robust 3D Representation Learning framework (VR3D). An offline pipeline extracts 3D

Fig 3 (page 3).

Fig 4 (page 3).

Fig 3: Hyperparameter analysis of the geometric weight

Fig 4: Retrieval results on AG-ReID.v1 under the A→G

Fig 5: Visualization of 3D-GSA on AG-ReID.v1. Orange

Fig 6: Visualization of RAF reliability weights under dif-

Limitations

• 3D human representation relies heavily on pre-trained SAM3 and SAM3D models, which may not generalize to very different environments or severe occlusions.
• The method assumes availability of accurate camera pose parameters for canonical space alignment; errors in pose estimation could degrade performance.
• Evaluation focuses on existing aerial-ground datasets but does not test robustness explicitly under adversarial occlusions or malicious attacks.
• View disentanglement is aided by view classification loss but explicit direct supervision of viewpoint invariance is not extensively quantified.
• The offline 3D reconstruction pipeline adds computational overhead not discussed in efficiency analysis.
• Code and trained weights are not yet released, limiting reproducibility until they are made available.

Open questions / follow-ons

• How well does VR3D generalize to completely unseen viewpoints or environmental conditions not present in training data?
• Can the offline 3D reconstruction pipeline be integrated into an end-to-end trainable framework for joint optimization?
• How sensitive is the model’s performance to errors in camera pose estimation or human segmentation masks?
• Could the 3D Geometry-Semantic Attention concept be extended to other cross-modal identity matching tasks (e.g., thermal-visual or radar-visual ReID)?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the insights from VR3D highlight the advantage of learning and utilizing 3D spatially grounded representations to achieve robustness against challenging viewpoint variations. While most CAPTCHA or bot-detection systems rely on 2D image or behavioral cues, incorporating 3D structure priors may enhance resilience to spoofing or mimicry attacks where appearance might be manipulated. The reliability-aware fusion mechanism can inspire adaptive aggregation of multi-modal signals based on confidence estimation, improving detection accuracy under variable capture conditions. Additionally, the methodology of leveraging pre-trained 3D models to complement sparse views could be applied in security-sensitive person identification or multi-camera tracking in public spaces protected by CAPTCHA-like anti-bot systems.

Cite

@article{arxiv2608_02598,
  title={ VR3D: View-Robust 3D Representation Learning for Aerial-Ground Person Re-Identification },
  author={ Chao Ji and Shiyu Xuan and Zechao Li },
  journal={arXiv preprint arXiv:2608.02598},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02598}
}

Read the full paper

• arXiv:2608.02598 (abstract)
• arXiv:2608.02598 (PDF)