Structural-Semantic Reciprocal Learning for Unsupervised Visible-Infrared Person Re-Identification

Source: arXiv:2607.15220 · Published 2026-07-16 · By Moyao Tian, Shijia Liu, Yan Yang, Xin Yuan, Minshi Chen, Wei Wang et al.

TL;DR

This paper addresses the problem of unsupervised visible-infrared person re-identification (USVI-ReID), a challenging cross-modal retrieval task due to the large modality gap between visible and infrared images and the lack of cross-modal identity annotations. Existing unsupervised methods use progressive association with clustering to generate pseudo-labels but suffer from reliance on ambiguous global features and the unchecked accumulation of noisy pseudo-labels in an open-loop training pipeline. The authors propose Structural-Semantic Reciprocal Learning (SSRL), a novel framework that converts open-loop association into a closed-loop system enabling self-correction. SSRL combines Fine-grained Structural Decoupling (FSD), which extracts independent body-part primitives as reliable spatial anchors to reduce cross-modal distortion, with Closed-loop Semantic Calibration (CSC), which reconstructs shared semantic prototypes each epoch to filter noisy pseudo-label supervision before re-clustering. This reciprocal interaction between structural and semantic modules jointly stabilizes the feature manifold and cleans pseudo-labels progressively, substantially improving cross-modal feature alignment.

Extensive experiments on two standard VI-ReID benchmarks, SYSU-MM01 and RegDB, demonstrate SSRL significantly outperforms state-of-the-art unsupervised methods, achieving 59.47% Rank-1 on SYSU-MM01 All Search and surpassing several supervised counterparts on RegDB with over 92% Rank-1. Ablations confirm both FSD and CSC contribute substantially, and parameter studies establish the robustness of the closed-loop learning mechanism and part-based structural decomposition to modality gaps. Visualizations via Grad-CAM and t-SNE illustrate that SSRL yields more spatially consistent attention and compact cross-modal clusters. This work demonstrates how combining architectural inductive biases to extract spatially consistent local features with algorithmic closed-loop semantic purification creates a stable and effective unsupervised learning paradigm for visible-infrared person re-identification.

Key findings

• SSRL achieves 59.47% Rank-1 and 55.35% mAP on SYSU-MM01 (All Search), outperforming previous USVI-ReID methods such as PCAL+ by +1.53% and +2.50% respectively.
• On RegDB, SSRL obtains 92.47% Rank-1 (Visible→Infrared) and 91.06% Rank-1 (Infrared→Visible), surpassing all unsupervised counterparts and several supervised methods.
• Fine-grained Structural Decoupling (FSD) alone improves SYSU-MM01 Rank-1 from 54.39% to 57.46% (+3.07%) and RegDB Visible→Infrared Rank-1 from 86.43% to 88.10% (+1.67%).
• Closed-loop Semantic Calibration (CSC) alone raises RegDB Rank-1 to 91.71%, though it may cause minor SYSU-MM01 performance drops without FSD.
• Ablations show part alignment loss Lpa and adjacent-part regularization Lpc incrementally raise Rank-1 on SYSU-MM01 from 55.55% to 59.47%.
• Prototype classification loss Lsp combined with cross-modal semantic consistency loss Lsc achieves better Rank-1 (59.47%) than either alone.
• Three-part horizontal partitioning (upper, middle, lower body) maximizes performance with weights α=[1/6,3/6,2/6], outperforming uniform or alternative ratios.
• Parameter sensitivity analysis shows optimal loss weights λp=0.3, λsp=0.1, λsc=0.1 with robust performance across ranges, confirming stable closed-loop learning.

Threat model

The adversary is implicitly the modality gap and noise in pseudo-labels arising from unsupervised clustering without manually annotated cross-modal identity labels. SSRL assumes no direct adversarial attacks but focuses on reducing noise accumulation and modality-induced distortions that lead to incorrect clustering and ID mis-linking. The method does not assume knowledge of ground-truth labels and cannot rely on supervised matching. It is not designed to counter malicious attacks but to stabilize unsupervised cross-modal representation learning.

Methodology — deep read

The authors propose SSRL, an unsupervised framework for visible-infrared person re-identification that jointly learns cross-modal modality-invariant features without identity labels. The threat model assumes an unsupervised setting where no cross-modal ID annotations are available; the method must rely on clustering pseudo-labels from noisy features under large RGB-IR modality gaps.

Data: Experiments use two benchmark datasets: SYSU-MM01 with 287,628 visible and 15,792 infrared images of 491 identities (395 train / 96 test) and RegDB with 4,120 visible and 4,120 infrared images from 412 identities (206 train / 206 test). Both datasets have no identity labels exposed to training in the unsupervised paradigm.

Architecture: SSRL consists of two key modules synergized in a reciprocal closed-loop training pipeline:

1. Fine-grained Structural Decoupling (FSD): Each input image (visible or infrared) is processed by modality-specific shallow encoders and a shared ResNet-50 backbone, outputting a feature map Am∈RC×H×W. This map is horizontally partitioned into three coarse anatomical regions (upper, middle, lower) using adaptive average pooling. Each region is fed into an independent bottleneck head, producing L2-normalized part-level primitives fpk_m as local spatial anchors. A global branch also produces holistic global features fg_m via generalized-mean (GeM) pooling.

FSD includes losses to encourage modality-specific compactness (non-parametric memory banks per modality), cross-modal part alignment between visible and infrared primitives with same pseudo-labels, and adjacent-part consistency regularization to preserve body topology and prevent spatial drift.

2. Closed-loop Semantic Calibration (CSC): At the start of each epoch, global embeddings fg_t are extracted and merged across modalities. Using the k-reciprocal Jaccard distance to suppress cross-modal false positives, DBSCAN clustering generates pseudo-labels Yt. Shared semantic prototypes Pt_c for each cluster are reconstructed fully from visible and infrared centroid features using a conditional aggregation rule. This prototype bank is freshly rebuilt every epoch to prevent historical bias.

The semantic prototypes are fed back into training by optimizing a prototype classification loss (softmax over prototypes) and a cross-modal semantic consistency loss between visible/infrared pairs with same pseudo-labels. This closes the loop by refining pseudo-supervision and purifying features before the next clustering cycle.

Training regime involves 50 epochs, batch size 64, Adam optimizer with learning rate 3.5e-4 and weight decay 5e-4. A two-stage optimization: Stage I with FSD only (λp=0.3, λpa=0.1, λpc=0.05, λadv=0.15) and Stage II activating CSC (λsp=0.1, λsc=0.1, temperature τ=0.05). Pseudo-labels are updated each epoch via DBSCAN using k-reciprocal Jaccard distance with tuned neighborhood and threshold parameters per dataset.

Evaluation uses standard CMC Rank-1/10 and mAP metrics under various search protocols on SYSU-MM01 and RegDB. Baselines include recent state-of-the-art unsupervised VI-ReID methods and several supervised methods for comparison. Ablation studies evaluate the impact of FSD, CSC, part weighting, and loss components. Parameter sensitivity explores effects of key hyperparameters.

Reproducibility: The implementation is in PyTorch with ImageNet pretrained ResNet-50 backbone. Dataset splits and protocols follow standard benchmarks. The paper does not mention public code release or frozen weights explicitly. However, methodology is described in sufficient detail for replication on standard datasets.

A concrete example epoch flow: At epoch t, visible and infrared images pass through the encoders to extract global and part-level features. These features form a cross-modal set for clustering via k-reciprocal Jaccard distance and DBSCAN, producing pseudo-labels. Prototypes are built and normalized for each cluster from visible and infrared centroids. Prototype classification and semantic consistency losses refine the embeddings by backpropagation. This updated model produces cleaner features for the next epoch's clustering, establishing a closed semantic feedback loop that progressively stabilizes supervisory signals and representations.

Technical innovations

• Fine-grained Structural Decoupling (FSD) partitions feature maps horizontally and processes part primitives with separate bottlenecks, preventing modality noise entanglement and providing spatial anchors robust to cross-modal distortion.
• Closed-loop Semantic Calibration (CSC) reconstructs semantic prototypes at each epoch from synchronized pseudo-label clustering, feeding back into training to suppress pseudo-label noise and refine the feature manifold in a self-correcting loop.
• Combination of architectural local structural decoupling with algorithmic closed-loop semantic purification enables reciprocal interaction that stabilizes both feature representations and supervisory signals in unsupervised VI-ReID.
• Use of modal-invariant k-reciprocal neighborhood Jaccard distance for robust cross-modal clustering to reduce false positive links between visible and infrared samples.

Datasets

• SYSU-MM01 — 287,628 visible + 15,792 infrared images — public benchmark
• RegDB — 4,120 visible + 4,120 infrared images — public benchmark

Baselines vs proposed

• PCAL+ [9]: SYSU-MM01 All Search Rank-1 = 57.94% vs SSRL: 59.47%
• H2H [4]: RegDB Visible→Infrared Rank-1 = 23.81% vs SSRL: 92.47%
• CARR [32]: SYSU-MM01 Indoor Search Rank-1 = 62.70% vs SSRL: 63.83%
• DEEN [17] (supervised): RegDB Visible→Infrared Rank-1 = 91.10% vs SSRL: 92.47%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15220.

Fig 2: Framework of Structural-Semantic Reciprocal Learning (SSRL). SSRL

Fig 4: Qualitative visualizations of the SSRL.

Fig 3: Parameter sensitivity analysis on SYSU-MM01(All Search).

Fig 4 (page 13).

Fig 5 (page 13).

Limitations

• No explicit adversarial robustness evaluation against adaptive cross-modal attacks.
• Only evaluates on two standard benchmarks; generalization to other heterogenous datasets unknown.
• Closed-loop clustering relies on DBSCAN sensitivity to epsilon and k parameters, potentially brittle in other data distributions.
• Lack of code release or pretrained weights limits immediate reproducibility by external parties.
• Cross-modal alignment is constrained to three horizontal partitions; alternative part decompositions or dynamic parts unexplored.
• Evaluation does not cover scenarios with severe occlusion or large-scale gallery settings.

Open questions / follow-ons

• How does SSRL perform under adversarially crafted cross-modal perturbations targeting pseudo-label reliability?
• Can the approach extend to more flexible or learned part partitions rather than fixed three-part horizontal stripes?
• Would integration of temporal or video information improve structural-semantic reciprocal learning for person ReID in cross-modal settings?
• Can the closed-loop semantic calibration mechanism be combined with transformer-based backbones or other modalities such as depth to improve generalization?

Why it matters for bot defense

For bot-defense and CAPTCHA engineering, SSRL provides insights into stabilizing cross-modal unsupervised learning pipelines via architectural and algorithmic design. The idea of fine-grained structural anchoring can inspire robust feature decomposition of adversarial inputs or multi-sensor signals. Closed-loop semantic calibration suggests feedback mechanisms that iteratively refine supervisory signals and reduce label noise, which could help in training unsupervised anomaly detectors or bot classifiers with noisy labels. Although focused on person re-identification, the reciprocity between structural spatial cues and semantic label purification highlights generalizable principles for robust representation learning under modality shifts and noisy supervision, relevant to CAPTCHAs deployed in heterogeneous sensing environments. However, direct application requires adaptation to bot-defense input modalities and threat models.

Cite

@article{arxiv2607_15220,
  title={ Structural-Semantic Reciprocal Learning for Unsupervised Visible-Infrared Person Re-Identification },
  author={ Moyao Tian and Shijia Liu and Yan Yang and Xin Yuan and Minshi Chen and Wei Wang and Xiao Wang },
  journal={arXiv preprint arXiv:2607.15220},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15220}
}

Read the full paper

• arXiv:2607.15220 (abstract)
• arXiv:2607.15220 (PDF)