Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning

Source: arXiv:2607.20874 · Published 2026-07-23 · By Zhihua Xu, Zhijing Yang, Yufeng Yang, Tianshui Chen

TL;DR

This paper addresses the underexplored problem of webly supervised multi-label recognition (WS-MLR), where models are trained using large-scale web images with noisy and incomplete multi-label annotations derived from image search keywords. Unlike single-label webly supervised learning, WS-MLR presents unique challenges due to more complex label noise (false positives and false negatives), semantic scattering of multiple objects per image, and class imbalance. To facilitate research in this area, the authors construct a unified benchmark consisting of two datasets, Web-COCO and Web-Pascal, containing roughly 290K and 236K web-sourced images respectively, covering the same categories as Microsoft COCO and Pascal VOC. They also re-implement representative multi-label, few-shot, and partial label recognition methods under a unified training protocol to allow fair comparisons.

The main technical contribution is a Dual-Branch Multi-Label Contrastive Learning (DBMLCL) framework that learns both category-specific instance-level feature vectors and category-level prototypes via two independent network branches. By computing similarities between instance features and prototypes, the model identifies and corrects noisy labels in the webly supervised data dynamically during training. The framework adopts tailored instance- and prototype-level contrastive losses to learn compact discriminative representations despite label noise. Extensive experiments on the constructed benchmark datasets show that DBMLCL consistently outperforms existing state-of-the-art baselines by 1-3% mAP across varying data proportions, demonstrating improved robustness to real-world web label noise and semantic complexity.

Key findings

• The constructed Web-COCO dataset contains 290,000 web images with 80 categories and shows severe label noise: average keyword-label precision is 46.1% and recall 64.6%.
• Web-Pascal dataset contains 236,043 web images with 20 categories and exhibits similar label noise and class imbalance challenges as Web-COCO.
• DBMLCL achieves a mAP of 71.4% on Web-COCO using full dataset training, outperforming the best baseline CSRA (70.3%) and P-GCN (69.5%) by 1.1% and 1.9% respectively.
• Across 20%, 40%, and 100% data proportions on Web-COCO, DBMLCL consistently delivers higher mAP (up to +3.6% at 20%), OF1, and CF1 compared to six baselines.
• On Web-Pascal, DBMLCL achieves competitive mAP (~87.4%) and surpasses baselines in OF1 and CF1, though CCD obtains slightly higher mAP likely due to better category alignment with CLIP features.
• Instance-level and prototype-level contrastive losses together enable the model to learn category-specific feature representations that can detect and correct noisy web labels effectively.
• Momentum-updated category-specific thresholds stabilize label correction and adapt to the training dynamics, improving robustness to noisy pseudo labels.
• Two-branch architecture encourages learning complementary features and enhances contrastive learning by processing disjoint subsets of training data separately.

Threat model

The adversary is implicit label noise present in webly supervised multi-label datasets, arising from keyword mismatch, missing co-occurring objects, and semantic ambiguity in web image search. The adversary cannot modify the model or training data intentionally but causes multi-label supervision to contain both false-positive and false-negative labels, complicating learning.

Methodology — deep read

1. Threat Model and Assumptions:

• The adversary is implicit: webly supervised learning assumes noisy, incomplete, and often incorrect keyword labels obtained from web image search. There is no direct adversarial manipulation but label noise arises naturally from search keyword mismatches and missing object annotations.
• The method addresses correcting both false positives (keyword labels assigned when object absent) and false negatives (missing labels for co-occurring objects).

2. Data:

• Web-COCO: ~290,000 images gathered via multi-category keyword combinations from Google, Baidu, Bing image search, matching MS-COCO's 80 categories.
• Web-Pascal: ~236,000 images similarly collected for Pascal VOC's 20 categories.
• 20,000 random Web-COCO images manually annotated for label noise evaluation; official MS-COCO/Pascal VOC validation/test sets used for final evaluation.
• Label vectors for training are derived from keywords: categories used as keywords are positive labels; others treated as negative or unobserved, resulting in noisy multi-label supervisory signals.

3. Architecture/Algorithm:

• Dual-branch design: two independent neural network branches (ResNet-101 backbone + semantic-aware representation learning module per branch).
• Each branch extracts category-specific instance-level features (a vector per category per image) and maintains category-level prototype vectors updated by momentum.
• Prediction: classification scores computed from instance features via gating/graph models + FC layers; sigmoid used for multi-label predictions.
• Losses:
  • Instance Contrastive Loss: aligns instance-level features of same categories between the two branches for same image while pushing apart negatives drawn from a circular queue of feature vectors not belonging to current image categories.
  • Prototype Contrastive Loss: clusters instance-level features close to their category prototypes and away from other categories to enhance inter-class discrimination.
• Label Correction:
  • After a warm-up of 5 epochs, use averaged prediction probabilities and cosine similarities between instance features and prototypes to detect noisy labels.
  • Labels corrected if prediction confidence exceeds category-specific thresholds (adapted by momentum).
  • Corrected labels used for further training updates, enabling refining supervision.

4. Training Regime:

• Batch size 8, Adam optimizer with initial LR 1e-5, weight decay per baseline originals (0 after correction phase).
• 20 epochs total with LR decayed after epoch 15.
• Input images resized/cropped to 448x448 with random cropping and horizontal flip augmentation.
• Loss balance factors: 0.01 for instance contrastive, 0.05 for prototype contrastive losses.
• Two branches trained on disjoint subsets of training data to encourage diverse representations.

5. Evaluation Protocol:

• Metrics: mean average precision (mAP) as primary, overall F1 (OF1), and per-class F1 (CF1) to assess multi-label prediction quality.
• Baselines: six representative traditional MLR methods (SSGRL, ML-GCN, CSRA, ASL, P-GCN, KGGR), plus CLIP-based CCD method.
• Ablations conducted to validate contributions of dual-branch, contrastive losses, and label correction.
• Data proportions evaluated at 20%, 40%, and 100% training data to study scalability.

6. Reproducibility:

• The dataset preprocessing pipeline, benchmark construction, and baseline implementations unified and released.
• Code, datasets, and trained models are publicly available on GitHub.

Example End-to-End:

• An input image is assigned noisy keyword-derived labels.
• It is passed simultaneously through the two branches extracting category-specific features.
• Contrastive losses align positive category features and push apart negatives using instance and prototype queues.
• After initial training, model predictions and feature-prototype similarities identify likely label noise.
• Labels are corrected dynamically based on learned thresholds, and subsequent training uses these refined labels.
• The two-branch structure helps learn diverse and robust features that improve multi-label recognition despite noisy web supervision.

Technical innovations

• Creation of a large-scale unified webly supervised multi-label recognition benchmark with Web-COCO and Web-Pascal datasets reflecting realistic, noisy web label distributions.
• Dual-branch architecture that processes disjoint data subsets independently to encourage diverse, complementary category-specific instance-level features.
• Joint use of instance-level and prototype-level multi-label contrastive learning losses to learn discriminative category-specific representations capable of correcting label noise.
• Dynamic label correction strategy based on combined model prediction confidence and instance-prototype cosine similarities with momentum-updated adaptive thresholds.

Datasets

• Web-COCO — 290,000 images — constructed by web crawling with multi-category keywords matching Microsoft COCO classes
• Web-Pascal — 236,043 images — constructed analogously for Pascal VOC categories

Baselines vs proposed

• CSRA on Web-COCO (100% data): mAP = 70.3% vs DBMLCL: 71.4%
• P-GCN on Web-COCO (100% data): mAP = 69.5% vs DBMLCL: 71.4%
• CCD on Web-COCO (100% data): mAP = 58.3% vs DBMLCL: 71.4%
• SSGRL on Web-COCO (100% data): mAP = 64.6% vs DBMLCL: 71.4%
• On Web-Pascal (100% data) CCD mAP = 87.7% vs DBMLCL mAP = 87.4% (CCD better by 0.3%), but DBMLCL outperforms CCD on OF1 and CF1 metrics.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.20874.

Fig 1 (page 1).

Fig 1: Some examples with false-positive and false-negative labels. The keywords are presented on the left and the

Fig 2: Precision and recall of pseudo labels in the Web-COCO dataset.

Fig 3: Distribution of image numbers (%) for each category in the Web-COCO dataset.

Fig 4: Precision and recall of pseudo labels in the Web-Pascal dataset.

Fig 5: Distribution of image numbers (%) for each category in the Web-Pascal dataset.

Fig 6: Illustration of the DBMLCL framework. It contains two branches, and each of them consists of a feature extractor,

Fig 7: The AP of each category of our proposed framework and the SSGRL baseline.

Limitations

• Webly supervised datasets still contain significant label noise and missing annotations; label correction is heuristic using similarity thresholds that may not perfectly clean labels.
• The model requires careful hyperparameter tuning for contrastive loss balancing and label correction thresholds, which may limit out-of-the-box usage.
• Evaluation is limited to two datasets constrained to COCO and Pascal VOC categories; generalization to broader or more fine-grained categories needs further study.
• No explicit adversarial robustness analysis or testing against intentional label poisoning or adversarial web retrieval scenarios.
• Limited ablation on alternative contrastive learning strategies or different backbone architectures; unclear how sensitive performance is to architectural choices.
• Although code and data are released, full reproducibility depends on consistent image retrieval results from search engines that may vary over time.

Open questions / follow-ons

• How does DBMLCL perform when transferred to more diverse multi-label domains beyond COCO/Pascal categories, including specialized or fine-grained classes?
• Can more adaptive or learned thresholding mechanisms improve label correction robustness without manual momentum tuning?
• What is the impact of different backbone architectures or vision-language pretrained models like newer CLIP variants when incorporated into DBMLCL?
• How resilient is DBMLCL to adversarially manipulated web images or intentional label poisoning?

Why it matters for bot defense

Webly supervised multi-label recognition with noisy labels shares conceptual alignment with bot defense mechanisms that must work with imperfect signals and noisy feedback. The DBMLCL framework's dual-branch contrastive learning and dynamic label correction can inspire robust training approaches for multi-label or multi-feature verification tasks in CAPTCHA analysis, where noisy or incomplete human interaction data often occurs. Moreover, the benchmark datasets and metrics provide a standardized way to evaluate algorithms trained on weak or noisy supervision—a common scenario in practical bot detection pipelines. The reliance on both instance-level and category-level feature alignment suggests a pathway to improve multi-factor verification mechanisms that combine local and global cues. However, direct application would require adapting to temporal and behavioral signals rather than static images. Overall, DBMLCL highlights techniques for handling noisy, incomplete annotations which is a central challenge in bot detection and CAPTCHA robustness.

Cite

@article{arxiv2607_20874,
  title={ Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning },
  author={ Zhihua Xu and Zhijing Yang and Yufeng Yang and Tianshui Chen },
  journal={arXiv preprint arXiv:2607.20874},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20874}
}

Read the full paper

• arXiv:2607.20874 (abstract)
• arXiv:2607.20874 (PDF)