MetaPerch: Learning from metadata for bioacoustics foundation models

Source: arXiv:2607.14072 · Published 2026-07-15 · By Mustafa Chasmai, Vincent Dumoulin, Jenny Hamer

TL;DR

This paper addresses the challenge of improving species identification in bioacoustics foundation models by leveraging auxiliary metadata available during training, such as recording location, time, and background species. Existing models typically rely only on vocalization audio and species labels, but large-scale citizen science platforms also provide rich metadata that correlates with species occurrences and can help mitigate domain shifts in species distribution, acoustic environment, and geographic coverage. MetaPerch introduces a multi-task learning framework that incorporates prediction heads for nine types of metadata alongside the primary species classification task, augmenting the learned representations with contextual signals. The authors conduct a thorough empirical evaluation over 17 bioacoustic datasets spanning diverse taxa, geographic regions, and recording conditions. Results demonstrate consistent improvements over strong baselines (BioBaseline and Perch 2.0) across multiple challenging benchmarks, including the BirdSet, BEANS, and WABAD datasets, with gains particularly pronounced in underrepresented geographic regions and when adapting to novel species or soundscapes. Ablations confirm that location, season, and background species metadata are among the most valuable for transfer performance.

Key findings

• MetaPerch improves average ROC-AUC on BirdSet by 0.015 over BioBaseline baseline (0.906 vs 0.891), narrowing gap to Perch 2.0.
• On BEANS classification tasks, MetaPerch increases accuracy by 0.016 on average compared to BioBaseline, notably +0.021 on Watkins and +0.012 on Bats datasets.
• On BEANS detection tasks, MetaPerch improves mean cmAP by 0.006 relative to BioBaseline.
• On WABAD, MetaPerch achieves 0.811 ROC-AUC for linear probes vs 0.741 for baseline (+0.07 delta).
• Metadata auxiliary tasks involving location, season, and background species contribute most to performance gains.
• Improvements are largest in geographically underrepresented regions (e.g., South America, Hawai'i) with +0.025 ROC-AUC on BirdSet datasets.
• Mixing metadata during audio data augmentation ('mixup') with multi-hot targets outperforms alternatives.
• Gradient reversal layers (adversarial training) help reduce learning spurious correlations from some metadata but details on their net effect are mixed.

Threat model

n/a - this is not a security-focused paper, but rather explores how auxiliary metadata supervision can improve species classification under domain shifts in bioacoustic monitoring.

Methodology — deep read

The threat model is not explicitly adversarial here, but the approach assumes metadata are readily available during training for citizen science recordings, while at test-time the model uses only audio inputs. The adversary model is thus not the focus. The authors aggregate over 1.5 million labeled recordings from four large-scale datasets: Xeno-Canto, iNaturalist, Tierstimmenarchiv, and FSD50K, collectively covering 14,785 species and 18,174 hours total. These datasets include diverse taxa and geographic regions, with metadata such as location (S2 cells), time (season), background species, recording quality, and others. Due to missing metadata in parts of datasets, they zero-out losses for absent data to handle incomplete metadata gracefully.

The baseline architecture (BioBaseline) is derived from Perch 2.0 but without self-distillation or source prediction. Audio spectrograms (5-second windows) are embedded with EfficientNet-B3 backbone. Features feed a prototype learning head and a linear classifier trained on species labels via cross-entropy loss. MetaPerch adds parallel multilayer perceptron (MLP) heads for auxiliary metadata prediction, each trained with cross-entropy classification losses weighted as hyperparameters. The metadata predictions include location (S2 cells), season, background species, and other modalities studied.

Training uses augmentations including mixup, with careful design to combine metadata labels—mixing present metadata as multi-hot labels works best. Missing metadata is handled by placeholder tokens and masking of losses. Adversarial training employs gradient reversal layers to prevent learning spurious correlations from metadata when beneficial. Hyperparameter tuning and model selection use Vizier black-box optimizer based on validation tasks capturing observed species classification, one-shot embedding retrieval, and novel species linear probing.

Evaluation uses 17 challenging bioacoustic datasets covering focal and passive acoustic monitoring (PAM) recordings with domain shifts in species, acoustic environment, and geography. Metrics include ROC-AUC for detection tasks, accuracy for classification, and mean average precision (cmAP) for detection. Models are evaluated via repeated training and averaging over multiple random seeds to report means and standard deviations.

The authors also conduct ablations on loss weighting schemes, mixup variants, and metadata usage to isolate contributions of metadata types and training design choices. End-to-end, an input audio window is transformed to a spectrogram, embedded by EfficientNet-B3, whose features are used simultaneously to predict the species class and multiple metadata labels, optimizing a combined multi-task loss weighted per metadata modality.

While code and models are released (link to github), some datasets contain restricted or missing metadata limiting full reproducibility for certain auxiliary tasks. The training infrastructure uses EfficientNet-B3 with mixup and Vizier hyperparameter optimization. Some design decisions such as mixup over incomplete metadata and adversarial training are empirically explored but not exhaustively theoretically justified.

Technical innovations

• Multi-task learning framework integrating nine types of metadata as auxiliary prediction tasks alongside species ID for bioacoustic foundation models.
• Handling of missing metadata via placeholder tokens and selectively zeroed losses during multi-loss training.
• Design of metadata mixup strategy mixing present metadata labels as multi-hot targets to align with audio data augmentation.
• Use of gradient reversal layers to adversarially suppress learning of spurious metadata correlations.
• Comprehensive empirical disentangling of diverse metadata modality contributions to species ID transfer under domain shifts.

Datasets

• Xeno-Canto — 13,555 hours — citizen science platform, bird vocalizations
• iNaturalist — 3,077 hours — citizen science platform, diverse taxa
• Tierstimmenarchiv — 1,462 hours — curated animal vocalizations
• FSD50K — 80 hours — environmental and anthropogenic sounds
• BirdSet — various bird-focused PAM datasets for domain-shift evaluation
• BEANS — 12 bioacoustic and miscellaneous audio datasets for classification and detection
• WABAD — 1,192 bird species global soundscape dataset

Baselines vs proposed

• BioBaseline: BirdSet mean ROC-AUC = 0.891 vs MetaPerch: 0.906 (+0.015)
• Perch 2.0: BirdSet mean ROC-AUC = 0.908 vs MetaPerch: 0.906 (-0.002)
• BioBaseline: BEANS classification mean accuracy = 0.854 vs MetaPerch: 0.870 (+0.016)
• BioBaseline: BEANS detection mean cmAP = 0.506 vs MetaPerch: 0.512 (+0.006)
• BioBaseline: WABAD linear probe ROC-AUC = 0.741 vs MetaPerch: 0.811 (+0.070)
• BioBaseline: WABAD prototype learning ROC-AUC = 0.928 vs MetaPerch: 0.946 (+0.018)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14072.

Fig 1: Problem setup.

Fig 2: Dataset statistics. Geographical distribution of the training (log scale) and evaluation (rough locations) datasets.

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The model relies on availability of metadata during training, which can be missing or imbalanced across datasets and locations.
• Metadata benefits vary by biome and scenario, with some domains (e.g., deserts, combined species & acoustic shifts) showing limited or negative improvements.
• Potential for models to learn spurious correlations from metadata that do not generalize ecologically, partially mitigated but not fully resolved by adversarial training.
• Validation metrics used for model selection do not always align perfectly with test performance, complicating hyperparameter tuning.
• Few-shot learning improvements from metadata diminish at higher shot counts, suggesting incomplete understanding of metadata utility across sample regimes.
• Some metadata modalities like recording quality, call type, or recordist notes contribute less or were excluded from final models, limiting generality.

Open questions / follow-ons

• How to disentangle and mitigate spurious metadata correlations beyond gradient reversal and adversarial training?
• Can metadata usage be extended effectively to zero-shot or few-shot species identification, especially for rare or unseen taxa?
• What are the best validation protocols and metrics to align hyperparameter tuning more closely with final transfer task performance?
• How can metadata incorporation be adapted to scenarios where metadata is unavailable or unreliable at deployment time?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, MetaPerch offers a demonstration of how auxiliary contextual metadata—beyond the primary input modality—can enhance model robustness and generalization under real-world distribution shifts. This suggests that incorporating domain metadata, such as IP geolocation, time of access, or device characteristics, as auxiliary tasks during training may yield richer, more discriminative embeddings in bot-detection systems. The careful handling of missing metadata and nonlinear multi-task weighting strategies provide practical lessons for combining heterogeneous input signals. Moreover, the use of adversarial training to reduce reliance on spurious metadata correlations parallels similar challenges in CAPTCHA robustness, where malicious actors may manipulate metadata to evade detection. While the content domain differs, the methodology and multi-modal learning insights carry over to security contexts with multi-faceted input signals and evolving attacker behaviors.

Cite

@article{arxiv2607_14072,
  title={ MetaPerch: Learning from metadata for bioacoustics foundation models },
  author={ Mustafa Chasmai and Vincent Dumoulin and Jenny Hamer },
  journal={arXiv preprint arXiv:2607.14072},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14072}
}

Read the full paper

• arXiv:2607.14072 (abstract)
• arXiv:2607.14072 (PDF)