Large Audio Language Models for Spoofing-Aware Speaker Verification

Source: arXiv:2607.14753 · Published 2026-07-16 · By Sofya Savelyeva, Mariia Perunova, Evgeny Kushnir, Artem Dvirniak, Dmitrii Korzh, Oleg Y. Rogov

TL;DR

This paper addresses the growing threat to automatic speaker verification (ASV) systems posed by sophisticated text-to-speech and voice cloning attacks, which enable high-quality spoofed audio that can impersonate legitimate speakers. Existing defenses mainly consist of modular cascaded pipelines that fuse separate ASV and spoofing countermeasure (CM) models, which yield good performance but lack unified end-to-end reasoning and interpretability. The authors investigate whether recent Large Audio Language Models (LALMs), pretrained on general audio tasks but not tailored for spoofing-aware speaker verification (SASV), can be adapted to perform unified three-way SASV decisions (target vs non-target human vs spoof) that also produce natural-language reasoning traces.

Through systematic experiments using zero-shot prompting, supervised fine-tuning with Low-Rank Adaptation (LoRA), chain-of-thought (CoT) reasoning traces, and reinforcement learning with group relative policy optimization (GRPO), the paper explores multiple adaptation strategies. The results confirm that LALMs without fine-tuning perform near chance in SASV, but significant improvements arise from task-specific supervised adaptation. A multi-headed loss combining three-class classification, additive angular margin (AAM) speaker discrimination loss, and binary spoof detection loss with hard-sample mining yields the strongest overall SASV accuracy and detection cost function metrics on the challenging ASVspoof 5 dataset. Reasoning-based training improves interpretability but does not uniformly enhance hard-label accuracy.

The adapted LALM system, based on SALMONN-7B and Qwen-Audio-7B models, is competitive with strong conventional fusion baselines of ECAPA-TDNN and WavLM/W2V2-based auxiliary spoof detectors, achieving better aggregated SASV accuracy and min a-DCF, though classical methods sometimes have lower equal error rates (EER) indicating stronger threshold-level speaker discrimination. The findings position LALMs as a promising foundation for unified SASV systems with natural-language auditing capabilities, while highlighting current trade-offs and needs for further research on reasoning quality and efficiency.

Key findings

• Zero-shot LALM models (SALMONN-7B, Qwen-Audio-7B) achieve near chance accuracy on ASVspoof 5 SASV (27.7% - 43.7%).
• LoRA fine-tuning substantially improves SASV accuracy to ~85% for both SALMONN and Qwen-Audio models.
• Adding an AAM loss head improves speaker verification accuracy but degrades spoof detection accuracy (CM accuracy drops from 96.5% to 21.0%).
• Including a dedicated binary spoof detection head recovers spoof detection accuracy to 80.0% and maintains good speaker verification.
• Hard-sample mining further improves overall accuracy to 89.3% and lowers min a-DCF to 0.19 on ASVspoof 5 eval set.
• Final SALMONN-based LALM outperforms baseline ECAPA2 + WavLM fusion on SASV accuracy (89.3% vs 79.8%) and min a-DCF (0.19 vs 0.41) under controlled evaluation.
• Reasoning-oriented training with chain-of-thought supervision and GRPO reinforcement achieves modest accuracy (~84-86%) and improved interpretability but no consistent gains in hard-label decision accuracy.
• Interpretability via natural-language reasoning traces is possible, but generated rationales require further evaluation for consistency and grounding.

Threat model

An adversary capable of producing realistic synthesized or voice-cloned speech to impersonate a target speaker, aiming to fool speaker verification systems into accepting spoof audio. The adversary may also use human impostor speech as non-target attacks. The system assumes no direct access to the model internals by the attacker and cannot detect spoofing solely by unconditional audio quality. The goal is to jointly verify speaker identity and reject spoofed trials in a unified decision.

Methodology — deep read

The paper investigates applying large audio language models (LALMs) to the task of Spoofing-Aware Speaker Verification (SASV), where the system must classify an enrollment-trial pair into one of three classes: target (genuine speaker), non-target (impostor human), or spoof (synthesized or manipulated audio).

1. Threat model & assumptions: The adversary is capable of generating high-quality spoof audio using recent TTS and voice cloning techniques that can potentially pass speaker verification systems. The system aims to jointly verify speaker identity and detect spoofing in a unified model.

2. Data: The primary dataset used for training and evaluation is ASVspoof 5, regarded as a challenging benchmark containing genuine and multiple types of spoofed audio. For genuine training data augmentation, bona fide utterances from VoxCeleb (disjoint speaker sets) are added. Approximately 1.8 million enrollment-trial pairs are constructed over 500 speakers. Audio is resampled to 16kHz, trimmed by voice activity detection, normalized per segment and cropped or padded to 6–7 seconds. Randomized augmentations such as adding noise, reverberation, and RawBoost are applied separately to enrollment and trial utterances during training to improve robustness.

3. Architecture / algorithm: Two LALMs are evaluated: SALMONN-7B, which concatenates enrollment and trial waveforms with a 1-second silence gap as single input, and Qwen2-Audio-7B, which supports multi-audio input processing. Both models combine audio encoders (Whisper, BEATS for SALMONN; transformer-based encoder for Qwen-Audio) and large language model backbones (Vicuna for SALMONN). LoRA adapters are applied to attention layers for efficient task-specific fine-tuning.

A composite objective with three heads is used: a 3-class softmax cross-entropy for SASV label prediction (target/non-target/spoof), an additive angular margin (AAM) loss applied only on bona fide samples for speaker embedding discrimination to improve ASV performance, and a separate binary cross-entropy loss for spoof detection. The total loss is weighted sum with tunable coefficients. This multi-headed approach balances the competing goals of speaker verification and spoof detection.

Hard sample mining identifies difficult enrollment-trial pairs with confusing similarities or spoofing characteristics to repeatedly train on boundary cases, improving discrimination.

4. Training regime: Models are trained using AdamW optimizer with learning rate 1e-5, batch size 1024, over 10–20 epochs with 1000 warmup steps. LoRA rank 128, alpha 256, and dropout 0.05 are used. Data augmentation is active during training. Reasoning supervision is added on a filtered subset (~90K pairs) where chain-of-thought (CoT) rationales are generated by a stronger cold-start model, filtered for prediction correctness and rationale consistency (Jaccard similarity > 0.35).

5. Evaluation protocol: Evaluation follows the ASVspoof 5 Track 2 open condition on a 20K stratified subsample. Metrics reported include accuracy, speaker verification accuracy, spoof detection accuracy, equal error rate (EER), and minimum detection cost function (min a-DCF). The composite SASV score is derived from the posterior probability of the "target" token output by the LALM. Comparison baselines include conventional fusion systems combining ECAPA-TDNN ASV embeddings with WavLM or W2V2-AASIST spoof countermeasures, fused by score power-fusion or decision trees.

6. Reproducibility: Code and model weights are not explicitly mentioned as publicly released; the ASVspoof 5 and VoxCeleb datasets are publicly accessible. Model adaptation depends on LoRA; reasoning data filtered from LALM-generated CoT traces on large-scale training data. Details on hyperparameters and preprocessing are thoroughly documented.

Concrete example: For a given enrollment and trial audio pair, the SALMONN model concatenates them with silence gap, extracts audio-conditioned embeddings via QFormer, then passes through the model to output probability distributions over three decision tokens. Training updates LoRA adapters and classifier heads via the weighted 3-loss objective with occasional hard mining of difficult pairs. Evaluation computes metrics by thresholding the target probability across the test set.

Reasoning supervision involves additionally training the model to output interpretable textual rationales in a predefined format (<think> cues, <reasons>, and <answer>), trained jointly to improve auditing capability though without uniformly improving accuracy.

Technical innovations

• Formulating SASV as a unified three-way classification task for LALMs with multi-headed loss functions optimizing speaker discrimination and spoof detection jointly.
• Applying parameter-efficient LoRA adaptation to large audio language models for efficient SASV fine-tuning.
• Introducing hard-sample mining strategy focused on hard yes/no/spoof pairs dynamically during training to improve boundary detection.
• Incorporating chain-of-thought (CoT) style reasoning supervision and reinforcement learning with group relative policy optimization (GRPO) to elicit interpretable decision rationales alongside classification.
• Systematic comparison of single unified LALM models against modular cascade baselines on challenging benchmarks with detailed ablations.

Datasets

• ASVspoof 5 — large-scale dataset with genuine, human impostor, and spoof audio samples — public
• VoxCeleb — large-scale genuine speaker dataset used for augmenting bona fide samples — public

Baselines vs proposed

• ECAPA2 + WavLM score fusion: accuracy = 79.78% vs SALMONN LoRA SFT + all adaptations: 89.30%
• ECAPA2 + W2V2-AASIST decision tree fusion: accuracy = 86.67% vs SALMONN LoRA SFT + all: 89.30%
• ECAPA2 + W2V2-AASIST threshold fusion: accuracy = 86.40% vs SALMONN LoRA SFT + all: 89.30%
• SALMONN hard-label model: accuracy = 86.02% vs conventional fusion baseline (same 90K pairs) accuracy = 72.00%
• SALMONN CoT-SFT reasoning-trace model: ASV accuracy = 93.95%, CM accuracy = 72.55%, overall accuracy = 83.92%
• SALMONN GRPO model: CM accuracy = 89.04% (highest among reasoning models), overall accuracy = 84.73%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14753.

Fig 1: Prompt variants for LALM-based SASV. Top: direct decision over

Fig 2: Overview of the proposed SALMONN-based SASV model.

Limitations

• Zero-shot performance of LALMs on SASV is near chance, requiring task-specific fine-tuning.
• Reasoning traces generated by CoT training are prone to hallucination and lack systematic human evaluation for quality and grounding.
• LALM-based SASV models demand significantly higher computational resources than conventional modular fusion pipelines.
• Trade-offs exist between optimizing speaker verification accuracy and spoof detection; improving one can degrade the other.
• Softmax decision token posteriors from LALMs are poorly calibrated for threshold-based metrics like EER.
• The reasoning supervision dataset is relatively small (90K pairs) compared to full training sets (1.8M pairs), possibly limiting benefits of reasoning training.

Open questions / follow-ons

• How can LALM-based SASV models improve calibration of softmax decision tokens for better threshold-based metrics such as EER?
• Can larger-scale or higher-quality human-annotated reasoning trace datasets improve interpretability and robustness without sacrificing accuracy?
• What are efficient methods to reduce the computational burden of LALM-based SASV models for practical deployment?
• How do LALM-based SASV models perform under diverse real-world conditions such as channel mismatch, background noise, and new spoofing techniques beyond the ASVspoof 5 benchmark?

Why it matters for bot defense

This work is directly relevant to bot-defense and CAPTCHA practitioners focused on voice authentication security. It demonstrates that large audio-language models, though not natively designed for spoofing-aware verification, can be adapted via targeted fine-tuning to achieve competitive joint speaker verification and spoofing detection in a single unified model. The approaches described, especially multi-headed losses balancing speaker discrimination and spoof detection, as well as hard-sample mining strategies, offer useful insights for developing more robust and interpretable voice authentication systems.

However, adopting LALMs for SASV in production bot-defense scenarios must also consider their significant computational cost and challenges with score calibration. The generation of natural-language reasoning traces provides a promising auditing tool to validate decisions beyond black-box score outputs, which could improve human oversight of voice verification systems vulnerable to deepfake attacks. Overall, this work suggests a conceptual shift from separate ASV and spoof detection modules toward integrated, auditable LALM-based solutions that may inspire future CAPTCHAs leveraging voice.

Cite

@article{arxiv2607_14753,
  title={ Large Audio Language Models for Spoofing-Aware Speaker Verification },
  author={ Sofya Savelyeva and Mariia Perunova and Evgeny Kushnir and Artem Dvirniak and Dmitrii Korzh and Oleg Y. Rogov },
  journal={arXiv preprint arXiv:2607.14753},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14753}
}

Read the full paper

• arXiv:2607.14753 (abstract)
• arXiv:2607.14753 (PDF)