Evidence Subspace Projection: Measuring How Much Evidence Explains Deepfake Detection in Self-Supervised Speech Models
Source: arXiv:2607.11538 · Published 2026-07-13 · By Yixuan Xiao, Cheng-Wei Lin, Xin Wang, Yassine El Kheir, Arnab Das, Tim Polzehl et al.
TL;DR
This paper addresses the interpretability gap in audio deepfake detection systems that use self-supervised learning (SSL) models as feature extractors. While SSL embeddings are state-of-the-art for detection, it remains unclear which internal model signals correspond to detection decisions and how they relate to evidence factors like attack type or acoustic characteristics. To fill this gap, the authors propose Evidence Subspace Projection (ESP), a novel method that maps neuron activation patterns from SSL models into a shared space representing both detection labels (bonafide/spoof) and various evidence factors (attack, codec, silence, etc.). By projecting the detection decision vector onto these evidence subspaces, ESP quantifies how much each evidence factor explains the model’s decision. Applied to two major SSL models (XLSR and HuBERT) under frozen, fine-tuned, and post-trained settings across six datasets, the method confirms known dataset-specific shortcuts (e.g., silence structure) and reveals that fine-tuning can reduce undesirable alignment with within-spoof variation but may amplify reliance on signal-level artifacts under homogeneous data. Post-training further suppresses many biases, but silence cues remain persistent. Overall, the work provides a quantitative, neuron-level interpretability framework for understanding SSL-based audio deepfake detectors.
Key findings
- Frozen XLSR and HuBERT SSL models encode dataset-specific shortcuts, especially silence, with up to ~10% explanatory power (E∆/erank) on ASV19-test.
- Gender-related neuron activation patterns are nearly orthogonal to detection decision axes, with E∆/erank below 1.4% across datasets and models, indicating little gender bias.
- Within-spoof categories (vocoder, attacks) show undesirable high alignment (e.g., cosine similarity 0.608 between Spectral Filtering+OLA vocoder and bonafide on ASV19-dev in frozen XLSR), indicating model vulnerability to spoof-type confounds.
- Fine-tuning on low-diversity ASV19 data amplifies signal-level shortcuts like silence and codec, increasing E∆/erank by up to 8.1% for silence, whereas high-diversity ASV5 training decorrelates these shortcuts.
- Post-training on large-scale deepfake data reduces within-spoof confounds below frozen levels (e.g., Attacks E∆/erank reduces from 4.4% to 1.7% on ASV19) and most signal-level dependencies but does not fully remove silence shortcut.
- HuBERT tends to have stronger initial alignment with within-spoof variations and is less able to unlearn these biases after fine-tuning, reflected in consistently higher E∆/erank and error rates than XLSR.
- Models with lower within-spoof alignment (lower E∆/erank) generally achieve lower equal error rates (EER), supporting the metric’s relevance to robustness; this holds in 18 of 22 dataset-model combinations.
- Frequency and harmonic-vs-noise (HN) cues show inconsistent and dataset-specific explanatory power, generally lower than silence or metadata-driven factors.
Threat model
The study implicitly assumes adversaries generating diverse types of synthetic speech (e.g., TTS, voice conversion) possibly combined with adversarial perturbations. The models attempt to detect bona fide vs spoofed inputs, with adversaries aiming to produce spoofed audio that evades detection. The adversary cannot alter the SSL front-end architecture but can generate new attack types and affect signal-level properties. The research examines how SSL models internally respond to these threats but does not attempt active attack or adversarial evaluation.
Methodology — deep read
The study focuses on interpreting frozen and trained SSL audio feature extractors (XLSR and HuBERT) used in deepfake detection. The threat model is implicit, assuming adversaries create diverse synthetic audio attacks; the authors aim to analyze what internal neuron activations SSL models use to detect these attacks, focusing on detection interpretability rather than robustness per se.
Data comprises six public datasets: ASV19, ASV19-dev, ASV21LA, ASV21DF, ASV5, and ITW, covering synthetic audio generated by different vocoders, attacks, codecs, genders, and transmission conditions. Each dataset contains frame-level labels derived from HuBERT quantization tokens for neuron activation analysis.
The core architecture comprises frozen SSL Transformer front-ends with neuron activation extracted layer-wise. For fine-tuned models, a simple backend MLP maps SSL embeddings to a binary detection output. Fine-tuning occurs on either ASV19 or ASV5 training sets; post-training (PT) uses a large-scale deepfake corpus, but only for XLSR.
Neuron activation is binarized at the top 1% activation threshold per token. Activation coverage matrices are computed per label by averaging activation indicators across tokens and layers, generating a high-dimensional neuron-activation vector representation per label. Residual representations (one-vs-rest contrast vectors) are formed by subtracting the mean activation of other labels in the same dataset to isolate label-specific directions.
Evidence subspaces are constructed from collections of residuals for evidence factors (e.g., different vocoders, attacks, genders, silence vs speech segments). Singular value decomposition yields orthonormal bases for these subspaces. The detection decision axis (difference vector between spoof and bonafide residuals) is projected onto each evidence subspace, yielding scalar explanatory power scores (E∆), normalized by effective rank of the evidence subspace (erank). This quantifies how much the model’s detection decision coincides with cues from each evidence factor.
Training regimes use fine-tuning with loop-padded 4-second inputs, no augmentation, and a standard MLP backend. Evaluations occur on multiple held-out test sets including cross-dataset generalization tests. Equal error rate (EER) is reported as detection accuracy metric.
The method is reproducible with code and data releases referenced, including pretrained SSL models and public ASVspoof datasets. However, the PT XLSR model is trained on a large scale corpus not fully public.
An end-to-end example: For a given dataset (e.g., ASV19) and evidence group (e.g., Vocoder), neuron activations are collected from SSL across all samples of each vocoder label. Residual vectors representing contrast between one vocoder and others are formed. SVD produces a subspace capturing vocoder variation directionality in neuron space. The detection decision axis vector is then projected into this vocoder subspace, and the squared norm ratio after normalization yields E∆/erank vocoder score, indicating how much detection relies on vocoder differences. Comparing this across training conditions reveals how training reduces or amplifies reliance on vocoder cues.
Technical innovations
- Introduction of Evidence Subspace Projection: representing evidence factors and detection decisions as residual vectors in shared neuron activation space to quantitatively measure explanatory power.
- Application of neuron activation probability analysis to deepfake audio SSL models, enabling neuron-level interpretability distinct from feature-based or sample-based methods.
- Normalization by effective rank of evidence subspaces to fairly compare explanatory power across groups with varying intrinsic dimensionality.
- Demonstration that frozen SSL models encode dataset-specific shortcuts and within-spoof confounds before fine-tuning or post-training, revealing vulnerabilities at neuron level.
Datasets
- ASVspoof 2019 (ASV19) — public spoofing dataset with synthetic and converted speech
- ASVspoof 2021 LA (ASV21LA) — public dataset focusing on Logical Access spoofing
- ASVspoof 2021 DF (ASV21DF) — public Deepfake dataset
- ASVspoof 5 (ASV5) — large-scale and diverse spoofing dataset
- ASV19-dev — development split of ASV19, used as unseen test set
- ITW — in-the-wild deepfake speech evaluation set
Baselines vs proposed
- Frozen XLSR: Silence E∆/erank up to ~10% on ASV19-test vs post-trained XLSR Silence E∆/erank partially suppressed to ~2–5%.
- Frozen HuBERT: Higher within-spoof alignment E∆/erank than XLSR (e.g., Attacks ~30% vs 19% on ASV5), correlating with higher EER (HuBERT 22.47% vs XLSR 17.98%).
- Fine-tuned XLSR (FT-19) on ASV19: EER 0.25% on ASV19 vs Frozen XLSR EER not explicitly stated but lower performance implied.
- Fine-tuned HuBERT (FT-5) on ASV5: EER 10.05% vs XLSR FT-5 5.69%, showing XLSR’s advantage after fine-tuning.
- Post-trained XLSR reduces within-spoof E∆/erank from 4.4% (frozen) to 1.7% (PT) on ASV19 attacks.
- Gender E∆/erank remains near zero across frozen and trained models, confirming weak correlation with detection.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.11538.
Fig 2: E∆/erank change relative to frozen models with absolute values.
Fig 3: shows E∆/erank for all evidence groups. Despite differ-
Limitations
- No explicit adversarial robustness evaluation; focus is interpretability rather than attack resilience.
- Post-training evaluation only conducted on XLSR, no PT HuBERT model to compare training effects completely.
- Possible confounding between correlated evidence factors such as codec and transmission not fully disentangled.
- The method relies on top-1% neuron activation threshold; sensitivity to this hyperparameter is not extensively studied.
- Analysis focuses on neuron activation patterns but does not link directly to sonic artifacts or perceptual features explaining them.
- Cross-dataset generalization tested but full distributional shifts or real-world noisy conditions remain open questions.
Open questions / follow-ons
- Can evidence subspace projection be extended to actively guide training towards decorrelating undesirable shortcuts?
- How sensitive are neuron activation patterns and explanatory power scores to the neuron activation threshold (top 1%) and choice of layers?
- Can the approach differentiate finer-grained causal factors among correlated metadata and signal-level properties, e.g., codec vs transmission?
- How do these neuron-level explanatory findings correspond to perceptual audio features or human-understandable artifacts?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners focused on audio deepfake detection, this paper offers a novel quantitative interpretability method to understand which internal neuron patterns SSL models rely on when making detection decisions. Recognizing that fine-tuned models may inadvertently amplify dataset-specific artifacts like silence or codec signatures highlights the need for diverse training data to improve generalization and robustness. The evidence subspace projection framework could guide practitioners in auditing and diagnosing model biases that lead to overfitting or vulnerability to specific spoofing attacks. Although it does not propose a new detection model, its insights are valuable for improving SSL-based detector reliability and for research into explainable AI that can justify audio deepfake detection decisions in security-sensitive contexts. The persistent reliance on silence cues even after training adjustments suggests caution in deploying these models without further mitigation.
Cite
@article{arxiv2607_11538,
title={ Evidence Subspace Projection: Measuring How Much Evidence Explains Deepfake Detection in Self-Supervised Speech Models },
author={ Yixuan Xiao and Cheng-Wei Lin and Xin Wang and Yassine El Kheir and Arnab Das and Tim Polzehl and Sebastian Möller and Ngoc Thang Vu },
journal={arXiv preprint arXiv:2607.11538},
year={ 2026 },
url={https://arxiv.org/abs/2607.11538}
}