Improving the performance of an ASV system using hybrid speech features

Source: arXiv:2607.20706 · Published 2026-07-22 · By Stanisław Ciszkiewicz, Artur Janicki

TL;DR

This paper addresses the challenge of improving Automatic Speaker Verification (ASV) system performance under noisy acoustic conditions by employing hybrid acoustic feature sets. ASV systems are vulnerable to environmental noise and spoofing attacks, and traditional features like Mel-Frequency Cepstral Coefficients (MFCC) degrade significantly in noise. The authors investigate combining classical cepstral features such as MFCC, Constant Q Cepstral Coefficients (CQCC), and Power-Normalized Cepstral Coefficients (PNCC), which is known for noise robustness, with an innovative nonlinear vocal fold model-based feature called RAB. Using the Google Speech Commands dataset, the study evaluates individual and hybrid feature sets under clean and noisy (babble and Volvo noise) conditions, measuring Equal Error Rate (EER) as the performance metric.

Results show that in clean conditions, PNCC and MFCC individually already achieve very low EER (~0.1%), with hybrid sets offering marginal gains. However, under noisy conditions, systems based on PNCC outperform others significantly. Crucially, adding the RAB descriptor to PNCC reduces EER by up to 4 percentage points in babble noise and 7 points in Volvo noise at low SNRs (0 dB). The findings demonstrate that the nonlinear vocal fold dynamics captured by RAB complements cepstral features, enhancing robustness to noise and improving speaker verification accuracy. The authors note that these promising results warrant validation on larger speaker sets and on more advanced neural backend models in future work.

Key findings

• Under clean acoustic conditions, PNCC and MFCC features achieve near-perfect performance with EERs of 0.10% and 0.11% respectively, leaving little room for improvement with hybrid features.
• In babble noise at 0 dB SNR, the PNCC + RAB hybrid reduces EER from 33.19% (PNCC alone) to 29.17%, a 4.02 percentage point improvement.
• In Volvo noise at 0 dB SNR, PNCC + RAB achieves 24.02% EER vs 26.70% for PNCC alone, an improvement of 2.68 percentage points; other hybrid sets with RAB show similar trends.
• The RAB descriptor alone performs poorly in noisy conditions, showing EERs above 30% under babble and Volvo noise at low SNRs.
• Feature sets containing PNCC consistently outperform MFCC and CQCC under noisy conditions across all tested SNR levels (0–20 dB).
• Hybrid features combining PNCC, CQCC, and RAB yield further EER reductions compared to single feature sets in adverse noise scenarios (e.g., CQCC + PNCC + RAB achieves 24.27% EER at 0 dB in Volvo noise).
• Adding RAB coefficients consistently lowers EER across SNR levels compared to feature sets without RAB, as shown in Figures 4 and 5.
• Using 16 Gaussian components for speaker GMM models with MAP adaptation on UBM achieves stable verification results on concatenated sequences of five utterances each.

Threat model

The adversary is modeled as environmental acoustic noise sources corrupting the speaker verification signal, specifically babble noise and vehicle engine noise, which degrade verification accuracy. The attacker does not have capabilities for spoofing, replay attacks, or direct manipulation of system internals. The system is assumed to be trained on clean speech and tested on noisy speech to evaluate robustness.

Methodology — deep read

1. Threat model & assumptions: The study focuses on the robustness of ASV systems against acoustic noise degradation rather than active adversarial spoofing attacks. The attacker in this context is an environmental noise source (babble, vehicle engine noise) that deteriorates the input signal quality. There is no explicit modeling of malicious spoofing or replay attacks in this evaluation.

2. Data: The Google Speech Commands dataset (approx. 65,000 short English word recordings) was used, selecting 30 speakers with the most recordings. For each speaker, 100 recordings were randomly selected and concatenated into sequences of 5 utterances to improve stability. A 4:1 train-test split resulted in 16 training and 4 testing samples per speaker. Testing compared every test recording against every speaker model to generate genuine and impostor score distributions (total 3,600 trials per condition). Noise was artificially added for noisy conditions via NoiseX-92 corpus with babble and Volvo noises at SNRs of 0, 5, 10, 15, and 20 dB.

3. Architecture / Algorithm: The ASV system pipeline included feature extraction and GMM classification. Four feature types were evaluated: classical MFCC, CQCC (uses Constant-Q transform instead of DFT), PNCC (power normalized Cepstral coefficients with Gammatone filterbank), and the nonlinear RAB descriptor based on a vocal fold dynamic model producing 3 parameters (R, A, B). Hybrid feature sets were constructed by concatenating feature vectors, e.g., PNCC+RAB. Features were normalized using train-set statistics. Speaker models used Gaussian Mixture Models with 16 Gaussian components. A Universal Background Model was trained on all speakers, and speaker-specific models adapted with MAP adaptation.

4. Training regime: Models trained on clean speech data only. Training details such as batch size, epochs, or optimizer are not applicable due to model type (GMMs). The GMM-UBM training roughly follows classical protocols with no mention of random seed control.

5. Evaluation protocol: Equal Error Rate (EER) was used as main metric for speaker verification performance. Tests were performed under clean and noisy (babble and Volvo noise) conditions. The test set consisted of impersonation scenarios (impostor samples) and genuine samples by testing against all speaker models. The use of concatenated utterances helps reduce variability in trial scores.

6. Reproducibility: No code release or model weights provided. Dataset is publicly available. The methodology is described clearly enough to be replicated using standard ASV tools. However, hyper-parameter details (e.g., GMM initialization) and random seed usage are not specified. Feature extraction details generally follow cited prior works.

A concrete example: For a test utterance corrupted by babble noise at 0 dB, PNCC features were extracted and concatenated with RAB coefficients estimated via nonlinear vocal fold model parameter regression. This hybrid vector was fed into the MAP-adapted GMM speaker model to compute log-likelihood scores. The scores were compared to impostor trials to compute EER. Across all such trials for the 30 speakers at this noise level, EER decreased from 33.19% for PNCC alone to 29.17% for PNCC+RAB, showing improved robustness via hybrid features.

Technical innovations

• Application of the nonlinear vocal fold dynamic model-based RAB descriptor as an acoustic feature for ASV, complementing standard cepstral features.
• Systematic evaluation of hybrid feature sets combining PNCC, CQCC, MFCC, and RAB descriptors to improve noise robustness in a classical GMM-UBM framework.
• Demonstration that adding only three RAB coefficients to PNCC features reduces EER by up to 7 percentage points under severe acoustic noise (Volvo noise at 0 dB SNR).
• Use of concatenated short utterance sequences to stabilize verification results on limited-vocabulary, text-dependent speaker verification tasks.

Datasets

• Google Speech Commands — approx. 65,000 recordings (publicly available)
• NoiseX-92 — noise corpus for babble and volvo noise simulation (publicly available)

Baselines vs proposed

• PNCC alone (babble noise, 0 dB SNR): EER = 33.19% vs PNCC+RAB: 29.17%
• PNCC alone (volvo noise, 0 dB SNR): EER = 26.70% vs PNCC+RAB: 24.02%
• MFCC alone (clean): EER = 0.11% vs MFCC+PNCC+RAB: 0.06%
• CQCC alone (babble noise, 0 dB SNR): EER=46.74% vs CQCC+PNCC+RAB: 34.77%
• RAB alone (babble noise, 0 dB SNR): EER=45.01% vs PNCC+RAB: 29.17%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.20706.

Fig 1: ASV system architecture used in experiments

Fig 2: CQCC features extraction schema

Fig 3: Simple schema of a nonlinear model of the vocal folds (based on [11])

Fig 4: shows the dependence of EER on SNR for various

Fig 5: Impact of RAB on EER across SNR levels for various feature sets –

Limitations

• Evaluation restricted to classical GMM-UBM backend; no state-of-the-art neural architectures tested, limiting generalization to modern ASV systems.
• Limited speaker population (30 speakers) from a relatively clean, constrained dataset with short utterances concatenated. Larger, more diverse datasets needed.
• No explicit testing against spoofing or adversarial attacks; focus was on additive noise robustness only.
• No cross-validation or random seed control details provided; stability of results under different splits unknown.
• RAB descriptor contributes only a small number of coefficients, possibly limiting its representational capacity against complex distortions.
• Noise types limited to babble and Volvo; generalization to other realistic acoustic scenarios not explored.

Open questions / follow-ons

• How does incorporating the RAB descriptor affect speaker verification performance in advanced neural network backends like ECAPA2 or CAM++?
• Can the gains from hybrid PNCC+RAB features be replicated on larger, more diverse speaker cohorts and truly real-world noisy conditions?
• What is the impact of RAB-augmented features on spoofing and replay attack resilience beyond noise robustness?
• Can more complex nonlinear vocal fold models or additional dynamic features further improve hybrid feature effectiveness?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work highlights the value of combining complementary acoustic feature sets to enhance voice biometric authentication robustness, especially under challenging acoustic noise. The demonstrated hybrid PNCC+RAB features reduce speaker verification error significantly in noisy environments, indicating that feature-level fusion with nonlinear vocal fold dynamic descriptors can harden ASV systems against environmental noise perturbations. However, these results are obtained using classical GMM-UBM backends and a limited speaker set; it remains to verify gains with contemporary neural embeddings and larger databases typically in deployed systems.

Therefore, CAPTCHA engineers considering voice-based authentication or fraud prevention via ASV should view hybrid feature extraction as a promising technique to improve noise robustness and reduce false accepts/rejects under realistic acoustic conditions. Incorporating nonlinear voice production characteristics via RAB descriptors could be an effective complement to cepstral features like PNCC. Rigorous evaluation in operationally relevant scenarios involving spoofing is a prudent next step before production adoption.

Cite

@article{arxiv2607_20706,
  title={ Improving the performance of an ASV system using hybrid speech features },
  author={ Stanisław Ciszkiewicz and Artur Janicki },
  journal={arXiv preprint arXiv:2607.20706},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20706}
}

Read the full paper

• arXiv:2607.20706 (abstract)
• arXiv:2607.20706 (PDF)