SLT 2026 REAL-TSE Challenge: Real-world Target Speaker Extraction from Conversational Recordings

Source: arXiv:2607.15198 · Published 2026-07-16 · By Shuai Wang, Zihan Qian, Ke Zhang, Jiangyu Han, Zikai Liu, Xiaoyang Yu et al.

TL;DR

The REAL-TSE Challenge presented at IEEE SLT 2026 addresses the task of target speaker extraction (TSE) in real-world conversational recordings featuring Mandarin and English. Unlike prior benchmarks relying on simulated mixtures of clean read speech, REAL-TSE evaluates extraction under practical conditions of natural speech overlap, reverberation, ambient noise, channel mismatch, and conversational dynamics. The challenge comprises two tracks: an Online track with strict low-latency constraints (<100 ms delay) and an Offline track allowing full utterance context and arbitrary inference cost. Systems are judged on four complementary metrics — Token Error Rate (TER), Speaker Similarity (SpkSim), perceptual quality (DNSMOS-P808), and target speaker activity F1 — capturing intelligibility, speaker identity, audio quality, and temporal accuracy of speech activity. The challenge released a diverse real conversation dataset totaling 11.3 hours, collected from multiple domains with multi-device recordings to test robustness. Baseline models based on BSRNN extractors trained only on simulated Libri2Mix highlighted the difficulty of real-world data, while 24 teams submitted improved systems leveraging real data adaptation, simulation, multi-objective losses, and post-processing. Condition-wise and metric correlation analyses revealed challenges in channel mismatch, real-data generalization, and evaluation metric reliability. The paper offers insights on tuning system design for deployment constraints and the complexity of multi-dimensional TSE evaluation in natural environments.

Key findings

• REAL-TSE dataset contains 6,991 trials over 2,309 mixtures totaling 11.3 hours of real Mandarin and English conversational audio, with natural overlap and environmental noise.
• Baseline BSRNN systems trained solely on synthetic Libri2Mix data perform poorly on REAL-TSE, with Token Error Rates (TER) above 0.65 and speaker similarity scores around 0.5 on development and evaluation sets.
• Top challenge submissions improve all metrics significantly over baselines but show trade-offs: no system dominates TER, SpkSim, DNSMOS-P808, and target-activity F1 simultaneously.
• Real-data adaptation and pseudo-labeling consistently outperform training on synthetic data alone, underscoring the value of authentic conversational recordings for TSE.
• Mixure microphone type impacts performance significantly; the H2 far-field microphone yields noticeably lower scores than H1 or phone mics, revealing challenges in far-field robustness.
• The target speaker activity ratio correlates strongly (+0.83 Pearson) with F1 detection accuracy, while correlation with TER intelligibility is weaker (-0.38), indicating activity ratio explains activity detection better than intelligibility.
• Perceptual quality metric DNSMOS-P808 correlates better with human MOS (LCC=0.46) than DNSMOS-OVRL (LCC=0.17), exposing vulnerabilities to metric-specific over-optimization during system tuning.
• Reported online track latencies largely fall within the 20-100 ms budget, but measured latency sometimes diverges from analytical estimates due to undocumented buffering or normalization.

Threat model

n/a — The paper does not define an explicit adversarial threat model. The focus is on robust extraction of a target speaker from complex real-world mixtures under natural conditions, assuming enrollment utterances and no adversarial attacks or spoofing.

Methodology — deep read

1. Threat Model & Assumptions: The challenge focuses on isolating a known target speaker from multi-talker mixtures using enrollment utterances. Adversaries or attackers are not modeled explicitly; the challenge simulates real-world conditions with reverberation, noise, overlap, channel mismatch, and conversational unpredictability. The systems must perform extraction independently per mixture-enrollment pair without reference to metadata or cross-sample info.

2. Data: The dataset consists of real Mandarin and English conversational recordings. The dev set (1,991 samples) is drawn from REAL-T, AMI, AISHELL-4, AliMeeting, DipCo, CHiME6, etc., offering 2.6 hours mixture audio and 0.67 hours enrollment audio. An evaluation set of 5,000 samples was split into EVAL-1 (2,000 samples from seen domains) and EVAL-2 (3,000 samples newly collected from diverse real scenarios like meetings, cafes, vehicles, with multiple microphone types). Mixture durations average ~17-18 seconds; enrollment segments average ~9-11 seconds. Overlap and target speaker activity ratios vary naturally between 0.18-0.99 and 0.27-1.0 respectively.

3. Architecture/Algorithm: Baseline models use a Band-Split RNN (BSRNN) extractor conditioned on speaker embeddings from ECAPA-TDNN or TF-Map features combined with speaker context. Models are either causal or non-causal by replacing bidirectional LSTMs with unidirectional ones. The challenge submissions explored varied extractors including BSRNN and TF-GridNet, multiple conditioning methods (global embeddings, frame-level, prefix tokens), and sometimes diarization-aware or generative refinement modules.

4. Training Regime: Baselines were trained for 150 epochs on fully overlapped synthetic Libri2Mix-100 mixtures at 16kHz, intentionally a lower bound due to data mismatch. Submitted systems trained on diverse synthetic and real data using multi-stage training, leveraging pseudo-label generation, model averaging, and rounds of self-training for real-data adaptation. Objective functions combined reconstruction (SI-SDR/SI-SNR, multi-res STFT) with auxiliary speaker similarity, ASR/token-level, VAD/activity, and perceptual losses. Training hyperparameters and data splits varied by team.

5. Evaluation Protocol: Systems were evaluated on four metrics: Token Error Rate (TER) from Zipformer ASR, Speaker Similarity (SpkSim) via cosine similarity of WeSpeaker ResNet-34 embeddings, perceptual quality DNSMOS-P808, and target-speaker activity detection F1 using FireRedVAD. Each system produces one output mixture-enrollment pair at a time, without cross-sample info. Online track systems must respect <100 ms algorithmic latency verified by a perturbation-based latency test that accounts for buffering and normalization delays. Final rankings average ranks across the four metrics.

6. Reproducibility: The challenge released baseline code and toolkits via WESEP for simulation, training, evaluation, and submission checking. The dataset splits for development and evaluation are public, with some evaluation sets withheld or anonymized to prevent overfitting. Model checkpoints for baselines are public but submitted models remain with teams. The open training data policy encourages use of public corpora with transparent data sources.

Example End-to-End: A submitted offline system first simulates training mixtures by mixing multi-speaker recordings with noise and RIRs, then adapts with pseudo-label generated teacher models on real conversational mixtures. Enrollment segments are encoded into embeddings conditioning a TF-GridNet extractor that processes full utterance mixtures to produce enhanced target speaker speech. The output is evaluated by Zipformer ASR for TER, WeSpeaker embeddings for SpkSim, DNSMOS-P808 for quality, and FireRedVAD for target voice activity detection F1. This procedure repeats over the 3,000 EVAL-2 unseen real mixtures collected from diverse devices and environments, with results aggregated for final ranking.

Technical innovations

• Separate online and offline TSE tracks with strict latency constraints for streaming vs full-context processing, reflecting realistic deployment scenarios.
• Multi-dimensional evaluation combining intelligibility (TER), speaker consistency (SpkSim), perceptual quality (DNSMOS-P808), and target-activity detection (F1), better capturing real TSE quality trade-offs.
• Use of real conversational multi-device recorded Mandarin and English datasets (REAL-T) with natural overlap, reverberation, and noise challenging synthetic training paradigms.
• Application of real data pseudo-labeling and multi-round self-training to bridge domain gaps from clean synthetic to realistic mixtures, improving real-world generalization.

Datasets

• REAL-TSE Challenge dataset — 6,991 mixture-enrollment pairs, 11.3 hours mixture audio — real Mandarin/English conversational recordings from REAL-T, AMI, AliMeeting, DipCo, CHiME6, and newly collected unseen sets
• Libri2Mix-100 — synthetic fully overlapped English read speech mixtures, 150 epochs training used for baselines

Baselines vs proposed

• BSRNN_EMB: DEV TER=0.693, SIM=0.501; EVAL-1 TER=0.816, SIM=0.507; EVAL-2 TER=0.838, SIM=0.357
• BSRNN_TFMAP_CAUSAL: DEV TER=0.652, SIM=0.535; EVAL-1 TER=0.801, SIM=0.553; EVAL-2 TER=0.808, SIM=0.391
• Top submitted systems outperform baselines across all metrics, but with complementary strengths and no single system uniformly best (Fig. 1).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15198.

Fig 2: Scenario-wise performance on the EVAL-2 set, computed by averaging all valid submissions separately for CN and EN and then taking a CN/EN

Fig 3: Enrollment-device by mixture-device performance matrix on EVAL-2.

Limitations

• Baseline models trained only on synthetic Libri2Mix represent a lower bound and do not reflect realistic data diversity or channel conditions.
• Challenge does not model adversarial or attack scenarios, focusing on robustness to natural real-world conditions only.
• Metric-specific over-optimization evident, especially with DNSMOS-OVRL and speaker similarity, limiting reliability of fully automatic evaluation.
• Evaluation scenario labels (meeting, restaurant, home, car) only provide coarse-grained difficulty diagnostics, insufficient to explain performance variability.
• Open training policy results in varied training data and procedures among teams, complicating exact attribution of performance gains.
• Latency measurements highlight inconsistencies between analytical and measured delay, indicating need for unified reporting standards.

Open questions / follow-ons

• How to design evaluation metrics and protocols that are robust to over-optimization and adversarial exploitation of neural quality estimators in TSE challenges?
• What extractor architectures and conditioning mechanisms best synergize with extensive real-data adaptation to close the gap with ideal upper bounds in real conversational TSE?
• How to model and mitigate channel mismatch effects more effectively, especially for far-field and cross-device enrollment vs mixture conditions?
• Can fine-grained labels or difficulty measures at the utterance or speaker interaction level better predict system performance and guide adaptive TSE methods?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the REAL-TSE Challenge offers key insights into the challenges of target speaker extraction in realistic multi-speaker, noisy, reverberant, and multi-device audio scenarios—conditions analogous to those seen in multi-user or adversarial voice environments. The multi-objective scoring framework highlights the complexity of balancing intelligibility, speaker identity, perceptual quality, and correct activity detection simultaneously, a useful reminder that exploiting single-metric optimizations may be risky. The open training policy and demonstrated gains from real-data adaptation and pseudo-labeling stress the importance of domain-relevant data and robust adaptation for reliable speaker extraction, informing design of audio-based bot detectors or voice biometric defenses. The challenge’s latency controls and streaming constraints also emphasize practical deployment considerations where low-delay but accurate speaker extraction is needed to interact in real time.

Cite

@article{arxiv2607_15198,
  title={ SLT 2026 REAL-TSE Challenge: Real-world Target Speaker Extraction from Conversational Recordings },
  author={ Shuai Wang and Zihan Qian and Ke Zhang and Jiangyu Han and Zikai Liu and Xiaoyang Yu and Haoyu Li and Marc Delcroix and Kai Yu and Lei Xie and Ming Li and Haizhou Li },
  journal={arXiv preprint arXiv:2607.15198},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15198}
}

Read the full paper

• arXiv:2607.15198 (abstract)
• arXiv:2607.15198 (PDF)