Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas

Source: arXiv:2607.02504 · Published 2026-07-02 · By Yuxuan Li, Lingxi Xie, Xinyue Huo, Jihao Qiu, Jiacheng Shao, Pengfei Chen et al.

TL;DR

This paper addresses the challenging task of speaker recognition in long-form TV dramas, where accurately attributing each spoken utterance to the correct character is essential for deeper video understanding and storyline comprehension. The authors introduce DramaSR-532K, a large-scale benchmark dataset with over 532K annotated utterances spanning more than 900 unique characters across 13 TV dramas totalling over 525 hours of video. This dataset captures the complexities of multimodal reasoning, requiring synthesis of acoustic, visual, and linguistic cues, including off-screen speaker scenarios and overlapping speech.

To tackle these challenges, the paper proposes DramaSR-LRM, a large reasoning model framework that autonomously leverages multiple multimodal tools—voiceprint similarity, hierarchical video captioning, and character relational reasoning—within an iterative refinement loop guided by supervised fine-tuning and reinforcement learning. Experimental results on the held-out test set (428K utterances) show DramaSR-LRM surpasses strong label propagation and speaker diarization baselines by 2.3% overall accuracy, with larger relative gains on short utterances (up to 9.2%) where traditional acoustic biometrics struggle. The approach demonstrates improved robustness and reasoning-driven fusion of multimodal evidence to disambiguate speaker identity in complex narrative contexts.

Key findings

• DramaSR-532K benchmark contains 532,000 annotated utterances over 13 TV dramas totaling 525 hours and 900+ primary characters.
• Label propagation baseline achieves 85.49% utterance-level accuracy on the test set (428K utterances).
• DramaSR-LRM raises accuracy to 87.79%, a 2.3% absolute improvement over label propagation.
• Largest accuracy improvements occur on very short utterances (<0.5s), with a 9.2% gain from 67.45% to 76.65%.
• DramaSR-LRM also improves robustness on low baseline dramas like 'Lost' (+5.16%) and 'Qin Empire 2' (+4.06%).
• Facial-aware guess baseline only achieves 22.54% accuracy, demonstrating task difficulty.
• Use of multimodal tools (voice similarity, video captioning, character relation) integrated via large reasoning model enables effective reasoning across visual, acoustic, and linguistic signals.
• Reinforcement learning with Group Relative Policy Optimization further improves reasoning consistency beyond supervised fine-tuning.

Threat model

The adversary is implicit: complex multi-character interactions, noisy acoustic conditions, overlapping speech, occluded or off-screen speakers, and ambiguous dialogue references serve as natural sources of error. The model assumes access to segmented utterances, synchronized transcriptions, and candidate character lists, but must overcome incomplete visual or acoustic evidence and ambiguous social context to attribute speakers correctly.

Methodology — deep read

The paper defines the speaker recognition task in long-form TV dramas as an open-set classification problem under strong multimodal contextual dependencies. The threat model implicitly assumes multiple interacting characters, noisy acoustic environments, partial or missing visual cues, and complex dialogue relations, but the candidate character list per drama is provided.

Data comes from 13 long-form TV dramas (3 English + 10 Chinese), totaling 525 hours. Utterances are temporally segmented and synchronized with OCR-extracted hard subtitles, further refined by multimodal large language models. Character libraries are constructed by extracting actor-role mappings from credits and web data plus extensive visual face detection/recognition samples. The dataset contains 532K utterances indexed to 900+ named characters plus ancillary ones. Human-in-the-loop annotation refines an initial label propagation baseline, yielding an estimated final label noise of 0.5%.

The baseline label propagation uses a spatiotemporal neighborhood assumption: acoustic embeddings (192-d from ERes2Net) are extracted per utterance, and seed voiceprint clusters per character are initialized via visual face presence within ±30s windows. Affinity propagation with iterative clustering assigns utterances to speakers based on cosine similarity, pruning low-confidence samples to [UNKNOWN].

DramaSR-LRM builds on this initialization with a reasoning model (Qwen3-8B backbone) trained to iteratively refine speaker attributions by querying three multimodal tools: (a) voiceprint similarity matrix computed per utterance-character pair; (b) hierarchical video captioning generating dense local clip-level and high-level segment summaries using Qwen3-VL-32B and Qwen3-32B models; (c) dynamic character relational knowledge extracted from dialogue to model social relations (e.g., familial terms).

Training proceeds in two stages: first supervised fine-tuning (SFT) on 10K chain-of-thought reasoning trajectories generated by a prompted Gemini-3-Pro teacher model, by selectively focusing on hard and ambiguous utterances. The LRM learns to autonomously call tools and output reasoning paths justifying speaker choices. To improve robustness, borderline samples are artificially perturbed to force decision correction beyond acoustic cues. Then reinforcement learning (RL) with Group Relative Policy Optimization (GRPO) optimizes the LRM’s decision policies using reward functions incorporating prediction accuracy and output format adherence.

At inference, DramaSR-LRM iteratively refines utterance-level labels by repeatedly running the LRM on sliding windows of dialogue and updated tool outputs, allowing resolution of multi-character dependencies and uncertain cases. The video caption data is precomputed once due to cost, while voice similarity and relational context are updated each iteration.

Evaluation uses utterance-level accuracy against human-verified labels across 11 held-out dramas (428K utterances), reporting breakdowns by utterance length, speaker density, language, and visual occlusion. Several baselines—facial guess, pyannote diarization, label propagation, Qwen3-8B raw and SFT—serve as comparison points. Ablations confirm that the multimodal reasoning pipeline and RL refinement provide consistent gains, especially for short utterances where acoustic identification is weak.

Code and data are planned for public release, but at present, exact frozen checkpoints or reproducibility details are not documented in full. The paper provides example chain-of-thought prompts showing LRM reasoning with tool calls on concrete utterances.

Technical innovations

• Creation of DramaSR-532K, a large-scale multimodal speaker recognition benchmark with 532K utterances and 900+ characters from long-form TV dramas.
• Introduction of DramaSR-LRM, a large reasoning model framework that autonomously integrates multimodal tools (voiceprint similarity, hierarchical video captioning, character relational reasoning) for speaker attribution.
• Use of iterative label refinement combining a label propagation baseline with the LRM to synthesize multimodal cues dynamically and resolve ambiguities.
• Training paradigm combining supervised fine-tuning guided by chain-of-thought rationales from Gemini-3-Pro and reinforcement learning via Group Relative Policy Optimization to optimize reasoning consistency under uncertainty.

Datasets

• DramaSR-532K — 532,000 utterances — 13 long-form TV dramas, 525 hours, proprietary
• A Lifelong Journey — ~50,000 utterances — used for supervised fine-tuning
• Empresses in the Palace — ~50,000 utterances — used for reinforcement learning

Baselines vs proposed

• Facial-aware guess: accuracy = 22.54% vs DramaSR-LRM: 87.79%
• pyannote diarization (label-aware): 79.82% vs DramaSR-LRM: 87.79%
• Label Propagation baseline: 85.49% vs DramaSR-LRM: 87.79%
• Qwen3-8B direct use: 27.40% vs DramaSR-LRM: 87.79%
• Qwen3-8B + SFT: 75.22% vs DramaSR-LRM: 87.79%
• Qwen3-8B + SFT + confidence sampling: 82.70% vs DramaSR-LRM w/ confidence sampling: 87.79%
• On very short utterances (<0.5s): label propagation 67.45% vs DramaSR-LRM 76.65% (9.2% absolute gain)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02504.

Fig 1: How the DramaSR-532K benchmark was established. We extract (1) transcript data from OCR, (2) cast information from

Fig 2 (page 3).

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• The benchmark and evaluation rely on pre-segmented utterances and pre-defined candidate speaker lists, not addressing end-to-end speaker diarization including segmentation.
• Visual spatiotemporal neighborhood assumption favors on-screen speakers, possibly underrepresenting off-screen speech and narration difficulty.
• Data annotation, while high quality, depends on a semi-automated pipeline seeded with label propagation, which risks biasing dataset labels.
• The iterative reasoning model depends on expensive pretrained LLM backbones and hierarchical captioning, which may limit scalability or real-time applications.
• No detailed adversarial robustness or analysis under adversarial audio or visual perturbations is included.
• Reproducibility is partially hindered by reliance on proprietary pretrained models (Gemini-3-Pro, Qwen3) and incomplete public release at the time of writing.

Open questions / follow-ons

• How to extend this framework to end-to-end speaker diarization, including automatic utterance segmentation and candidate discovery?
• Can the model effectively handle off-screen narration and speakers absent from the visual field without reliance on candidate lists?
• How would adversarial attacks or environmental noise affect the robustness of the multimodal reasoning model?
• Can open-world modeling be scaled to generate linguistic descriptions for unknown or ancillary speakers dynamically?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper provides a compelling example of leveraging multimodal large reasoning models to disambiguate complex speaker identity tasks in noisy, real-world scenarios. The use of integrated acoustic, visual, and linguistic signals mediated via tool-use and reasoning is directly relevant for designing robust user verification systems that go beyond simple biometrics, particularly in contexts where isolated cues are weak or misleading, such as short utterances or occluded views.

Furthermore, the iterative refinement framework using supervised fine-tuning guided by chain-of-thought rationales and reinforcement learning could inspire similar approaches to continuously improve recognition accuracy in adaptive bot detection systems. However, the computational complexity and large model reliance may present engineering challenges for real-time CAPTCHA or bot-defense applications. Overall, the paper’s methodology exemplifies the potential of large reasoning-centric multimodal models to enhance identity and behavior recognition in complex multimedia environments.

Cite

@article{arxiv2607_02504,
  title={ Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas },
  author={ Yuxuan Li and Lingxi Xie and Xinyue Huo and Jihao Qiu and Jiacheng Shao and Pengfei Chen and Jiannan Ge and Kaiwen Duan and Qi Tian },
  journal={arXiv preprint arXiv:2607.02504},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02504}
}

Read the full paper

• arXiv:2607.02504 (abstract)
• arXiv:2607.02504 (PDF)