AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis

Source: arXiv:2607.15755 · Published 2026-07-17 · By Zhenqi Jia, Yuan Zhao, Aruukhan, Rui Liu, Haizhou Li

TL;DR

This paper addresses key limitations in Conversational Speech Synthesis (CSS) related to the restricted predefined emotion label space (typically 7 basic emotion categories) and the interference of redundant multimodal tokens in multi-turn dialogue history on context understanding. To overcome these, the authors propose AuEmoChat, a novel framework for authentic emotion understanding and rendering in CSS. AuEmoChat introduces AuEmoCodec, which learns a discrete, fine-grained authentic emotion token space from large-scale emotional speech data via finite scalar quantization (FSQ), going beyond limited coarse emotion categories. Additionally, AuEmoToMe, an authentic-emotion-guided token merging algorithm, reduces redundancy in multimodal dialogue context tokens, preserving emotion-relevant information. Combined with an autoregressive text-speech model and a novel Authentic Emotion Flow Matching mechanism that conditions on merged context, target authentic emotion tokens, and acoustic priors, the framework synthesizes more natural, contextually appropriate, emotionally expressive speech.

Extensive experiments on the NCSSD-EmCap dataset (~384 hours, 25 speakers, 18,580 dialogues) show that AuEmoChat significantly outperforms state-of-the-art CSS models on subjective metrics like naturalness (N-DMOS 4.171 vs 3.955) and emotion expressiveness (E-DMOS 3.979 vs 3.756), and objective metrics such as WER (9.14% vs 15.07%), Mel Cepstral Distortion (6.847 vs 7.686), speaker similarity, and emotion classification accuracy in both basic and authentic emotion spaces. Ablations confirm the necessity of authentic emotion tokenization, AuEmo-guided token merging, and flow matching conditioning for performance gains. The work advances authentic emotion representation and context modeling in CSS for user-agent interaction.

Key findings

• AuEmoChat improves Naturalness-DMOS from 3.955 (Chain-Talker) to 4.171 and Emotion-DMOS from 3.756 to 3.979 (Table 1), indicating more natural and emotionally expressive speech.
• Word Error Rate (WER) drops from 15.07% (Chain-Talker) to 9.14%, and Mel Cepstral Distortion (MCD) improves from 7.686 to 6.847, showing better pronunciation and acoustic fidelity.
• Speaker Similarity (SpkSIM) increases modestly from 77.56 to 78.03, confirming better speaker voice preservation.
• Emotion Accuracy (EmoACC) under 7-category space improves from 57.16% to 61.04%, and Authentic Emotion Accuracy (AuEmoACC) under learned token space rises from 24.12% to 28.71%, demonstrating improved authentic emotion modeling.
• Ablation removing AuEmoToMe token merging reduces N-DMOS by approximately 0.13 and E-DMOS by ~0.11, and worsens EmoACC and AuEmoACC by ~2-3%, indicating token merging effectively reduces redundant context interference.
• Replacing AuEmo token space with limited emotion labels or open-vocabulary labels lowers emotional expressiveness by more than 3-4% accuracy, confirming the benefit of discrete authentic emotion tokens.
• Removing AuEmo classifier guidance in flow matching causes performance drops: E-DMOS reduced by about 0.07 and AuEmoACC decreases by ~1.5%, illustrating its role in aligning emotion.
• AuEmoCodec activates about 750 emotion codebook tokens out of 1000, reflecting a rich but compact authentic emotion space learned from data.

Methodology — deep read

1. Threat Model & Assumptions: The paper focuses on synthesizing human-like emotional speech in multi-turn user-agent conversational settings. The adversary or threat model itself is not explicitly defined, as this is a generative modeling task rather than an adversarial security scenario.

2. Data: The authors use the NCSSD-EmCap dataset, a composite of three high-quality dialogue speech corpora resulting in ~384 hours of conversational speech, 18,580 dialogues, 245,984 utterances from 25 speakers (balanced genders). The data is split 8:1:1 for train, validation, and test. Emotional labels are estimated via Gemini-2.5-Flash to provide perceived emotion scores along seven basic axes for training the emotion codec.

3. Architecture/Algorithm:

• AuEmoCodec: Trained separately to learn a discrete authentic emotion token space via Finite Scalar Quantization (FSQ). Emotional speech is encoded into a low-rank quantization space with quantization levels [8, 5, 5, 5], producing a 1000-size discrete codebook of AuEmo tokens, 750 of which activate after training. It reconstructs multi-axis perceived emotion scores rather than speech waveform to focus purely on emotion representation.
• Tokenization: Dialogue turns are tokenized into text tokens (BPE tokenizer), speaker embeddings, speech tokens (CosyVoice2 speech tokenizer), and AuEmo tokens.
• AuEmoToMe: An authentic-emotion-guided token merging algorithm inspired by ToMe, merges redundant tokens in text and speech sequences in each utterance's dialogue context. It computes cosine similarity between tokens and merges pairs with high similarity, weighting the merge by similarity to the utterance's AuEmo token to preserve emotional cues.
• Text-Speech Autoregressive Model: Takes merged dialogue tokens plus target utterance speaker/text tokens as input. Predicts target AuEmo token first, then generates speech tokens.
• Merged Context Encoder: Bidirectional GRU encodes merged tokens to provide dialogue context features.
• AuEmo Encoder: Embeds predicted AuEmo token.
• Authentic Emotion Flow Matching: A conditional flow matching network generates mel-spectrogram conditioned on merged dialogue context encoding, AuEmo encoding, and extracted acoustic prior of predicted speech tokens. It uses an auxiliary pretrained AuEmo classifier to guide intermediate spectral states towards the target authentic emotion, activated at later diffusion timesteps to avoid noisy gradients early.

4. Training Regime:

• AuEmoCodec trains on large-scale emotional speech to minimize MSE between reconstructed and ground-truth perceived emotion scores, thus learning the discrete emotion space.
• AuEmoChat model trains end-to-end on NCSSD-EmCap, using 4 NVIDIA A100 GPUs, batch size 4, gradient accumulation of 8.
• Detailed hyperparameters for flow matching, merging ratios, embedding sizes are referenced but not exhaustively detailed in the truncated text.

5. Evaluation Protocol:

• Subjective metrics: Naturalness-DMOS and Emotion-DMOS rated by 30 trained evaluators.
• Objective metrics: WER, Mel Cepstral Distortion (MCD), Speaker similarity (SpkSIM), basic 7-class Emotion Accuracy (EmoACC), and Authentic Emotion Accuracy (AuEmoACC).
• Ablation studies systematically test the removal or replacement of key components: AuEmo Tokenizer variants, AuEmoToMe merging and merging strategy, flow matching emotion conditioning, and classifier guidance.
• Statistical significance cited via 95% confidence intervals for subjective scores.

6. Reproducibility:

• Code and demos are promised on GitHub (anonymous repo link).
• Dataset used is NCSSD-EmCap, which is public/open.
• Model checkpoints and detailed training procedures seem not fully public or detailed.

Overall, the workflow example: For an input dialogue history, tokenize all modalities including speech with AuEmoCodec to get discrete authentic emotion tokens. Apply AuEmoToMe to merge redundant tokens guided by authentic emotion anchors. Use the merged tokens as input to an autoregressive model that first predicts the target authentic emotion token, then the speech tokens. Convert predicted tokens to mel-spectrogram via flow matching conditioned jointly on merged context and AuEmo embedding, guided by an emotion classifier to enforce authentic emotional expression, and finally synthesize audio waveform with HiFi-GAN vocoder.

Technical innovations

• AuEmoCodec: Learning a discrete authentic emotion token space via finite scalar quantization on large-scale emotional speech, reconstructing multi-axis perceived emotion scores instead of waveforms.
• AuEmoToMe: An authentic-emotion-guided token merging algorithm that reduces redundant multimodal dialogue tokens while preserving emotion-relevant information, improving context modeling in conversational speech synthesis.
• Authentic Emotion Flow Matching: A conditional flow matching network combining merged dialogue context, predicted authentic emotion tokens, and acoustic priors, with a classifier-guided intermediate state regularization for more emotionally consistent speech generation.
• Integration of authentic emotion tokens directly into an autoregressive text-speech model that sequentially predicts emotion tokens and speech tokens for conversational synthesis.

Datasets

• NCSSD-EmCap — ~384 hours, 18,580 dialogues, 245,984 utterances, 25 speakers — public composite dataset integrating DailyTalk, NCSSD, MultiDialog

Baselines vs proposed

• BaseCSS: N-DMOS=3.431 ± 0.023 vs AuEmoChat=4.171 ± 0.026; E-DMOS=3.299 vs 3.979; WER=27.28% vs 9.14%
• ECSS: N-DMOS=3.675 ± 0.022 vs 4.171; E-DMOS=3.509 vs 3.979; WER=25.10% vs 9.14%
• GPT-Talker: N-DMOS=3.812 ± 0.023 vs 4.171; E-DMOS=3.654 vs 3.979; WER=20.49% vs 9.14%
• Chain-Talker: N-DMOS=3.955 ± 0.025 vs 4.171; E-DMOS=3.756 vs 3.979; WER=15.07% vs 9.14%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15755.

Fig 1: Previous CSS models rely on a limited emotion

Fig 2: The left side illustrates the overall framework of the proposed AuEmoChat, which includes: Multimodal Dialogue

Fig 3: Analysis results of different token merging rates

Limitations

• The authentic emotion token space is learned from perceived emotion scores along seven basic axes; thus it may still be limited in capturing full nuance of extremely subtle emotions.
• No explicit adversarial robustness evaluation or stress testing against out-of-distribution dialogue styles or speakers was conducted.
• Model training details of flow matching and token merging hyperparameters lack exhaustive disclosure, potentially hindering exact reproducibility.
• User evaluations are limited to English-proficient annotators, while dataset and model applicability to other languages remains unexamined.
• The merging ratio in AuEmoToMe is a hyperparameter with some sensitivity, requiring careful tuning per dataset.
• Emotion perception annotations rely on an automatic system (Gemini-2.5-Flash), whose error or bias may affect authentic emotion token quality.

Open questions / follow-ons

• How can the authentic emotion token space be generalized or adapted to multilingual conversational speech synthesis?
• What is the performance and robustness of AuEmoChat under adversarial or noisy dialogue scenarios where emotional cues are ambiguous or conflicting?
• Can the token merging approach be dynamically adaptive to dialogue length and emotion complexity during inference?
• How does incorporating user feedback or reinforcement learning affect the quality of authentic emotional expression?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, AuEmoChat advances the ability to generate speech with nuanced, contextually consistent authentic emotions in conversational agents. This improves the naturalness and unpredictability of human-agent voice interactions, raising the bar for bots to mimic true human affective speech convincingly. Incorporating AuEmo token representations and context-aware token merging can enhance the subtlety of auditory challenges aimed at differentiating humans from automated agents. Moreover, the discrete authentic emotion token space may itself inspire new metrics or signals for detecting synthetic speech artifacts in bot detection systems. However, researchers should consider the complexity and computational costs involved, and potential robustness limits when deploying such models in adversarial environments. Overall, this work enriches the foundational techniques for emotion-aware speech synthesis, contributing tools relevant for next-generation bot detection and CAPTCHA challenges that leverage affective voice cues.

Cite

@article{arxiv2607_15755,
  title={ AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis },
  author={ Zhenqi Jia and Yuan Zhao and Aruukhan and Rui Liu and Haizhou Li },
  journal={arXiv preprint arXiv:2607.15755},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15755}
}

Read the full paper

• arXiv:2607.15755 (abstract)
• arXiv:2607.15755 (PDF)