HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech
Source: arXiv:2606.28249 · Published 2026-06-26 · By Sihang Nie, Xiaofen Xing, Rui Xing, Haoming Li, Ruitong Xiao, Jingyuan Xing et al.
TL;DR
This paper addresses key limitations in emotional text-to-speech (TTS) models based on large language models (LLMs), particularly the common issue of flattened, averaged prosody caused by supervised fine-tuning. The authors identify two structural mismatches hindering existing preference-driven optimization approaches: an information conflict where emotion and semantic content share a monolithic latent space causing gradient conflicts and reward hacking, and a scale gap where sparse sentence-level rewards fail to guide dense frame-level generation. To overcome these, they propose HPRO, a Hierarchical Progressive Reward Optimization framework. This introduces the HD-Emo codec, a differentiable reward model that extracts speech into distinct content and style preference tokens, structurally isolating emotional optimization from semantic content. HPRO then progressively aligns reward signals at frame, word, and sentence levels to bridge sparse-to-dense optimization.
Experiments on large emotional TTS datasets (LSSED and EmoVoice-DB) demonstrate that HPRO improves fine-grained emotional expressiveness while preserving linguistic intelligibility better than strong zero-shot baselines like CosyVoice2/3, IndexTTS2, and HD-PPT. Ablations highlight the value of separating content and emotion tokens and progressively applying hierarchical rewards to stabilize training. Overall, HPRO advances preference-driven emotional TTS by explicitly resolving conflicting objectives and optimization scales to produce more natural and emotionally nuanced speech.
Key findings
- HPRO reduces word error rate (WER) to 4.02% on emotional TTS test sets, improving over CosyVoice2 (5.45%) and IndexTTS2 (6.74%) by over 1.4 percentage points, indicating stronger semantic preservation.
- HPRO achieves the highest wVAD concordance correlation coefficient (CCC) of 0.339 and EMO-SIM of 0.672 among evaluated models, demonstrating superior fine-grained emotional expressiveness.
- Progressive hierarchical optimization boosts WER from 5.42% (CosyVoice2-SFT baseline) to 3.99% with frame and word-level rewards, illustrating the benefit of dense and boundary-aware supervision.
- Removing content supervision increases WER drastically to 13.61%, confirming that separating content tokens is critical to prevent semantic degradation from emotional optimization.
- Ablation removing both frame-level and word-level rewards (simulating DiffRO) worsens WER to 4.35% and reduces wVAD-CCC to 0.315 compared to full HPRO, demonstrating the necessity of hierarchical reward levels.
- HPRO obtains MOS-N (naturalness) of 4.171 ± 0.318 and MOS-E (emotion consistency) of 3.650 ± 0.347 in subjective tests, outperforming most baselines except IndexTTS2 on MOS-E but with better naturalness.
- Codebooks of size 1296 (content tokens) and 64 (style tokens) enable effective compression and discrete token extraction with finite scalar quantization (FSQ).
- Progressive temperature annealing from 2 to 0.8 during training stabilizes discrete token sampling for hierarchical optimization.
Threat model
The paper assumes an internal model optimization threat in preference-driven emotional TTS whereby naive optimization of a monolithic reward can cause reward hacking, i.e., aggressively maximizing emotional scores at the cost of semantic and acoustic integrity. The adversary is the inherent conflicting gradient signals within the shared latent space of content and emotion, as well as sparse global supervision failing to guide detailed generation steps causing undesirable output. There is no external attacker; rather, the threat is the model’s internal optimization dynamics causing degradation.
Methodology — deep read
The threat model assumes a preference-driven optimization setting for text-to-speech generation, where the model is optimized to synthesize speech with naturalness and emotional expressiveness without semantic degradation. The adversary is the internal reward optimization process, which can lead to reward hacking if information conflict or scale gaps exist.
The authors use three datasets: LibriSpeech (960 hours of ASR data), LSSED (206 hours of speech emotion recognition data), and EmoVoice-DB (40 hours of expressive emotional TTS data). LibriSpeech anchors content extraction. The emotional datasets serve for emotional style learning and evaluation. The test sets from LSSED and EmoVoice-DB are used for zero-shot TTS evaluation. Text transcriptions and emotion labels are available, and forced alignment is employed for word-level boundaries.
The HD-Emo codec is a novel differentiable reward model that extracts distinct content and style preference tokens from input speech tokens. Speech tokens are extracted with the CosyVoice2 tokenizer and processed through dual extractors (8-layer conformers) with finite scalar quantization (FSQ) bottlenecks yielding discrete content tokens (size 1296 codebook) and style tokens (size 64). Content tokens are supervised by an ASR objective using a Whisper-medium decoder to enforce semantic alignment. Style tokens are supervised hierarchically: sentence-level emotion classification via a pre-trained emotion2vec model, and fine-grained word-level Valence-Arousal-Dominance (wVAD) trajectories aligned with forced-aligned text boundaries and predicted by Wav2vec2-ft models. Dynamic feature modulation (inspired by Emo-FiLM) fuses content and style tokens for speech token reconstruction via cross-entropy loss.
The LLM backbone (Qwen2.5-0.5B) generates differentiable speech tokens using straight-through Gumbel-Softmax to enable gradient propagation. Hierarchical reward functions are defined at three levels: frame-level L1 regression losses aligning latent representations pre-quantization of content/style tokens, word-level wVAD CCC and ASR cross-entropy losses to preserve fine-grained emotional and semantic consistency, and sentence-level emotion distribution cross-entropy loss. A KL divergence regularizes the original speech token distribution.
Training uses a three-stage progressive optimization scheme: Stage I optimizes frame-level preference alignment and KL loss with higher Gumbel temperature (τ=2) for smooth gradients; Stage II adds word-level emotional and ASR supervision with τ annealed to 1; Stage III introduces sentence-level emotion classification with τ=0.8 for sharper token generation. Loss weights are dynamically adjusted at each stage to balance objectives.
The HD-Emo codec content extractor is pre-trained on LibriSpeech with ASR losses first, then trained on emotional datasets with other modules while freezing the content branch. The entire codec is trained with Adam optimizer for 100 epochs on 8 NVIDIA RTX 4090 GPUs with learning rate 1e-4. The LLM is optimized with Adam at 1e-5 following the progressive schedule.
Evaluation is zero-shot synthesis where test utterances are synthesized with random speaker references. Subjective evaluation uses 90 utterances rated by 18 participants on 5-point MOS scales for naturalness and emotional consistency. Objective metrics include WER (via whisper-large-v3), wVAD CCC, EMO-SIM, and DNSMOS perceptual quality, computed from final synthesized waveforms to avoid evaluation circularity. Multiple baselines are compared including CosyVoice2/3, IndexTTS2, HD-PPT, and a simulated DiffRO single-scale reward variant.
An end-to-end example: The LLM generates a differentiable discrete token sequence which the frozen HD-Emo codec maps into content and style latent spaces. Frame-level regression aligns with ground truth tokens, word-level wVAD and ASR losses enforce local emotional and semantic fidelity, and sentence-level emotion classification aligns overall affect. Progressive schedule initially focuses on alignment, then word-level refinement, and finally global emotion, enabling stable training and mutual information isolation between content and style to mitigate reward hacking and semantic collapse.
Code and audio samples are publicly available, enabling partial reproducibility, though some pretrained models (e.g., emotion2vec) come from external repositories. The emotional datasets are publicly known but licensing is standard for research.
Technical innovations
- Introduction of the HD-Emo codec, a differentiable dual-stream token codec that structurally separates speech into content and style preference tokens to mitigate gradient conflicts in emotional TTS optimization.
- A hierarchical progressive reward optimization (HPRO) framework that progressively aligns frame-level, word-level, and sentence-level objectives to bridge dense generation and sparse global reward scales.
- Use of finite scalar quantization (FSQ) bottlenecks to impose discrete preference token spaces facilitating stable differentiable optimization.
- Application of dynamic feature modulation (inspired by Emo-FiLM) to fuse content and style tokens for robust speech reconstruction while maintaining semantic and emotional disentanglement.
Datasets
- LibriSpeech — 960 hours — public ASR corpus
- LSSED — 206 hours — large-scale speech emotion recognition dataset
- EmoVoice-DB — 40 hours — expressive emotional TTS corpus
Baselines vs proposed
- CosyVoice2: MOS-N = 4.094 ± 0.257 vs HPRO: 4.171 ± 0.318
- CosyVoice2: WER = 5.45% vs HPRO: 4.02%
- IndexTTS2: MOS-E = 3.692 ± 0.213 vs HPRO: 3.650 ± 0.347
- IndexTTS2: WER = 6.74% vs HPRO: 4.02%
- HD-PPT: wVAD-CCC = 0.323 vs HPRO: 0.339
- Simulated DiffRO (w/o frame & wVAD): WER = 4.35%, EMO-SIM = 0.662 vs HPRO: 4.02%, 0.672
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28249.
Fig 1: Motivation. (a) DiffRO Framework. Single-scale reward optimization
Fig 2: Overview of the HD-Emo Codec. Monotonic speech tokens are processed by dual preference extractors with FSQ bottlenecks to obtain content and
Fig 3: Overview of the HPRO framework. The LLM generates differentiable speech tokens via Gumbel-Softmax, which are mapped by HD-Emo codec into
Limitations
- Evaluation is constrained to zero-shot TTS settings using specific emotional datasets; generalizability to unseen languages, speakers, or TTS domains is untested.
- While hierarchical rewards mitigate reward hacking, absolute prevention is not guaranteed, and the complex multi-objective training may still face optimization instability in other setups.
- The HD-Emo codec requires pre-trained ASR and emotion models whose quality constrains downstream performance; dependency on these external components could limit deployment.
- Word-level emotional supervision depends on forced alignment quality; errors in alignment could degrade fine-grained prosody modeling.
- Progressive training schedules and temperature annealing require careful hyperparameter tuning, which may limit straightforward reproducibility or transfer to other LLM backbones or tokenizers.
- Objective metrics focus on correlation with emotion and semantic accuracy but may not fully capture subjective perceptual nuances for all user groups.
Open questions / follow-ons
- Can the hierarchical progressive reward framework be generalized beyond emotional TTS to other speech attributes or stylistic controls for fine-grained generation?
- How can the progressive reward integration be automated or learned dynamically rather than manually scheduled with preset temperatures and loss weights?
- What are the robustness and adversarial properties of the disentangled token codec when exposed to out-of-domain speech or noisy references?
- Can word-level reward mechanisms be extended from supervised wVAD features to learned latent affective representations or other auxiliary modalities for improved expressiveness?
Why it matters for bot defense
For bot-defense and CAPTCHA engineers focusing on speech generation to detect or mitigate automated speech-based attacks or bots, this work's methods for disentangling content and emotional style tokens may inform advanced detection features that can analyze and differentiate natural human prosody versus synthetic or manipulated speech. The hierarchical reward framework addressing the scale gap between sparse global labels and dense fine-grained generation could inspire more nuanced evaluation metrics for voice biometrics or speech liveness verification.
Additionally, understanding how hierarchical preference extraction prevents reward hacking in TTS models can help anticipate potential vulnerabilities or manipulation vectors in speech bots that use emotional speech synthesis. While not directly tackling security, these insights into emotional TTS generation's internal dynamics are relevant background knowledge for designing robust voice CAPTCHA challenges or bot-detection heuristics utilizing expressive speech cues.
Cite
@article{arxiv2606_28249,
title={ HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech },
author={ Sihang Nie and Xiaofen Xing and Rui Xing and Haoming Li and Ruitong Xiao and Jingyuan Xing and Baiji Liu and Xiangmin Xu },
journal={arXiv preprint arXiv:2606.28249},
year={ 2026 },
url={https://arxiv.org/abs/2606.28249}
}