WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

Source: arXiv:2607.06461 · Published 2026-07-07 · By Sihang Nie, Jinxin Ji, Xiaofen Xing, Deyi Tuo, Chengbin Jin, Jialong Mai et al.

TL;DR

This paper addresses the significant gap in fine-grained, multidimensional word-level control in large language model (LLM)-based text-to-speech (TTS) systems. Current end-to-end TTS approaches predominantly generate speech implicitly with poor ability to manipulate duration, energy, pitch, tone, and boundaries explicitly at the word level, which is critical for applications like audiobook narration and video dubbing. The authors present WordVoice, a unified framework that enables precise, explicit word-level acoustic control. Key contributions include WordVoice-5A, a massive 4.7k-hour bilingual dataset annotated with five acoustic dimensions at the word level, created via a novel linguistically-guided pipeline. Architecturally, WordVoice introduces a bound-token mechanism inside an autoregressive LLM to explicitly plan word-level prosody before generating speech tokens, plus a fine-grained word-level acoustic modulation module at the waveform synthesis stage to compensate for information loss from discrete token quantization. Experiments on Chinese and English demonstrate that WordVoice achieves highly decoupled, multi-dimensional control while maintaining competitive naturalness and zero-shot stability compared to strong baselines. Subjective and objective evaluations confirm significant gains in control precision and naturalness without sacrificing speaker similarity. This work effectively transforms implicit TTS generation into an interpretable, explicit acoustic planning process, opening new possibilities for precise stylistic speech synthesis.

Key findings

• Constructed WordVoice-5A, a 4.7k-hour bilingual (Chinese and English) dataset with rigorous five-dimensional word-level annotations: duration, boundary, energy, pitch, and tone.
• Introduced a bound-token guided autoregressive LLM mechanism that explicitly predicts per-word acoustic attributes forming a word-level style token, enabling decoupled, multi-task prosodic planning.
• Added a fine-grained frame-level acoustic modulation module in the Flow Matching vocoder stage, upsampling word-style tokens according to predicted durations and pauses, to recover acoustic details lost in quantization.
• Subjective listening tests showed WordVoice-Control mode achieved statistically significant improvements in Naturalness (N-MOS) and Word Style Control (Ctrl-MOS) over CosyVoice3 baseline (p < 0.01), with minor trade-off in speaker similarity.
• Objective metrics indicate WordVoice-Control cuts Mean Absolute Error (MAE) in duration, energy, and pitch by ~30-50% and reduces boundary error rate roughly by half compared to CosyVoice3 on both Chinese and English test sets.
• Comparison with MagicTTS shows WordVoice attains lower duration MAE (e.g., 0.0383 vs 0.0438 on Chinese) and boundary error rate (12.82% vs 16.10%), demonstrating superior explicit temporal control.
• Decoupled single-attribute intervention experiments confirm WordVoice can control duration, energy, pitch, boundary, and tone largely independently, except tone is somewhat entangled dynamically with other features.
• Ablation of the fine-grained Flow Matching modulation module significantly worsens energy and pitch errors, confirming its critical role in compensating continuous acoustic details absent in discrete token encoding.

Threat model

n/a — The paper focuses on TTS synthesis technical innovations and does not emphasize adversarial threat modeling or security considerations.

Methodology — deep read

The paper designs and evaluates a two-stage TTS framework focused on explicit word-level acoustic control.

1. Threat Model & Assumptions: The adversary is not explicitly discussed as this is a synthesis system paper, but the framework assumes availability of high-quality paired speech-text data with word-level acoustic annotations to enable precise prosodic planning and control. The environment assumes user intervention can occur at inference to specify word-level attribute values.

2. Data: The authors construct WordVoice-5A by re-annotating existing corpora including LEMAS, producing ~4684 hours of bilingual speech (Chinese and English). This dataset includes rigorous five-dimensional annotations per word: duration, boundary (five discrete levels of pause length), energy, pitch, and tone (seven phonetic morphologies). Annotation uses Montreal Forced Aligner and Qwen3 Forced Aligner for timestamp extraction, followed by a linguistically-guided optimization pipeline with loudness-based edge refinement, valley snapping, and consistency checks across aligners (top 8% highest quality data retained).

3. Architecture / Algorithm: The overall system is built on CosyVoice3 backbone. The core novelty lies in two modules:

• WordVoice-LLM: an autoregressive large language model (based on Qwen2.5-0.5B) generates speech tokens in a sequence interleaved with a special bound token ⟨b⟩ per word. Upon decoding ⟨b⟩ for word i, a lightweight Word Decoder (2-layer MLP) predicts five acoustic word-level attributes (duration, boundary, energy, pitch, tone). These are embedded and concatenated with the semantic word embedding to form a word-level style token. The style token conditions the subsequent chunk generation of speech tokens for that word, effectively ordering first prosodic planning then speech generation.
• WordVoice-FM: in the Flow Matching vocoder stage, the discrete word-level style tokens are length-regulated and upsampled frame-wise according to predicted word durations and pauses, creating a frame-level style token sequence. A Word Style Modulation module injects this conditioning via learned scale and shift parameters that modulate the normalized speech token representation at each frame before waveform generation, recovering fine-grained acoustic details lost in token quantization.

4. Training Regime: The LLM stage is initialized from Qwen2.5-0.5B. The speech token sequence generation is trained autoregressively with negative log likelihood loss. The five acoustic attribute predictions are quantized into discrete bins and optimized with cross-entropy loss under an uncertainty-weighted multi-task loss schema simultaneously with generation loss. The vocoder Flow Matching model is trained for 20 epochs using a velocity field regression loss conditioned on modulated speech tokens. Random masking of 30% input speech tokens during training enforces reliance on style tokens.

5. Evaluation Protocol: Evaluated on held-out WordVoice-5A test splits with ~2000 Chinese and 1500 English utterances. Compared baselines include Mel-Recon (upper bound reconstruction), CosyVoice3 (state-of-the-art LLM-based TTS), MagicTTS (word-level temporal control). Metrics include subjective MOS for naturalness, speaker similarity, and control precision (Ctrl-MOS). Objective measures assessed word error rate (WER) via Qwen3-ASR-1.7B, and acoustic attribute control precision via mean absolute error (MAE) for continuous attributes and error rate (ER) for discrete attributes, extracted from generated audio via Qwen3 Forced Aligner. Statistical tests (Mann-Whitney U) verify significance.

6. Reproducibility: Code and demo audio samples are publicly released. The WordVoice-5A dataset is open sourced. Models and training details (epochs, hyperparameters, GPU counts) are shared. Some pretraining from Qwen model is required. Details on random seeds or exact hyperparameters are not fully detailed, leaving partial gaps.

Example end-to-end: Given input text, the LLM decodes tokens interleaved with ⟨b⟩ tokens triggering prediction of duration, boundary, energy, pitch, and tone for each word. The word embedding plus style token form conditioning for generating corresponding speech token chunk. The Flow Matching vocoder upsamples these style tokens frame-wise and modulates continuous token embeddings to synthesize the waveform. Manual attribute values can override LLM predictions at inference for explicit control.

Overall, the methodology combines linguistically-grounded dataset curation, novel architecture integrating explicit acoustic planning in LLM decoding, and fine-grained acoustic modulation to enable deterministic, decoupled word-level multi-dimensional TTS control.

Technical innovations

• A bound-token guided autoregressive LLM decoding mechanism that explicitly predicts five acoustic attributes per word mid-generation, enabling explicit and interpretable acoustic planning.
• A linguistically-guided annotation pipeline producing a large-scale bilingual dataset with five-dimensional word-level prosodic annotations (duration, boundary, energy, pitch, tone) using dual-aligner consistency and a novel pause hierarchy.
• A fine-grained acoustic style modulation module in the Flow Matching vocoder stage that injects frame-level conditioning upsampled from word-level style tokens, compensating for quantization-induced acoustic detail loss.
• A dual-mode inference framework allowing both autonomous prosodic planning and zero-shot user-controlled manual intervention of any acoustic attribute per word without retraining.

Datasets

• WordVoice-5A — 4684 hours — open-sourced bilingual dataset constructed by re-annotating LEMAS and other corpora

Baselines vs proposed

• CosyVoice3: N-MOS (Chinese) = 3.55 ± 0.08 vs WordVoice-Control: 3.69 ± 0.08
• CosyVoice3: Ctrl-MOS (Chinese) = 3.03 ± 0.10 vs WordVoice-Control: 3.45 ± 0.09
• CosyVoice3: Duration MAE (Chinese) = 0.0549 vs WordVoice-Control: 0.0349
• CosyVoice3: Boundary ER (Chinese) = 32.47% vs WordVoice-Control: 12.72%
• MagicTTS-Dur: Duration MAE (Chinese) = 0.0438 vs WordVoice: 0.0383
• MagicTTS-Pau: Boundary ER (Chinese) = 16.10% vs WordVoice: 12.82%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.06461.

Fig 1: WordVoice framework. By introducing explicit word-

Fig 2: The linguistically-guided annotation pipeline. (a) Alignment & Clean: Refining MFA and Qwen3FA timestamps via

Fig 3: Overall architecture of WordVoice. (a) WordVoice-LLM: During autoregressive decoding, the bound token ⟨b⟩triggers

Fig 4: 2D density heatmaps of generated versus target

Limitations

• Small trade-off in speaker similarity (Spk-MOS) due to localized stylistic variations causing minor perturbations in global acoustic transitions.
• Slightly increased word error rate (WER) in controlled mode, suggesting a marginal impact on linguistic robustness from explicit conditioning.
• Tone attribute is less strictly decoupled from other acoustic features because of its dynamic contour nature, complicating independent control.
• Details on random seed control and full hyperparameter search are not fully disclosed, which may affect exact reproducibility.
• No reported adversarial robustness testing or evaluation under distribution shifts beyond normal test sets.
• The LLM backbone is pretrained (Qwen2.5-0.5B), which may limit accessibility or generalization to other model families.

Open questions / follow-ons

• How to integrate WordVoice explicit acoustic planning into instruction-based or prompt-driven TTS models for even broader stylistic and semantic control?
• Can the framework be extended to disentangle tone attribute control more fully from other prosodic components given its dynamic nature?
• How robust is WordVoice under distributional shifts such as different speakers, domains, or noisy inputs in zero-shot or few-shot setups?
• What are the potential benefits of leveraging the explicit multi-dimensional acoustic plans as intermediate representations for speech editing or voice conversion tasks?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners working on audio challenges or spoken interaction, WordVoice presents key advances toward deterministic, fine-grained control of speech prosody at the word level. Precise temporal and multi-dimensional acoustic manipulation enables generation of highly expressive and synchronized speech which could improve naturalness and robustness of audio CAPTCHA systems or voice-based chatbot defenses. The explicit acoustic planning and modulation approach may offer interpretable points for defense against synthetic voice spoofing by facilitating nuanced control over subtle prosodic cues. Furthermore, the large-scale WordVoice-5A dataset with high-quality word-level annotations could serve as a valuable resource for training or evaluating models focused on robust, controllable speech generation. Practitioners may consider incorporating elements of explicit token-level acoustic planning or multi-dimensional modulation to enhance adversarial detection or to generate speech stimuli with strict temporal alignment requirements in multi-modal bot detection workflows.

Cite

@article{arxiv2607_06461,
  title={ WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS },
  author={ Sihang Nie and Jinxin Ji and Xiaofen Xing and Deyi Tuo and Chengbin Jin and Jialong Mai and Xiangmin Xu },
  journal={arXiv preprint arXiv:2607.06461},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.06461}
}

Read the full paper

• arXiv:2607.06461 (abstract)
• arXiv:2607.06461 (PDF)