WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech Technology · Depth: Expert, extended

Summary

WordVoice is a novel framework for Large Language Model (LLM)-based Text-to-Speech (TTS) that enables explicit, multi-dimensional word-level control, addressing the coarse-grained limitations of existing systems. It introduces WordVoice-5A, a massive 4.7k-hour bilingual dataset with five-dimensional word-level annotations (duration, boundary, energy, pitch, and tone), created through a rigorous linguistically-guided pipeline. The framework integrates a bound-token mechanism within the LLM for "acoustic planning" and a fine-grained acoustic modulation module in the token-to-waveform stage. Experiments on a test set of 2,000 Chinese and 1,500 English utterances demonstrate superior, decoupled control over acoustic dimensions while maintaining competitive zero-shot synthesis stability, outperforming baselines like CosyVoice3 and MagicTTS in control precision. The system is trained on 8 NVIDIA A800 GPUs.

Key takeaway

For Machine Learning Engineers developing advanced TTS systems, WordVoice offers a solution to the challenge of fine-grained acoustic control. You can now achieve deterministic, word-level manipulation of duration, pitch, energy, boundary, and tone, which is critical for applications like audiobook narration and video dubbing. Consider integrating explicit acoustic planning and modulation modules into your LLM-based TTS pipelines to enhance control precision and interpretability.

Key insights

WordVoice enables precise, decoupled word-level control in LLM-based TTS by transforming implicit generation into explicit acoustic planning.

Principles

• Explicit acoustic planning improves TTS control and naturalness.
• Linguistically-guided data annotation is crucial for fine-grained control.
• Decoupled control allows independent manipulation of acoustic attributes.

Method

WordVoice uses a bound-token mechanism in an LLM for macro-prosodic planning, predicting five acoustic attributes. A fine-grained style modulation module in the Flow Matching stage then ensures micro-acoustic fidelity by upsampling and injecting word-level style tokens.

In practice

• Use WordVoice-5A for training controllable TTS models.
• Implement bound-token mechanisms for explicit attribute prediction.
• Apply fine-grained modulation to compensate for token quantization loss.

Topics

• Text-to-Speech
• Large Language Models
• Word-Level Control
• Acoustic Planning
• Data Annotation
• Flow Matching

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026 — Explicit language tags and lexical matching significantly improve cross-lingual voice cloning performance and reduce accent leakage.
• DramaBox: An Open-Weight TTS Model Built Around Stage Directions — DramaBox uses stage directions to control TTS delivery, offering nuanced, context-aware speech generation.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data — An audio-LLM can effectively filter noisy speech-to-speech translation data using a Rank-to-Distill strategy, improving S2ST performance.
• TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech — TalkTag automates fine-grained morphosyntactic error annotation in spoken transcripts using an LLM, even with scarce data.
• Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt — TimePro-RL enhances LALMs' temporal perception by integrating explicit time prompts and optimizing with reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.