WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

2026-07-07 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Speech Technology · Depth: Expert, quick

Summary

WordVoice is a novel framework designed to provide explicit and decoupled multi-dimensional word-level control for Large Language Model (LLM)-based Text-to-Speech (TTS) systems, addressing the limitations of implicit end-to-end generation in scenarios like audiobook narration and video dubbing. The system introduces WordVoice-5A, a massive 4.7k-hour bilingual dataset with five-dimensional word-level annotations (duration, boundary, energy, pitch, and tone), created via a rigorous linguistically-guided pipeline. WordVoice transforms implicit generation into an explicit "acoustic planning" process using a bound-token mechanism within the LLM, allowing adaptive multi-task prosodic planning and flexible manual intervention. Additionally, it augments the token-to-waveform stage with a fine-grained acoustic modulation module to strictly align word-level attributes between highly compressed discrete tokens and continuous waveforms. Experiments demonstrate WordVoice achieves superior, decoupled control while maintaining competitive zero-shot synthesis stability.

Key takeaway

For NLP Engineers developing advanced TTS systems, WordVoice offers a solution to the challenge of fine-grained acoustic control. You can now achieve explicit, decoupled manipulation of word-level attributes like duration, pitch, and tone, which is critical for high-quality audiobook narration or video dubbing. Consider integrating this "acoustic planning" paradigm to enhance your system's precision and adaptability, moving beyond coarse-grained implicit generation.

Key insights

WordVoice enables precise, decoupled word-level acoustic control in LLM-based TTS through explicit planning and a new annotated dataset.

Principles

• Explicit acoustic planning enhances TTS control.
• Fine-grained datasets are crucial for word-level control.
• Decoupled control improves stylistic manipulation.

Method

WordVoice constructs WordVoice-5A, then uses a bound-token mechanism for LLM-based "acoustic planning," augmented by a fine-grained acoustic modulation module for token-to-waveform alignment.

In practice

• Apply WordVoice for precise audiobook narration.
• Use for accurate video dubbing synchronization.
• Manually intervene in prosody for specific effects.

Topics

• LLM-based TTS
• Word-level Control
• Acoustic Planning
• Speech Synthesis
• WordVoice-5A Dataset
• Prosody Control

Best for: Research Scientist, AI Engineer, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Improving quality and robustness in LLM-based text-to-speech systems — LLM-based TTS quality improves through targeted techniques addressing accent leakage, expressiveness, and reliability.
• Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models — Integrating trainable audio-side prompts with text-side prompt tuning significantly enhances few-shot adaptation in Audio-Language Models.
• microsoft / VibeVoice — VibeVoice offers open-source, long-form, and real-time voice AI models for advanced speech recognition and synthesis.
• Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation — A novel S2ST system and evaluation metric significantly improve cross-lingual lexical stress transfer from English to Chinese.
• A Guide to Voice Cloning on Voxtral with a Missing Encoder — Voxtral-4B-TTS offers high-quality TTS, but its voice cloning is limited by missing encoder weights.
• Phonetic Error Analysis of Raw Waveform Acoustic Models — Raw waveform acoustic models achieve state-of-the-art TIMIT PER, with error patterns revealing specific phonetic class improvements from BLSTMs and transfer learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.