UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Speech Technology · Depth: Expert, long

Summary

UtterTune is a lightweight adaptation method that enhances pronunciation control in multilingual text-to-speech (TTS) systems, specifically targeting Japanese. It fine-tunes a large language model (LLM) architecture-based TTS, CosyVoice 2-0.5B, using low-rank adaptation (LoRA). This approach addresses challenges in accurately modeling Japanese grapheme-to-phoneme (G2P) mapping and pitch accent, which are difficult for LLM-TTS models that process raw text via byte-pair encoding without explicit G2P modules. UtterTune integrates LoRA layers into every self-attention block of CosyVoice 2's text-speech LM and introduces two special tokens, <phoneme> and </phoneme>, to guide phoneme-level segmental pronunciation and pitch accent. This method requires less than 0.5% additional parameters and less than one hour of training on an NVIDIA RTX 4090 GPU. Evaluations showed it increased naturalness MOS from 3.44 to 3.88, maintained speaker similarity, kept character error rate (CER) below 7%, and boosted accent correctness from 0.472 to 0.975 on stress-test material, without degrading performance in other languages.

Key takeaway

For machine learning engineers developing multilingual TTS systems, UtterTune offers a highly efficient solution to address language-specific pronunciation challenges without extensive retraining. You should consider implementing LoRA with explicit phoneme-level input tags to gain fine-grained control over segmental pronunciation and pitch accent, especially for languages like Japanese with complex G2P rules. This approach allows you to significantly improve target language quality while preserving the base model's performance and speaker similarity across other languages, with minimal computational overhead.

Key insights

LoRA-based adaptation enables precise phoneme and pitch accent control in multilingual LLM-TTS for specific languages like Japanese.

Principles

• LoRA preserves base model knowledge.
• Targeted adaptation rectifies language-specific deficiencies.
• Special tokens can guide implicit G2P models.

Method

Integrate LoRA layers into Transformer self-attention blocks and introduce special input tokens (<phoneme>, </phoneme>) to wrap phonemic sequences with accent notation, fine-tuning only these new parameters and token embeddings.

In practice

• Apply LoRA for language-specific TTS control.
• Use special tokens for explicit pronunciation guidance.
• Adapt LLM-TTS for complex G2P languages.

Topics

• Text-to-Speech
• Multilingual TTS
• Low-Rank Adaptation
• Large Language Models
• Japanese Pronunciation
• Pitch Accent Control
• CosyVoice 2

Code references

• shuheikatoinfo/UtterTune
• r9y9/pyopenjtalk
• espnet/espnet

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech — A diarization-guided, multi-stage ASR adaptation system significantly improves multilingual two-speaker conversational speech recognition.
• BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech — Localizing text-side components, including tokenizers and language models, significantly enhances TTS quality for specific accents and code-switching.
• DLRG@DravidianLangTech 2026: Dual-Purpose Whisper Adaptation for Tamil Dialect Identification and Dialectal Speech Recognition — A Tamil-adapted Whisper Medium model can effectively unify dialect identification and dialectal ASR tasks.
• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition — MoLGE uses language grouping and LoRA to efficiently scale multilingual ASR, overcoming capacity dilution.
• Continual Adaptation for Pacific Indigenous Speech Recognition — Adapting speech foundation models to distant, low-resource languages causes severe representational drift and catastrophic forgetting.
• GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speech — GLASS uses reward-trained LoRA adapters for composable, disentangled acoustic style control in zero-shot TTS.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.