BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

BlueMagpie-TTS is a novel text-to-speech system specifically designed for Taiwanese Mandarin, addressing common issues like accent mismatch, text over-segmentation, and poor code-switching pronunciation. The system comprises three main components: PangolinTokenizer, Barbet, and a modified VoxCPM2 acoustic stack. PangolinTokenizer, a byte-level BPE tokenizer trained on Taiwan-context data, achieves the lowest token rate of 0.485 tokens/character with the smallest vocabulary among eight compared tokenizers. Barbet, a billion-parameter Traditional-Chinese language model trained on PangolinTokenizer, functions as the text-semantic frontend, ranking first on the 14-task TAIDE-14 evaluation with 0.7488 bits per byte. BlueMagpie-TTS integrates Barbet with the pretrained VoxCPM2 acoustic stack via a learned bridge, then fine-tunes the combined model. This approach reduced the Character Error Rate (CER) from 11.45% to 4.81% on a 1,000-sentence Taiwan-localized test set, a 58.0% relative reduction. It also outperformed a voice-fine-tuned baseline by 25.2% CER and was preferred by 65.6% of listeners in a blind study.

Key takeaway

For NLP Engineers developing text-to-speech systems for specific regional accents like Taiwanese Mandarin, especially with code-switching, you should prioritize a bottom-up text-side adaptation. Investing in a custom, token-efficient tokenizer and a localized language model frontend, rather than solely fine-tuning acoustic models, significantly improves intelligibility and naturalness. This approach reduces Character Error Rate by over 25% compared to acoustic-only fine-tuning, making your TTS output more accurate and preferred by native speakers.

Key insights

Localizing text-side components, including tokenizers and language models, significantly enhances TTS quality for specific accents and code-switching.

Principles

• Token efficiency lowers inference cost and improves LM performance.
• Frontend adaptation is critical for accent and code-switching quality.
• Reusing pretrained acoustic stacks reduces training data needs.

Method

Train a byte-level BPE tokenizer on context-specific data, then a Traditional-Chinese language model. Attach this frontend to a pretrained acoustic stack via a learned bridge, followed by joint fine-tuning.

In practice

• Implement byte-level BPE for lossless text representation.
• Stratify tokenizer training corpora for local context.
• Use a two-stage bridge and joint fine-tuning process.

Topics

• Taiwanese Mandarin TTS
• Code-Switching Speech
• Tokenization Efficiency
• Language Model Frontend
• Byte-level BPE
• Speech Synthesis Localization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Voice Cloning For Any Language | Fine-Tuning Tortoise-TTS | Part 2 — Adapt Tortoise-TTS for custom languages by fine-tuning the autoregressive model and modifying inference code.
• CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding — CleanCodec reframes audio tokenization as a selective information bottleneck, encoding only perceptually important speech features.
• FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS — FlowEdit enables lifelong TTS pronunciation adaptation via latent embedding edits stored in a Hopfield Network, reducing errors by 92.7%.
• CN-NewsTTS Bench: a target-level automatic benchmark for raw-input Chinese news TTS pronunciation — CN-NewsTTS Bench evaluates raw-input Chinese news TTS systems on complex written forms without external aids.
• Improving quality and robustness in LLM-based text-to-speech systems — LLM-based TTS quality improves through targeted techniques addressing accent leakage, expressiveness, and reliability.
• Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal — Lightweight pronunciation assessment uses native speech token surprisal and text-guided alignment, avoiding costly labeled learner data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.