Voice Cloning For Any Language | Fine-Tuning Tortoise-TTS | Part 1

2024-03-17 · Source: Martin Thissen · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, extended

Summary

This content details the process of fine-tuning the Tortoise-TTS model for voice cloning in any language, specifically demonstrating German. It outlines the essential steps, beginning with preparing a high-quality speech dataset, such as a 97-hour German dataset from 117 speakers, and formatting it to the LG speech standard. The process involves modifying the Tortoise-TTS fine-tuning code to support non-English characters by adjusting `cleanup.py` for transliteration, adding special characters to `symbols.py`, and configuring `custom_language_gpt.yaml` for custom tokenizer vocabulary and dataset paths. It also covers environment setup, including fixing the Transformers library version to 4.29.2, downloading pre-trained model weights for the VQVAE and autoregressive GPT-2 models, and training a custom tokenizer using all dataset transcriptions. Finally, it addresses resampling audio samples to 22.05 kHz and initiating the fine-tuning of the autoregressive model, with checkpoints saved every 500 steps for potential resumption.

Key takeaway

For Machine Learning Engineers aiming to adapt Tortoise-TTS for non-English voice cloning, you must meticulously prepare your language-specific dataset in LG speech format and customize the model's code to handle unique characters and train a new tokenizer. Ensure your audio is resampled to 22.05 kHz and consider using a cloud GPU for the potentially long fine-tuning process, saving checkpoints regularly to preserve progress and allow for training resumption.

Key insights

Fine-tuning Tortoise-TTS for new languages requires custom data preparation, code modifications, and tokenizer training.

Principles

• Data formatting is critical for model compatibility.
• Custom tokenizers improve language-specific representation.

Method

Prepare an LG speech-formatted dataset, modify Tortoise-TTS code for language-specific characters and tokenizer, train a custom tokenizer, resample audio, then fine-tune the autoregressive model.

In practice

• Use `German transliterate` for German text cleaning.
• Fix Transformers version to 4.29.2 to avoid errors.
• Resample audio to 22.05 kHz for Tortoise-TTS input.

Topics

• Tortoise-TTS Fine-tuning
• Voice Cloning
• Multilingual Text-to-Speech
• Custom Tokenization
• Speech Data Preparation

Best for: Machine Learning Engineer, Deep Learning Engineer, AI Student

Related on AIssential

• Voice Cloning For Any Language | Fine-Tuning Tortoise-TTS | Part 2 — Adapt Tortoise-TTS for custom languages by fine-tuning the autoregressive model and modifying inference code.
• A Guide to Voice Cloning on Voxtral with a Missing Encoder — Voxtral-4B-TTS offers high-quality TTS, but its voice cloning is limited by missing encoder weights.
• KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026 — Cross-lingual voice cloning can be significantly improved by combining language prompting, RL fine-tuning, and lexical matching.
• Turned Telegram Into a Local AI Voice Studio — Local AI voice synthesis via Telegram bot offers convenient, high-quality voice generation, design, and cloning.
• How Mistral’s Voxtral TTS is Redefining Multilingual Voice Cloning with a Hybrid Autoregressive and Flow-Matching Architecture — Voxtral TTS uses a hybrid autoregressive and flow-matching architecture for high-fidelity multilingual voice cloning.
• VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation — VoiceTTA enhances zero-shot TTS for uncommon styles via RL-based test-time adaptation, optimizing learnable prefixes with novel F0/energy style rewards.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Martin Thissen.