Fine-tuning Whisper Across 81 Languages

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Shivam Singh and Alex Warstadt fine-tuned the Whisper large-v3 model independently across 81 languages featured in the FLEURS benchmark. This process resulted in improved Word Error Rate (WER) for all 81 languages, achieving an average reduction of nearly 30%. However, the extent of improvement varied significantly, with the language's writing system identified as the strongest predictor of success. Languages utilizing Latin and Cyrillic scripts consistently reached single-digit WERs, while those with unique scripts, such as Thai, Georgian, Burmese, and Khmer, showed the least benefit. The research also established a strong correlation (Spearman ρ ≈ -0.78) between Whisper's BPE compression ratio and fine-tuning headroom, indicating that tokenization serves as a fundamental bottleneck. Model weights are slated for release upon publication.

Key takeaway

For Machine Learning Engineers optimizing ASR models for multilingual applications, you should prioritize fine-tuning Whisper large-v3, expecting nearly 30% WER reduction on average. If your target languages include unique scripts like Thai or Khmer, anticipate less improvement and investigate custom tokenization strategies. Your efforts will yield the best results for Latin and Cyrillic script languages, often achieving single-digit WERs, but be prepared to address tokenization limitations for others.

Key insights

Fine-tuning Whisper large-v3 significantly reduces WER across 81 languages, with script type and BPE compression ratio predicting success.

Principles

• Fine-tuning improves ASR performance universally.
• Script type predicts ASR fine-tuning efficacy.
• BPE compression ratio indicates tokenization bottleneck.

Method

Fine-tuned Whisper large-v3 independently on each of 81 FLEURS benchmark languages, then analyzed WER improvements against script type and BPE compression ratio.

In practice

• Prioritize fine-tuning for Latin/Cyrillic script languages.
• Investigate tokenization for unique script languages.

Topics

• Whisper large-v3
• Fine-tuning
• Automatic Speech Recognition
• Word Error Rate
• Multilingual Models
• Tokenization
• BPE Compression

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Improving low-resource ASR using bilingual fine-tuning with language identification: a cross-linguistic evaluation — Bilingual fine-tuning enhances low-resource ASR, particularly when language identification is accurate or explicitly guided during inference.
• Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER) — Benchmark contamination significantly inflates reported ASR performance, especially for dialectal variations.
• BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language — Fine-tuning pre-trained ASR models with modest, newly created corpora can establish functional speech recognition for low-resource languages.
• A Comparative Study of Pretrained Transformer Models for Quranic ASR: Speech Representations, Label Formats, and Dataset Composition — Fine-tuning pretrained Transformers with specific speech representations and label formats significantly improves Quranic ASR accuracy and efficiency.
• Voice Cloning For Any Language | Fine-Tuning Tortoise-TTS | Part 2 — Adapt Tortoise-TTS for custom languages by fine-tuning the autoregressive model and modifying inference code.
• Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades — ASR-stage information loss, especially single-character errors in Korean, is the primary bottleneck for ASR-LLM cascade performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.