Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR

2026-07-10 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Lightweight speech recognition models, particularly architectures like Moonshine, often struggle with morphologically rich, non-Latin languages such as Bengali. This study identifies the core issue as the model's English-centric byte-level tokenizer, which excessively fragments Bengali words, leading to high token fertility (9.16) and catastrophic autoregressive collapse during inference. To address this, a novel vocabulary transplantation pipeline is proposed. This method replaces the decoder vocabulary with the native-script BanglaBERT WordPiece vocabulary and resizes the corresponding token embedding matrix. Experimental results show a significant reduction in token fertility to 1.30, decreasing autoregressive sequence length by 85.8% and entirely mitigating decoding instability. Evaluated on the 882-hour Lipi-Ghor dataset, the modified architecture achieves a competitive 21.54% Word Error Rate (WER) and a Real-Time Factor (RTF) of 0.0053, providing a scalable blueprint for cross-script adaptation without resource-intensive pre-training.

Key takeaway

For machine learning engineers deploying compact ASR models for non-Latin languages, the "Tokenizer Transplantation" method offers a robust solution to autoregressive collapse. You should consider implementing this vocabulary transplantation pipeline to adapt existing edge-efficient ASR architectures like Moonshine for morphologically rich languages. This approach significantly improves stability and performance, achieving a 21.54% WER and 0.0053 RTF on Bengali, all without requiring extensive retraining or resource-intensive pre-training.

Key insights

Replacing English-centric tokenizers with native-script vocabularies prevents autoregressive collapse in edge ASR for morphologically rich languages.

Principles

• English-centric tokenizers cause high fertility in non-Latin ASR.
• High token fertility leads to autoregressive collapse.
• Vocabulary transplantation enables cross-script ASR adaptation.

Method

A vocabulary transplantation pipeline replaces the decoder vocabulary with a native-script WordPiece vocabulary and resizes the token embedding matrix to mitigate autoregressive collapse.

In practice

• Adapt compact ASR models for new languages.
• Improve ASR stability on morphologically rich scripts.
• Avoid costly pre-training for cross-script ASR.

Topics

• Automatic Speech Recognition
• Tokenizer Adaptation
• Autoregressive Collapse
• Bengali NLP
• Edge AI
• Moonshine ASR

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR — Replacing English-centric tokenizers with native-script vocabularies prevents autoregressive collapse in edge ASR for morphologically rich languages.
• UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognition — UMA-Split extends unimodal aggregation to English non-autoregressive ASR by allowing aggregated frames to map to multiple fine-grained tokens.
• Scaling ASR for Hutsul Dialect: Multi-Speaker Data Collection, Enhanced Transcription and Cross-Speaker Evaluation — Scaling ASR for low-resource dialects requires multi-speaker data and a RAG-enhanced transcription pipeline.
• ROMPAR: Morphological Completion and Demographic Unlearning for Romanian-Accented Speech Recognition — A multi-task adversarial framework with LLM-guided decoding improves ASR for accented speech by reducing demographic bias and completing morphology.
• Improving quality and robustness in LLM-based text-to-speech systems — LLM-based TTS quality improves through targeted techniques addressing accent leakage, expressiveness, and reliability.
• Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech — A diarization-guided, multi-stage ASR adaptation system significantly improves multilingual two-speaker conversational speech recognition.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.