Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR

2026-07-13 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, long

Summary

A novel tokenizer transplantation pipeline addresses autoregressive collapse in lightweight Automatic Speech Recognition (ASR) models like Moonshine when applied to morphologically rich, non-Latin languages such as Bengali. The core issue identified is the original English-centric byte-level tokenizer, which excessively fragments Bengali words, leading to high token fertility (9.16) and decoding instability. The proposed solution replaces the decoder's vocabulary with a native-script BanglaBERT WordPiece vocabulary and resizes the token embedding matrix. This method reduces token fertility to 1.30, decreasing autoregressive sequence length by 85.8% and entirely mitigating decoding instability. Evaluated on the 882-hour Lipi-Ghor dataset, the modified 61.5M parameter architecture achieves a competitive 21.54% Word Error Rate (WER) and an impressive Real-Time Factor (RTF) of 0.0053, outperforming larger models in efficiency. This research offers a scalable blueprint for cross-script adaptation of compact ASR models without extensive pre-training.

Key takeaway

For Machine Learning Engineers deploying lightweight ASR models to non-Latin, morphologically rich languages, you should prioritize tokenizer adaptation to prevent autoregressive collapse. Your existing English-optimized models will likely fail due to high token fertility. Instead, transplant a native-script vocabulary like BanglaBERT after initial acoustic fine-tuning. This approach, combined with a two-stage recovery optimization, achieves competitive accuracy and high inference efficiency on edge devices, avoiding resource-intensive pre-training.

Key insights

Replacing English-centric tokenizers with native-script vocabularies prevents autoregressive collapse in edge ASR for morphologically rich languages.

Principles

• High token fertility causes autoregressive collapse.
• Decouple acoustic learning from language modeling.
• Two-stage recovery stabilizes new decoder embeddings.

Method

Fine-tune acoustic encoder, transplant native-script vocabulary, then apply a two-stage recovery optimization with high and then reduced learning rates.

In practice

• Adapt Moonshine-Base with BanglaBERT WordPiece.
• Remap special tokens like <CLS> to BOS.
• Use BF16 AMP for memory efficiency.

Topics

• Automatic Speech Recognition
• Tokenizer Transplantation
• Moonshine ASR
• Bengali Language Processing
• Edge AI
• Word Error Rate

Code references

• Sanjidh090/moonshine-base-bn

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR — Replacing English-centric tokenizers with native-script vocabularies prevents autoregressive collapse in edge ASR for morphologically rich languages.
• Scaling ASR for Hutsul Dialect: Multi-Speaker Data Collection, Enhanced Transcription and Cross-Speaker Evaluation — Scaling ASR for low-resource dialects requires multi-speaker data and a RAG-enhanced transcription pipeline.
• UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognition — UMA-Split extends unimodal aggregation to English non-autoregressive ASR by allowing aggregated frames to map to multiple fine-grained tokens.
• Voice Cloning For Any Language | Fine-Tuning Tortoise-TTS | Part 2 — Adapt Tortoise-TTS for custom languages by fine-tuning the autoregressive model and modifying inference code.
• Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model — Discrete diffusion language models can perform parallel speech transcription by refining whole transcripts.
• FreyaTTS Technical Report — FreyaTTS offers efficient, high-quality Turkish TTS via a compact, tokenizer-free, non-autoregressive Diffusion Transformer.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.