In-Place Tokenizer Expansion for Pre-trained LLMs

2026-07-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Tokenizer expansion is an in-place method designed to upgrade the tokenizers of pre-trained Large Language Models, addressing inefficiencies for languages not initially prioritized. Fixed tokenizers often result in high token fragmentation for later-added languages, increasing latency, compute, and energy consumption. This recipe, suitable for model producers, continues existing BPE merges on a multilingual corpus, ensuring most source tokens remain unchanged and new tokens have exact decompositions. The process involves copying existing embedding rows and initializing new ones as the mean of their source sub-token embeddings. A two-stage adaptation, comprising embedding-only training followed by full-model continued pre-training, restores source-checkpoint quality. Applied to LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, it produced LFM2.5-8B-A1B with a 128K tokenizer. This resulted in \$2.4\times$ fewer tokens for Hindi, \$2.6\times$ for Vietnamese, and up to \$4.0\times$ for Thai, yielding an estimated \$2.2$-\$3.7\times$ per-character decode speedup. The model weights and expanded tokenizer are released.

Key takeaway

For Machine Learning Engineers tasked with expanding language support for deployed LLMs, this in-place tokenizer expansion offers a critical pathway. You can significantly reduce token fragmentation and improve decode speeds for new languages, such as achieving \$2.2$-\$3.7\times$ speedups for Hindi and Vietnamese, without compromising original model quality. Consider implementing this two-stage adaptation to efficiently upgrade your model's multilingual capabilities and enhance user experience.

Key insights

In-place tokenizer expansion efficiently upgrades LLMs for new languages, reducing token fragmentation and improving decode speed without quality loss.

Principles

• Tokenizer fixed at pre-training reflects initial priorities.
• Fragmented tokens raise latency, compute, and energy.
• Embedding/LM-head matrices are material for compact models.

Method

Continue BPE merges on multilingual corpus; copy existing embedding rows; initialize new rows as mean of sub-token embeddings. Adapt via embedding-only training, then full-model continued pre-training.

In practice

• Upgrade pre-trained LLMs for new language support.
• Reduce per-character decode time for underrepresented languages.
• Expand vocabulary without full model retraining.

Topics

• Tokenizer Expansion
• Large Language Models
• Multilingual NLP
• BPE Merges
• Model Adaptation
• LFM2-8B-A1B

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Tokenization Explained: How Large Language Models Break Text Into Tokens — Subword tokenization is crucial for LLMs, balancing vocabulary size and text representation efficiency.
• Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation — Efficient adaptation of multilingual embedding models to specific languages is achievable through tokenizer surgery and offline distillation.
• Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language — A compact 135M parameter LLM for Bangla achieves 1B-tier performance by efficiently merging tokens, overcoming low-resource script challenges.
• Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung — Effective NLP for script-diverse, low-resource languages demands tailored adaptation methods that overcome tokenizer fragmentation and misleading evaluation metrics.
• Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit — Latent Context Language Models compress LLM input context before decoding, significantly boosting speed and memory efficiency without major accuracy loss.
• Understanding Large Language Models: From Neural Networks to Production Inference — LLMs process prompts through tokenization, embeddings, and self-attention to probabilistically generate text one token at a time.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.