Modular Monolingual Adaptation using Pretrained Language Models

2026-06-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

This work introduces a modular monolingual adaptation strategy for pretrained language models (PLMs) to improve performance in low-resource languages like Scottish Gaelic, Irish, and Quechua (8.5k training instances). Instead of full model finetuning, the proposed method replaces the tokenizer with a language-specific one, initializes new embeddings (using model-based, FastText, or random strategies), freezes these embedding layers, and trains only the non-embedding parameters using a masked language modeling objective. Experiments across BERT, mBERT, and mmBERT models demonstrate that this non-embedding training consistently outperforms or matches full finetuning on mask-filling tasks, while requiring significantly fewer trainable parameters (reducing them by approximately 25% for mBERT) and lowering training costs. For downstream NER and POS tagging tasks, the modular approach performs comparably to full finetuning, validating its effectiveness and efficiency. The custom tokenizer is crucial for performance gains, while embedding initialization choice has a minor impact.

Key takeaway

For Machine Learning Engineers adapting pretrained language models to low-resource languages, you should adopt a modular approach by using a custom tokenizer and freezing embedding layers. This strategy, training only non-embedding parameters, consistently outperforms full finetuning and LoRA on NLU tasks. It also significantly reduces trainable parameters and VRAM usage. Prioritize custom tokenization for substantial performance gains, as embedding initialization choice has a minor impact. This method offers a more efficient and effective path for low-resource language model development.

Key insights

Freezing embeddings during monolingual adaptation of PLMs for low-resource languages prevents overfitting and improves performance.

Principles

• Full model tuning is often unnecessary for low-resource adaptation.
• Custom language-specific tokenizers significantly boost performance.
• Freezing embeddings acts as an effective regularization strategy.

Method

Build a custom WordPiece tokenizer (30k vocab), initialize embeddings (model, FastText, or random), freeze input/output embeddings, then train non-embedding parameters with MLM objective.

In practice

• Implement custom tokenizers for low-resource language LMs.
• Freeze embedding layers to prevent overfitting in low-resource settings.
• Prioritize non-embedding parameter tuning over full finetuning.

Topics

• Low-Resource Languages
• Pretrained Language Models
• Modular Adaptation
• Tokenizer Customization
• Embedding Freezing
• Natural Language Understanding
• Parameter Efficiency

Code references

• knalin55/MMA-PLM

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Modular Monolingual Adaptation using Pretrained Language Models — Modular PLM adaptation, freezing embeddings while tuning the rest, improves low-resource language performance.
• Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models — Kronecker Embeddings drastically cut LLM input-side parameters while improving performance and spelling robustness.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximal Update (\$μ\$P) improves LLM hyperparameter transfer by maximizing the embedding layer's learning rate, a bottleneck in standard parameterization.
• Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling — SMET stabilizes dynamic sparse training for LLMs by addressing cold-start issues and optimizing memory, making sparse pre-training practical.
• Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation — LoRA-Pre reduces optimizer memory overhead by applying low-rank approximation to momentum states in large model training.
• Solving the Embedding Step in NLP Projects — Effective embeddings are crucial for NLP model generalization and reliable downstream task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.