Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

2026-07-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

VEXMLM, a vocabulary-extended variant of XLM-R, addresses the degraded performance of multilingual pre-trained language models on low-resource, non-Latin-script languages like Amharic and Tigrinya. This model extends XLM-R's vocabulary with 30,000 Ge'ez-script subwords, derived from a language-specific SentencePiece tokenizer trained on curated Amharic and Tigrinya corpora. Its embeddings are initialized by averaging constituent subwords from XLM-R's original tokenizer. VEXMLM employs a two-stage training strategy: continued masked language modeling, followed by supervised fine-tuning on question answering (QA), named entity recognition (NER), and sentiment analysis (SA). On Amharic/Tigrinya QA, VEXMLM achieves 87.0 EM / 90.0 F1, significantly outperforming XLM-R's 66.0 EM / 78.0 F1 and Glot500's 74.0 EM / 78.0 F1. For SA, it reaches 80.0% accuracy, compared to 77.0% for XLM-R and 46.0% for Glot500. Furthermore, VEXMLM improves OOV-token entity accuracy on NER from 81.4% to 94.3% across 11 African languages.

Key takeaway

For NLP Engineers developing models for low-resource, non-Latin-script languages, consider implementing a vocabulary extension strategy. Your models can achieve substantial performance gains by training language-specific tokenizers and extending existing PLM vocabularies with new subwords. This approach, combined with a two-stage pre-training and fine-tuning regimen, significantly reduces out-of-vocabulary rates and improves task performance, as demonstrated on Ge'ez-script languages.

Key insights

Vocabulary extension and two-stage training significantly boost multilingual PLM performance on low-resource, non-Latin-script languages.

Principles

• Latin-centric tokenizers degrade non-Latin PLM performance.
• Vocabulary extension improves OOV rates for low-resource scripts.
• Two-stage training transfers gains to related unaugmented languages.

Method

Train a language-specific SentencePiece tokenizer. Extend PLM vocabulary with new subwords. Initialize new embeddings by averaging original subword embeddings. Apply two-stage training: masked language modeling then supervised fine-tuning.

In practice

• Extend PLM vocabularies for non-Latin scripts.
• Use averaged embeddings for new subword initialization.
• Implement two-stage pre-training for low-resource tasks.

Topics

• Multilingual PLMs
• Low-Resource NLP
• Ge'ez Script Languages
• Vocabulary Extension
• SentencePiece Tokenizers
• Transfer Learning

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung — Effective NLP for script-diverse, low-resource languages demands tailored adaptation methods that overcome tokenizer fragmentation and misleading evaluation metrics.
• DONDO: Open w2v-BERT Speech-Recognition Base Models for African Languages — Open w2v-BERT 2.0 ASR base models for 27 African languages achieve competitive WERs via annealed fine-tuning and prefix-frame conditioning.
• SIEMENS at #SMM4H–HeaRD 2026: The Impact of Training Strategy and Backbone Selection on BERT-based Multilingual Clinical NER — 5-fold training with checkpoint merging and strategic backbone selection significantly improves multilingual clinical NER.
• Child Support: Leveraging Lexifiers Resources to Support Creoles ASR — Utilizing lexifier language resources via artificial data generation significantly boosts low-resource Creole ASR performance.
• Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language — A compact 135M parameter LLM for Bangla achieves 1B-tier performance by efficiently merging tokens, overcoming low-resource script challenges.
• In-Place Tokenizer Expansion for Pre-trained LLMs — In-place tokenizer expansion efficiently upgrades LLMs for new languages, reducing token fragmentation and improving decode speed without quality loss.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.