Benchmarking Byte-Pair Encoding Tokenizers on Different Languages with Bits per Byte

2026-06-26 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A systematic extrinsic cross-language comparison of BPE, SuperBPE, and MorphBPE tokenizers reveals their performance across typologically diverse languages. Researchers evaluated these tokenizers on English, Mandarin, and Hungarian, utilizing bits per byte (BPB) normalized perplexity as the metric, with vocabulary sizes of 8K, 16K, and 32K. The study found that SuperBPE matched BPE for English but underperformed by 0.01–0.06 BPB for Hungarian and Mandarin, indicating that cross-whitespace merging is counterproductive for non-English languages. MorphBPE consistently performed worse than BPE across all settings, showing gaps of 0.02–0.04 BPB at the 32K vocabulary size. These results suggest that standard BPE remains a surprisingly effective baseline.

Key takeaway

For NLP Engineers selecting tokenizers for multilingual language models, you should consider standard BPE as a robust baseline. Newer variants like SuperBPE and MorphBPE, despite theoretical advancements, may underperform BPE, especially for non-English languages like Hungarian and Mandarin. Rigorously benchmark any alternative tokenizer against BPE using metrics like bits per byte normalized perplexity before deployment to ensure optimal cross-lingual performance.

Key insights

Standard BPE tokenization remains surprisingly effective across diverse languages, often outperforming newer variants.

Principles

• Linguistic theory alone does not guarantee practical tokenizer improvements.
• Cross-whitespace merging can be counterproductive for non-English languages.

Method

Extrinsic cross-language comparison of BPE, SuperBPE, and MorphBPE tokenizers using bits per byte (BPB) normalized perplexity on English, Mandarin, and Hungarian with 8K, 16K, and 32K vocabulary sizes.

In practice

• Prioritize standard BPE for multilingual NLP tasks.
• Rigorously benchmark advanced tokenizers for non-English languages.
• Evaluate tokenizer performance using extrinsic metrics.

Topics

• Byte-Pair Encoding
• Tokenization
• Multilingual NLP
• Language Models
• SuperBPE
• MorphBPE

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models — Cross-lingual fairness and tokenization efficiency are not fundamentally at odds in multilingual LLMs.
• What is Tokenization? All you need to know — Tokenization is a critical NLP decision impacting model efficiency, robustness, and language representation, not just a preprocessing step.
• SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark — SomaliWeb v1 provides a high-quality Somali corpus, tokenizer, and language-ID benchmark, addressing critical resource gaps.
• I Built a BPE Tokenizer From Scratch in Python And Finally Understand How LLMs Work — Building a BPE tokenizer from scratch demystifies how LLMs process text, revealing its underlying simplicity.
• Gemma 4 31B Quantization Comparison: Best FP8, NVFP4, and INT4 Models — Quantization effectively compresses Gemma 4 31B, reducing memory and improving speed with minimal accuracy loss.
• CLP-Transfer: Cross-Lingual and Progressive Transfer Learning — CLP-Transfer simplifies cross-lingual model transfer using token overlap and a small helper model, but shows limited downstream task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.