MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment

2026-06-25 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

MinGram, a Minimalist Unigram tokenizer, simplifies the traditionally heavy and complex training of Unigram tokenizers while maintaining their elegant token-list representation. It achieves this by utilizing a BPE-derived seed vocabulary, Hard EM on a minimum-token path, and a single flat score-pruning step, eliminating the suffix array, forward-backward pass, and iterative prune loop. By prioritizing token count and using a Unigram score only as a tiebreak, MinGram delivers superior compression compared to both BPE and standard Unigram across six languages. A compression-oriented variant matches the strongest token-count compressors while retaining substantially higher morphological alignment. In downstream language-model training, Unigram-family tokenizers, including MinGram, consistently outperform BPE in bits-per-byte.

Key takeaway

For Machine Learning Engineers or NLP Scientists selecting a tokenizer for language model pre-training or text compression, MinGram offers a compelling alternative. You should consider adopting MinGram to achieve better compression than BPE and standard Unigram, while also improving morphological alignment. Its simplified training process and superior bits-per-byte performance in downstream tasks make it an efficient choice for optimizing model efficiency and data representation.

Key insights

MinGram simplifies Unigram tokenizer training, achieving superior compression and morphological alignment for language models.

Principles

• Unigram tokenizers allow straightforward vocabulary editing.
• Simplified training can enhance tokenizer efficiency and performance.
• Prioritizing token count improves text compression.

Method

MinGram's training uses a BPE-derived seed, Hard EM on a minimum-token path, and a single flat score-pruning step.

In practice

• Implement MinGram for improved text compression.
• Utilize MinGram to enhance morphological alignment.
• Apply MinGram in language model training for better bits-per-byte.

Topics

• MinGram
• Unigram Tokenizer
• BPE
• Tokenization
• Text Compression
• Morphological Alignment
• Language Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base — BrahmicTokenizer-131K offers balanced, high-performance tokenization for Indic, English, EU, and code languages at a 131K vocabulary budget.
• I Built a BPE Tokenizer From Scratch in Python And Finally Understand How LLMs Work — Building a BPE tokenizer from scratch demystifies how LLMs process text, revealing its underlying simplicity.
• What Language is This? Ask Your Tokenizer — UniLID leverages UnigramLM tokenization for efficient, robust language identification, especially in low-resource contexts.
• Tokenization’dan TF-IDF’e: Metin Verisini Makine Öğrenmesine Hazırlamak — This content outlines the fundamental steps for preparing raw text data for machine learning models within an NLP pipeline.
• A Triadic Suffix Tokenization Scheme for Numerical Reasoning — Triadic Suffix Tokenization improves LLM numerical reasoning by explicitly encoding magnitude and decimal depth into number tokens.
• MAAM: Anchor-Preserving Compression and Contextual Calibration for Chinese Discriminatory Language Detection — MAAM uses anchor preservation and contextual calibration for robust, compact Chinese discriminatory language detection.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.