Token Cost Inequality: Measuring Tokenization Disparities Across Scripts in Roman Urdu and Urdu

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study on Token Cost Inequality (TCI) investigates tokenization disparities between Urdu and Roman Urdu, revealing that semantically equivalent content incurs systematically different tokenization costs across scripts. Researchers introduced TCI, a metric quantifying relative tokenization efficiency, and a multi-axis framework covering token cost, fragmentation, and fixed-budget retention. Findings across cl100k, mT5, and ByT5 tokenizer families indicate that disparities are highly tokenizer-dependent. An "efficiency-retention paradox" shows Roman Urdu preserves more character-level content than native Urdu under fixed token budgets, due to character-per-token density differences. Minimal gains from normalization suggest tokenizer design is the primary cause, impacting input cost estimation and multilingual evaluation.

Key takeaway

For NLP Engineers designing multilingual benchmarks or estimating input costs for Urdu, recognize that Roman Urdu preserves more character-level content than native Urdu under fixed token budgets. This disparity, driven by tokenizer design rather than orthography, means your token budget assumptions will yield unequal surface coverage across scripts. Adjust evaluation metrics and cost models to account for these script-dependent tokenization efficiencies.

Key insights

Semantically equivalent content incurs systematically different tokenization costs across scripts.

Principles

• Tokenization disparities are strongly tokenizer-dependent.
• Fixed token budgets yield unequal surface-coverage across scripts.
• Token cost alone does not fully explain truncation behavior.

Method

Introduce Token Cost Inequality (TCI) metric and a multi-axis framework (token cost, fragmentation, fixed-budget retention) for cross-script tokenization efficiency.

In practice

• Quantify relative tokenization efficiency using TCI.
• Evaluate multilingual models under constrained token budgets.

Topics

• Tokenization
• Multilingual Language Models
• Urdu
• Roman Urdu
• Token Cost Inequality
• NLP Benchmarking

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs — African languages incur significant tokenization penalties in LLMs, leading to higher costs and reduced context capacity.
• Differentially-Private Text Rewriting reshapes Linguistic Style — DP text rewriting alters linguistic style, homogenizing discourse towards a non-involved register.
• Transformer models for Urdu Language — Urdu's low-resource status in AI is improving through Transformer models and increased data development.
• Tokenisation via Convex Relaxations — A new tokenization algorithm, ConvexTok, uses convex optimization for globally optimal vocabulary construction.
• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation — VLMs exhibit substantial script-dependent bias, failing to consistently process identical content across different orthographies.
• Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer — Cross-lingual fine-tuning primarily improves task-format alignment, not language-specific knowledge transfer, especially for weaker models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.