Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study

2026-05-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A comparative study benchmarked seven foundation models from five providers on 273 Ukrainian court decisions from the EDRSR, focusing on tokenizer fertility and zero-shot performance across three tasks. The research found that tokenizer fertility varied by 1.6x, with Qwen3 models consuming 60% more tokens than Llama-family models for the same input, directly impacting API costs. NVIDIA Nemotron Super 3 (120B) achieved the highest composite score of 83.1, surpassing Mistral Large 3 (675B total, 41B active), despite Mistral having significantly more parameters and costing three times as much via API. Additionally, few-shot prompting consistently degraded performance by up to 26 percentage points, a finding confirmed by ablations for Ukrainian-language demonstrations.

Key takeaway

For AI/ML teams evaluating foundation models for Ukrainian legal text processing, your model selection process should critically include tokenizer efficiency analysis to manage API costs. You should also default to zero-shot prompting, as few-shot examples can significantly degrade performance for morphologically rich languages like Ukrainian, contrary to common intuition. This approach will optimize both cost and accuracy.

Key insights

Tokenizer efficiency and zero-shot performance vary significantly across foundation models on Ukrainian legal text.

Principles

• Tokenizer fertility impacts API costs.
• Zero-shot often outperforms few-shot for morphologically rich languages.

Method

Benchmarking models on Ukrainian legal text, measuring tokenizer fertility and zero-shot performance, followed by stratified and prompt-sensitivity ablations.

In practice

• Prioritize tokenizer analysis before model selection.
• Use zero-shot prompting as a default for Ukrainian language tasks.

Topics

• Foundation Models
• Ukrainian Legal Text
• Tokenizer Fertility
• Zero-Shot Performance
• Few-Shot Prompting

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The easiest way to improve prompts — Zero-shot and few-shot prompting are core techniques for controlling language model output based on guidance provided.
• The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs — African languages incur significant tokenization penalties in LLMs, leading to higher costs and reduced context capacity.
• Command R+ — The AI field faces challenges in LLM evaluation, zero-shot translation, and a dynamic job market.
• Compute Optimal Tokenization - AI at Meta — Optimal tokenization for language models depends on compute, with parameter counts scaling by data bytes, not tokens.
• DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition — Simulating human pilot annotation with LLMs to refine instructions significantly boosts zero-shot NER performance.
• Getting Started with Zero-Shot Text Classification — Zero-shot classification reinterprets text labeling as a natural language reasoning task, eliminating the need for task-specific training data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.