Tokenisation via Convex Relaxations

2026-05-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

ConvexTok introduces a novel tokenization algorithm that reformulates tokeniser construction as a linear program, solved via convex optimization. Unlike traditional greedy methods such as BPE and Unigram, which make locally optimal decisions, ConvexTok aims for global optimality across the entire vocabulary. This approach consistently improves intrinsic tokenization metrics and enhances the bits-per-byte (BpB) efficiency of language models. While its impact on downstream task performance is less consistent, ConvexTok offers a unique capability: users can certify their tokeniser's optimality, empirically found to be within 1% of the theoretical optimum at common vocabulary sizes.

Key takeaway

For NLP engineers optimizing language model performance, consider integrating ConvexTok into your tokenization pipeline. This method offers a verifiable path to near-optimal tokenization, potentially reducing bits-per-byte and improving intrinsic metrics compared to greedy algorithms like BPE. Evaluate its impact on your specific downstream tasks, but leverage its optimality certification to ensure foundational efficiency.

Key insights

A new tokenization algorithm, ConvexTok, uses convex optimization for globally optimal vocabulary construction.

Principles

• Tokenizer construction can be framed as a linear program for global optimization.
• Greedy tokenization methods are inherently suboptimal.

Method

Formulate tokeniser construction as a linear program, then solve it using convex optimization tools to yield a globally optimal tokeniser.

In practice

• Improve language model efficiency by reducing bits-per-byte (BpB).
• Certify tokeniser optimality within 1% of theoretical bounds.
• Enhance intrinsic tokenization metrics.

Topics

• Tokenization
• Convex Optimization
• Linear Programming
• Natural Language Processing
• Language Models
• BPE

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Compute Optimal Tokenization — Optimal language model scaling depends on data in bytes, not tokens, with a compute-dependent optimal compression rate.
• Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning — ICT stabilizes LLM reasoning by focusing on token-level distributional deviations, preventing entropy collapse and explosion.
• Mean-Field Parallel Decoding for Discrete Diffusion Language Models — A training-free framework coordinates parallel token updates in discrete diffusion models, improving generation quality and latency.
• The Language of AI: Understanding Tokens in NLP — Tokens are the fundamental units AI uses to process text, impacting cost, memory, and understanding in LLMs.
• Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice — Adaptive image tokenization using global tokens and dynamic filtering improves compression and reconstruction quality.
• Emergent retokenization symmetry in large language models: phenomenology and applications — Large language models partially develop retokenization symmetry, allowing diverse outputs from semantically equivalent input segmentations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.