Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

2026-07-24 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Byte-Prefix Marginalization (BPM) is a novel method designed to consolidate open-weight language models from diverse families into a compact student model using on-policy distillation (OPD). This technique addresses the common challenge where full-vocabulary OPD typically requires a shared tokenizer, and existing cross-tokenizer approaches often discard teacher probability mass or misassign it. BPM re-expresses the teacher's next-token distribution over the student vocabulary within a shared byte space. It assigns each teacher token's probability to the longest student token whose byte representation acts as a prefix of the teacher token's bytes, aggregates mapped mass, and handles unmatched mass in a residual category. This process creates a vocabulary-complete, byte-aligned, and mass-preserving target for dense OPD. BPM exactly recovers the teacher-induced byte-prefix marginal in over 99% of training positions and uses a mass-preserving lower bound otherwise. Evaluated with Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM consistently improved six-benchmark avg@8 by 3.7-6.6 points on mathematics and programming tasks, outperforming current cross-tokenizer methods.

Key takeaway

For Machine Learning Engineers consolidating diverse open-weight language models, especially when facing tokenizer incompatibility during on-policy distillation, you should consider Byte-Prefix Marginalization (BPM). This method provides a robust, mass-preserving approach to align teacher and student vocabularies in a shared byte space, significantly improving distillation efficacy. Implementing BPM can lead to student models that outperform those trained with existing cross-tokenizer methods, yielding 3.7-6.6 point gains on benchmarks.

Key insights

BPM enables effective cross-tokenizer on-policy distillation by re-expressing teacher distributions in a shared byte space, preserving probability mass.

Principles

• Cross-tokenizer OPD benefits from byte-aligned probability mapping.
• Mass preservation is crucial for effective distillation.
• Longest byte-prefix matching improves token alignment.

Method

BPM re-expresses teacher next-token distributions in a shared byte space by assigning teacher token probabilities to the longest student token byte-prefixes, aggregating mass, and handling residuals.

In practice

• Consolidate diverse LLMs into smaller student models.
• Improve distillation performance across different tokenizers.
• Apply byte-prefix matching for vocabulary alignment.

Topics

• On-Policy Distillation
• Byte-Prefix Marginalization
• Language Models
• Tokenization
• Model Consolidation
• Vocabulary Alignment

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Cross-Tokenizer LLM Distillation through a Byte-Level Interface — Byte-Level Distillation simplifies cross-tokenizer knowledge transfer by using a common byte-level interface.
• SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation — SimCT recovers lost supervision in cross-tokenizer on-policy distillation by comparing multi-token continuations.
• Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models — TIDE enables cross-architecture distillation for dLLMs, improving performance and efficiency by transferring knowledge from large to small models.
• SRA: Span Representation Alignment for Large Language Model Distillation — Aligning tokenizer-agnostic spans via their attention-weighted Center of Mass improves cross-tokenizer knowledge distillation.
• TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management — TokenMizer uses a knowledge graph to structure LLM session history, significantly reducing context tokens and improving recall.
• EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models — EasyOPD unifies fragmented on-policy distillation methods for LLMs through a shared verl-based framework, improving reproducibility and extensibility.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.