Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization (BPM) addresses the challenge of consolidating diverse capabilities from open-weight language models with heterogeneous tokenizers into a single compact student model. BPM re-expresses a teacher's next-token distribution over the student vocabulary in a shared byte space. It maps teacher token probabilities to the longest student token byte prefixes, aggregates mass, and handles unmatched mass with a residual category, ensuring a vocabulary-complete, byte-aligned, and mass-preserving target. This method exactly recovers the teacher-induced byte-prefix marginal at over 99% of training positions. Across experiments with Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers and Qwen3.5-2B as student, BPM consistently improved six-benchmark avg@8 by 3.7–6.6 points over baselines. It also identified and mitigated a "whitespace collapse" issue in code generation by masking all-whitespace target positions, preventing a significant performance drop from 49% to 9% for one pair.

Key takeaway

For machine learning engineers consolidating diverse LLM capabilities or improving inference efficiency, Byte-Prefix Marginalization (BPM) offers a robust solution for cross-tokenizer on-policy distillation. You should consider implementing BPM, as it consistently outperforms existing methods by 3.7–6.6 avg@8 points, ensuring accurate knowledge transfer across models with different tokenizers. Crucially, apply the precomputed whitespace-row mask to prevent performance collapse in code generation tasks.

Key insights

Byte-Prefix Marginalization (BPM) enables robust cross-tokenizer on-policy distillation by aligning teacher and student distributions in a shared byte space.

Principles

• Align distributions in a shared byte space for cross-tokenizer transfer.
• Preserve total probability mass in transformed target distributions.
• Greedy longest-prefix matching can map teacher to student tokens.

Method

BPM maps teacher token probabilities to the longest student token byte prefixes, aggregates mass, and assigns unmatched mass to a residual category. It minimizes a generalized Jensen–Shannon divergence.

In practice

• Precompute and apply a mask to exclude all-whitespace target positions.
• Map teacher stop tokens to the student's realized stop token.

Topics

• Cross-Tokenizer Distillation
• On-Policy Distillation
• Byte-Prefix Marginalization
• Language Model Efficiency
• LLM Code Generation
• Model Consolidation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization — BPM enables effective cross-tokenizer on-policy distillation by re-expressing teacher distributions in a shared byte space, preserving probability mass.
• Cross-Tokenizer LLM Distillation through a Byte-Level Interface — Byte-Level Distillation simplifies cross-tokenizer knowledge transfer by using a common byte-level interface.
• MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference — MarginGate achieves deterministic LLM inference by selectively verifying only low-margin steps, significantly reducing overhead.
• Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking — Fine-tuning LLaMA 3 (8B) with knowledge distillation creates efficient, accurate RAG rerankers, avoiding quadratic costs.
• Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models — TIDE enables cross-architecture distillation for dLLMs, improving performance and efficiency by transferring knowledge from large to small models.
• LoRi: Low-Rank Distillation for Implicit Reasoning — Reasoning trajectories in LLMs exhibit low-rank structure, enabling efficient distillation into compact latent processes.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.