X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

X-Token is a novel logit-distribution-based approach for cross-tokenizer knowledge distillation, enabling student models to learn from teachers with incompatible vocabularies. It addresses two key shortcomings of prior full-distribution, logit-based methods: "uncommon-token failure" and "over-conservative matching." X-Token introduces a sparse projection matrix W, initialized from tokenizer-level string rules, and two complementary loss formulations: P-KL, which removes partitioning to align full distributions, and H-KL, which relaxes matching via top-ranked mappings under W. Empirically, X-Token on Llama-3.2-1B outperforms the state-of-the-art GOLD by +3.82 average points with a Qwen3-4B teacher and by +0.5 with a Phi-4-Mini teacher. A two-teacher setup (Phi-4-mini + Llama-3B) further improves performance by +1.3 points.

Key takeaway

For machine learning engineers building models with diverse teacher sources, X-Token offers a robust solution for cross-tokenizer knowledge distillation. You should select between P-KL and H-KL based on a coverage audit of critical tokens, using P-KL for "uncommon-token failure" scenarios and H-KL for "sound-partition" regimes. This approach allows you to effectively combine heterogeneous teachers, potentially surpassing same-family distillation performance.

Key insights

X-Token enables cross-tokenizer knowledge distillation by projecting student distributions into teacher vocabulary space, overcoming prior matching failures.

Principles

• Tokenizer-dependent loss selection is crucial.
• Complementary teachers drive multi-teacher gains.
• Simple static weighting can outperform adaptive schemes.

Method

X-Token uses span alignment, a projection matrix W (initialized from string rules), and two loss formulations: P-KL for full distribution alignment and H-KL for relaxed matching.

In practice

• Use P-KL when critical tokens fall into the uncommon set.
• Apply H-KL when token alignment is structurally sound.
• Combine teachers with diverse strengths for multi-teacher KD.

Topics

• Cross-tokenizer KD
• Knowledge Distillation
• Projection Matrix W
• P-KL Loss
• H-KL Loss
• Multi-teacher Learning
• Vocabulary Alignment

Code references

• huggingface/trl

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation — Pedagogically-inspired knowledge distillation systematically improves student model performance and efficiency.
• TIP: Token Importance in On-Policy Distillation — Informative tokens for on-policy distillation arise from high student entropy or low student entropy with high teacher-student divergence.
• Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillation — Multi-objective LLM unlearning requires harmonizing data representation and employing bidirectional distillation for balanced efficacy and robustness.
• Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models — Switch-KD unifies VLM knowledge transfer in a shared text-probability space for efficient deployment.
• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection — Aligning supervision targets with an LLM's native generation distribution significantly mitigates catastrophic forgetting during knowledge injection.
• Consistently Informative Soft-Label Temperature for Knowledge Distillation — Fixed-temperature knowledge distillation creates inconsistent soft-label entropy, hindering effective knowledge transfer.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.