When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

Position-Weighted On-Policy Self-Distillation (PW-OPSD) is introduced to address the uniform token weighting limitation in standard On-Policy Self-Distillation (OPSD) for mathematical reasoning. The authors developed a "branch-viability diagnostic" to evaluate teacher-token reliability, discovering that an "oriented within-sequence position score" is the most effective predictor, achieving an area-under-ROC-curve (AUROC) of 0.83 with a 95% cluster-bootstrap interval of [0.66,0.95] on Qwen3-4B, significantly outperforming local uncertainty scores (at most 0.57). Motivated by this, PW-OPSD applies an increasing sigmoid position weight to the forward-KL objective, providing stronger supervision for later, more reliable tokens. Evaluations show PW-OPSD improves AIME 2024 and AIME 2025 Avg@12 scores by +1.0 and +1.1 points, respectively, and demonstrates consistent aggregate Avg@12 improvements on DeepSeek-R1-Distill-Llama-8B and Olmo-3-7B-Think, all without additional teacher computation.

Key takeaway

For Machine Learning Engineers optimizing reasoning model distillation, you should consider implementing position-weighted supervision. Standard on-policy self-distillation often overweights early, less reliable tokens. By applying an increasing position weight, you can enhance learning on more reliable later tokens, improving performance on benchmarks like AIME 2024 and 2025 by over 1.0 points without increasing teacher computation. This approach helps you focus training where the teacher signal is strongest.

Key insights

Teacher-token reliability in reasoning distillation is trajectory-structured, not solely based on local ambiguity.

Principles

• Local ambiguity does not equate to low teacher reliability.
• Early tokens in a reasoning trace are less reliable.
• Later tokens in a reasoning trace are more reliable.

Method

PW-OPSD applies an increasing sigmoid position weight to the forward-KL objective in OPSD, enhancing supervision for later, more reliable tokens without extra teacher computation.

In practice

• Use a branch-viability diagnostic to assess token reliability.
• Weight distillation loss based on token position in sequence.
• Prioritize later tokens for stronger supervision in reasoning.

Topics

• On-Policy Self-Distillation
• Teacher Reliability
• Mathematical Reasoning
• Qwen3-4B
• Position-Weighted Learning
• Knowledge Distillation

Code references

• SaFo-Lab/PW-OPSD

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• TIP: Token Importance in On-Policy Distillation — Informative tokens for on-policy distillation arise from high student entropy or low student entropy with high teacher-student divergence.
• Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding — Quality-aware self-distillation enhances GUI grounding by filtering unreliable teacher signals and calibrating supervision strength.
• The Signal is in the Steps: Local Scoring for Reasoning Data Selection — Local Naturalness improves LLM reasoning distillation by evaluating short, sequential steps, outperforming global log-probability in multi-teacher settings.
• SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling — SG-OPD improves on-policy distillation by verifying teacher reliability and selectively applying updates based on sign-consistency.
• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.
• X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation — X-Token enables cross-tokenizer knowledge distillation by projecting student distributions into teacher vocabulary space, overcoming prior matching failures.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.