Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new study proposes a "state distribution view" for large language model post-training methods, complementing traditional loss function analysis. For autoregressive policies, a state is defined as a prompt plus a generated prefix. Supervised fine-tuning (SFT) uses fixed dataset states, while reinforcement learning (RL) and on-policy distillation (OPD) train on states induced by the current learner. Using Qwen3-0.6B-Base on GSM8K, with TruthfulQA and MMLU for retention evaluation, the research reveals three key findings. First, mild SFT improves GSM8K with minimal forgetting, but stress SFT causes significant retention loss. Second, OPD from a degraded SFT teacher surprisingly outperforms that teacher across all three benchmarks. Third, a lightweight on-policy RL run enhances GSM8K while preserving retention. These results emphasize that the source and locality of training states are as crucial as the supervision signal itself.

Key takeaway

For machine learning engineers optimizing large language models, understanding the state distribution during post-training is crucial. Your choice between fixed-state methods like SFT and learner-induced state methods like RL or OPD directly impacts model performance and retention. Prioritize mild SFT for stability, and consider on-policy distillation to potentially exceed baseline teacher performance, even from a degraded source, while lightweight on-policy RL can offer improvements without significant forgetting.

Key insights

Post-training effectiveness hinges significantly on the state distribution used for supervision, not just the loss function.

Principles

• Training state distribution is as crucial as the supervision signal.
• SFT uses fixed states; RL/OPD use learner-induced states.
• On-policy distillation can surpass degraded teachers.

In practice

• Consider mild SFT to minimize retention loss.
• Explore OPD for performance gains, even with weaker teachers.
• Implement lightweight on-policy RL for retention-preserving improvements.

Topics

• Large Language Models
• Supervised Fine-tuning
• Reinforcement Learning
• On-Policy Distillation
• State Distribution
• Model Retention

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• OPD+: Rethinking the Advantage Design for On-Policy Distillation — Stop-gradient in on-policy distillation biases reward estimates; OPD+ offers a corrected framework for improved teacher-student model transfer.
• Continual Learning with RL for LLMs — On-policy Reinforcement Learning inherently mitigates catastrophic forgetting and enhances generalization in LLMs, unlike Supervised Finetuning.
• Stage-1 Controls the Entropy Regime, Not the Outcome — Stage-1 warm-starts in VLM post-training primarily control policy entropy, not the final performance outcome.
• Post-Training Matters More Than Pretraining Now: SFT, RLHF, DPO, and GRPO. — Post-training methods are now critical for advancing LLM capabilities beyond mere imitation to complex reasoning.
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — OPD success hinges on compatible thinking patterns and genuinely new teacher knowledge, not just higher scores.
• A Predictive Law for On-Policy Self-Distillation From World Feedback — On-Policy Self-Distillation (OPSD) performance is predictably linear with the initial student-self-teacher performance gap, enabling pre-training outcome anticipation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.