Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

The article introduces CurioSFT, an entropy-preserving Supervised Fine-Tuning (SFT) method for large reasoning models, designed to overcome the overconfidence and reduced generation diversity caused by standard SFT. This limitation often constricts the exploration space for the subsequent Reinforcement Learning (RL) stage. CurioSFT employs two key components: Self-Exploratory Distillation, which aligns the model with a self-generated, temperature-scaled teacher, and Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to amplify exploration at reasoning tokens while stabilizing factual ones. Experiments on mathematical reasoning tasks using the Qwen2.5-Math-7B base model and OpenR1-Math-46K dataset show CurioSFT improves vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. Crucially, it yields an average improvement of 5.0 points in the RL stage, demonstrating that preserved exploration capabilities translate to concrete gains. The method also exhibits robustness across models like Qwen3-4B-Base and Llama-3.1-8B-Instruct.

Key takeaway

For Machine Learning Engineers optimizing large reasoning models with SFT-then-RL, you should consider integrating entropy-preserving SFT methods like CurioSFT. This approach mitigates the diversity collapse caused by standard SFT, providing a significantly better initialization for the RL stage. By adaptively encouraging exploration at reasoning tokens while stabilizing factual ones, you can achieve greater performance potential, yielding substantial gains on both in-distribution and out-of-distribution tasks.

Key insights

Standard SFT limits RL exploration by reducing model diversity; CurioSFT preserves entropy via adaptive self-distillation for superior downstream performance.

Principles

• Overconfidence in SFT constricts subsequent RL exploration.
• Entropy regularization requires token-level adaptivity to be effective.
• Stable teacher signals improve self-distillation guidance.

Method

CurioSFT employs Self-Exploratory Distillation with a temperature-scaled teacher and Entropy-Guided Temperature Selection, which adaptively modulates temperature based on token uncertainty via binary search.

In practice

• Implement self-distillation with a higher-entropy teacher distribution.
• Dynamically adjust distillation temperature based on token uncertainty.
• Prioritize exploration for high-entropy reasoning tokens.

Topics

• Supervised Fine-tuning
• Reinforcement Learning
• Large Language Models
• Entropy Preservation
• Self-Distillation
• Mathematical Reasoning

Code references

• huggingface/math-verify
• project-numina/aimo-progress-prize

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.
• SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR — SFT overtraining causes entropy collapse, leading to GRPO rank inversion and failure, identifiable by pre-RL entropy.
• InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting — InfoSFT improves LLM fine-tuning by weighting tokens based on informativeness, balancing novelty and stability.
• PriFT: Prior-Support Guided Supervised Fine-Tuning — PriFT improves SFT generalization by reweighting tokens based on a frozen pretrained model's "prior support" for stability.
• Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models — Entropy-Gradient Inversion, a negative correlation between token entropy and logit gradients, fingerprints LRM reasoning.
• Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning — Data strategy for SFT-then-RL pipelines should align with each stage's distinct role in skill acquisition and consolidation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.