The Sequence Knowledge #898: The Trace Is the Teacher: Distilling Reasoning Into Small Models

2026-07-21 · Source: TheSequence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

In January 2025, DeepSeek successfully distilled advanced reasoning capabilities from its large R1 model into smaller, open-source models using a surprisingly simple method. DeepSeek R1 generated approximately 800,000 detailed "worked solutions" or chains of thought, which were then filtered for accuracy and clarity. These traces were used for plain supervised fine-tuning on various off-the-shelf models, including Qwen (1.5B, 7B, 14B, 32B) and Llama (8B, 70B). Notably, this process avoided complex techniques like reinforcement learning or reverse KL. The results were remarkable: a distilled 32B model began solving competition math, and a 7B model exhibited emergent behaviors such as self-verification and branching reasoning, demonstrating that small dense models could reason like models ten times their size through naive sequence-level imitation.

Key takeaway

For Machine Learning Engineers aiming to deploy efficient reasoning models, DeepSeek's success with naive sequence-level imitation suggests re-evaluating complex distillation methods. You should prioritize generating high-quality, detailed reasoning traces from large teacher models and apply straightforward supervised fine-tuning. This approach can yield smaller models with surprising emergent reasoning capabilities, potentially reducing computational costs and deployment complexity without sacrificing advanced problem-solving.

Key insights

Naive sequence-level imitation of detailed reasoning traces effectively distills complex reasoning into smaller language models.

Principles

• Reasoning distillation benefits from detailed traces.
• Simple supervised fine-tuning can be powerful.
• Emergent reasoning behaviors are possible.

Method

Generate extensive, detailed reasoning traces from a large teacher model. Filter these traces for correctness and readability. Apply plain supervised fine-tuning via next-token prediction on smaller student models.

In practice

• Generate detailed "chains of thought" from large LMs.
• Filter generated traces for correctness.
• Use plain supervised fine-tuning for distillation.

Topics

• Reasoning Distillation
• Supervised Fine-tuning
• Large Language Models
• Small Language Models
• Chain of Thought
• Emergent Capabilities

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces — Large reasoning models employ a two-phase, coarse-to-fine mechanism for multi-label tasks, which can be mechanistically distilled.
• Hidden Thoughts Are Not Secret: Reasoning Trace Exposure in LLMs — Reasoning Exposure Prompting (REP) reveals hidden LLM internal traces, preserving valuable reasoning signals for model distillation.
• LoRi: Low-Rank Distillation for Implicit Reasoning — Hidden-state reasoning trajectories exhibit low-rank structure, enabling efficient distillation for implicit reasoning.
• Sylloscope at SemEval-2026 Task 11: Decoupling Logic from Belief via DeepSeek-Enhanced Distillation in Qwen Models — Distilling DeepSeek-R1's logical auditing into Qwen-3 models via LoRA helps decouple logic from belief.
• Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning — Effective on-policy self-distillation for reasoning compression requires optimizing the prefix support where supervision is applied.
• Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning — Improving initial rollout distribution is crucial for efficient self-reasoning distillation in large language models.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by TheSequence.