Self-Policy Distillation via Capability-Selective Subspace Projection

2026-05-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Self-Policy Distillation (SPD) is a novel method for bootstrapping large language models (LLMs) by training on their own generations without external curation signals. Existing self-distillation techniques either rely on costly external feedback or train on raw outputs, which often entangle task-relevant capabilities with noise like stylistic patterns or errors. SPD addresses this by extracting a low-rank capability subspace from the model's gradients on correctness-defining tokens. It then projects key-value (KV) activations into this subspace during self-generation and fine-tunes on the resulting raw outputs using standard next-token prediction loss. Experiments across code generation, mathematical reasoning, and multiple-choice QA show SPD achieves up to 13% improvement over state-of-the-art self-distillation methods without external signals and up to 16% over pre-trained baselines, demonstrating 15% better out-of-domain generalization.

Key takeaway

For AI Scientists and Machine Learning Engineers aiming to improve large language model performance through self-distillation, especially with frontier models lacking external feedback, consider implementing Self-Policy Distillation (SPD). This method offers a robust approach to selectively enhance task-specific capabilities by focusing on relevant signals within self-generated data. You can expect significant gains, including up to 15% better out-of-domain generalization, without the overhead of external curation.

Key insights

Self-Policy Distillation (SPD) enhances LLMs by selectively distilling task-relevant capabilities from self-generated data without external signals.

Principles

• Self-generated outputs often dilute task-relevant signals with noise.
• Capability-selective distillation improves signal for specific tasks.
• External signals for output curation are costly and often unavailable for frontier models.

Method

SPD extracts a low-rank capability subspace from gradients on correctness-defining tokens, projects key-value (KV) activations into this subspace during self-generation, and fine-tunes on raw outputs with next-token prediction loss.

In practice

• Improving LLM performance in code generation.
• Enhancing mathematical reasoning capabilities.
• Boosting accuracy in multiple-choice QA tasks.

Topics

• Self-Policy Distillation
• Large Language Models
• Self-Distillation
• Capability Subspace
• Gradient Projection
• Fine-tuning

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.
• Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving — Small VLMs can achieve superior autonomous driving performance and efficiency through multi-teacher knowledge distillation and targeted attention signals.
• Hugging Face Journal Club: Embarrassingly Simple Self-Distillation Improves Code Generation — Self-distillation via temperature-tuned SFT improves model performance by refining token distribution.
• A Survey of On-Policy Distillation for Large Language Models — OPD overcomes off-policy distillation's exposure bias by enabling LLMs to learn from self-generated trajectories with teacher feedback.
• Self distillation — Self-distillation improves models by creating superior training targets through systemic asymmetry, not by generating new information.
• Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding — Quality-aware self-distillation enhances GUI grounding by filtering unreliable teacher signals and calibrating supervision strength.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.