Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A new framework based on Self-Distillation Fine-Tuning (SDFT) has been introduced to counteract performance degradation in Large Language Models (LLMs) caused by catastrophic forgetting during Supervised Fine-Tuning (SFT), quantization, and pruning. This framework, which leverages the model's own historical states as a teacher, effectively restores model capabilities without external dependencies. The research provides a theoretical explanation, positing that LLM generative capability relies on the high-dimensional manifold constructed by its hidden layers. Centered Kernel Alignment (CKA) is used to quantify the alignment between student and teacher activation trajectories, demonstrating a strong correlation between performance recovery and manifold alignment. Experiments on Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct across Tooluse and Science tasks show SDFT restores performance, with specific gains of +15-22% in task-specific accuracy for quantization recovery and significant restoration of general capabilities.

Key takeaway

For NLP engineers and research scientists dealing with LLM degradation from fine-tuning or compression, adopting the Self-Distillation Fine-Tuning (SDFT) framework offers a robust recovery mechanism. You should consider using SDFT with historical model states as teachers to restore performance and general capabilities, rather than resorting to costly retraining. Furthermore, integrating Centered Kernel Alignment (CKA) into your evaluation pipeline can provide a quantitative metric for diagnosing and monitoring the severity of capability loss and the effectiveness of recovery efforts.

Key insights

Self-distillation recovers LLM performance by realigning the student model's high-dimensional internal representation manifold with the teacher's optimal structure.

Principles

• LLM generative capability relies on hidden layer manifold structure.
• SDFT acts as an "anchor" pulling degraded parameters to a high-performance manifold.
• CKA quantifies manifold alignment, invariant to orthogonal transformations.

Method

The Self-Distillation Recovery Framework uses a model's historical checkpoints as a teacher to guide a degraded student model, optimizing for both capability recovery and task adaptation through a dual-objective process.

In practice

• Apply SDFT to recover LLM performance post-SFT, quantization, or pruning.
• Use CKA as a diagnostic tool for forgetting severity and manifold misalignment.
• For small LLMs, bootstrap ICL via off-policy distillation before SDFT.

Topics

• Self-Distillation Fine-Tuning
• LLM Performance Recovery
• Catastrophic Forgetting
• Model Compression
• High-Dimensional Manifold

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting — Self-Distillation Fine-Tuning (SDFT) recovers LLM performance by realigning the student model's high-dimensional manifold.
• SHARD: Safe and Helpful Alignment via Self-Reframing Distillation — Large language models can self-refactor sensitive prompt responses into safe, helpful outputs via distillation.
• Memory Sparse Attention to Get to 100 Million Tokens in Context — Self-distillation can harm reasoning by suppressing uncertainty, while sparse attention enables extreme context scaling.
• Learning from the Self-future: On-policy Self-distillation for dLLMs — d-OPSD adapts self-distillation for dLLMs by using suffix conditioning and step-level supervision, significantly boosting sample efficiency.
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective — PEFT evaluation should balance task adaptation (plasticity) with pretrained knowledge retention (stability), where orthogonal finetuning excels.
• Distilling Safe LLM Systems via Soft Prompts for On Device Settings — Soft prompt distillation effectively transfers guard model safety to LLMs for resource-constrained on-device deployment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.