Prediction-Only Distillation in Linear and Logistic Regression

2026-07-16 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel self-distillation (SD) approach, termed the "fresh-X prediction-mixed scheme," addresses scenarios where original training data is unavailable, and only a trained predictor and fresh unlabeled covariates exist. This method involves training a pure-distilled student on fresh covariates pseudo-labeled by the teacher, then combining teacher and student predictions affinely. For ridge regression under proportional asymptotics, the optimally mixed prediction risk is shown to be strictly smaller than the teacher risk, even with out-of-distribution or isotropic fresh covariates. The optimal mixing weight, while not identifiable from unlabeled data alone, can be consistently estimated using a small independent labeled calibration set in a single post-training step. Furthermore, for binary logistic regression, prediction mixing demonstrates superior performance compared to both the teacher and the pure-distilled classifier.

Key takeaway

For machine learning engineers deploying models where original training data is no longer accessible, you should consider implementing prediction-only distillation. By training a student model on fresh unlabeled data pseudo-labeled by your existing teacher model and affinely combining their predictions, you can achieve strictly lower prediction risk. Crucially, estimate the optimal mixing weight using a small, independent labeled calibration set to maximize performance gains in both linear and logistic regression tasks.

Key insights

Prediction-only distillation improves model risk by combining teacher and student predictions, even with out-of-distribution data.

Principles

• Self-distillation can reduce prediction risk without original training data.
• Optimal affine mixing of teacher and student predictions is beneficial.
• Benefits extend to out-of-distribution fresh covariates.

Method

Train a pure-distilled student on fresh unlabeled covariates pseudo-labeled by the teacher. Combine teacher and student predictions affinely. Estimate the optimal mixing weight using a small labeled calibration set post-training.

In practice

• Implement prediction mixing for ridge regression models.
• Utilize a small labeled calibration set for mixing weight estimation.
• Apply prediction mixing to binary logistic regression classifiers.

Topics

• Self-Distillation
• Prediction Mixing
• Ridge Regression
• Logistic Regression
• Model Calibration
• Out-of-Distribution Data

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Prediction-Only Distillation in Linear and Logistic Regression — Prediction-only self-distillation with affine mixing of teacher and student predictions improves risk, even with out-of-distribution data.
• Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models — Self-distillation is statistically optimal among spectral shrinkage estimators in spiked covariance models.
• Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding — Quality-aware self-distillation enhances GUI grounding by filtering unreliable teacher signals and calibrating supervision strength.
• Optimal Self-Distillation for Rectified Flow via Linear Probing — Optimal self-distillation provably improves rectified flow models by deriving and applying an optimal mixing coefficient for teacher and true velocities.
• Logit Distillation on Manifolds: Mapping by Learning — Manifold-based logit distillation with LoRA efficiently reduces student model parameters while improving performance over other distillation methods.
• How Small Models Learn to Think Like Giants — Knowledge distillation effectively transfers capabilities from large teacher models to smaller student models, enhancing efficiency and performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.