Prediction-Only Distillation in Linear and Logistic Regression

2026-07-20 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new study, "Prediction-Only Distillation in Linear and Logistic Regression," introduces a "fresh-X prediction-mixed" self-distillation (SD) scheme for scenarios where original training data is unavailable. This method trains a pure-distilled student on fresh unlabeled covariates pseudo-labeled by a teacher model, then combines teacher and student predictions affinely. For ridge regression under proportional asymptotics, the derived optimally mixed prediction risk is strictly smaller than the teacher risk, even with out-of-distribution fresh covariates or isotropic covariance. The optimal mixing weight, while not identifiable from unlabeled data alone, can be consistently estimated using a small independent labeled calibration set. The research also demonstrates that this prediction mixing can outperform both the teacher and the pure-distilled classifier in binary logistic regression.

Key takeaway

For research scientists aiming to improve deployed model performance without access to original training data, this prediction-only distillation method offers a viable path. You should consider implementing the "fresh-X prediction-mixed" scheme, which combines teacher and student predictions. Crucially, plan to allocate a small, independent labeled calibration set to accurately estimate the optimal mixing weights, ensuring consistent performance gains in both linear and logistic regression tasks.

Key insights

Prediction-only self-distillation with affine mixing of teacher and student predictions improves risk, even with out-of-distribution data.

Principles

• Prediction mixing can reduce model risk below the teacher's.
• Performance gains are possible even with out-of-distribution fresh covariates.
• Optimal mixing weights require a small labeled calibration set.

Method

Train a student on fresh unlabeled covariates pseudo-labeled by the teacher, then form a final predictor by affinely combining the teacher's and student's predictions.

In practice

• Improve ridge regression model performance.
• Enhance binary logistic regression classifiers.
• Refine models when original training data is lost.

Topics

• Self-distillation
• Ridge Regression
• Logistic Regression
• Prediction Mixing
• Out-of-Distribution Data
• Model Calibration

Best for: AI Scientist, Research Scientist

Related on AIssential

• Prediction-Only Distillation in Linear and Logistic Regression — Prediction-only distillation improves model risk by combining teacher and student predictions, even with out-of-distribution data.
• Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models — Self-distillation is statistically optimal among spectral shrinkage estimators in spiked covariance models.
• Optimal Self-Distillation for Rectified Flow via Linear Probing — Optimal self-distillation provably improves rectified flow models by deriving and applying an optimal mixing coefficient for teacher and true velocities.
• Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding — Quality-aware self-distillation enhances GUI grounding by filtering unreliable teacher signals and calibrating supervision strength.
• Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation — DO-ALL uses Dataset Distillation to create stable, privacy-conscious source references for robust continual test-time adaptation.
• Logit Distillation on Manifolds: Mapping by Learning — Manifold-based logit distillation with LoRA efficiently reduces student model parameters while improving performance over other distillation methods.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.