Residual CP and normalized residuals — why your prediction bands are wrong

2026-05-06 · Source: Valeriy’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

Valeriy Manokhin's analysis, "Residual CP and normalized residuals — why your prediction bands are wrong," published on May 6, 2026, critiques common methods for constructing prediction intervals in machine learning models. The article highlights that traditional approaches often fail to account for heteroscedasticity, leading to inaccurate and unreliable prediction bands. It specifically discusses the limitations of using constant prediction intervals across the entire range of predictions and introduces the concept of normalized residuals as a more robust method. The author argues that properly normalized residuals, particularly in the context of conformal prediction (CP), can yield more accurate and adaptive prediction intervals that reflect varying levels of uncertainty.

Key takeaway

For Data Scientists building predictive models, understanding the limitations of constant prediction intervals is crucial. Your current prediction bands might be misleading if your data exhibits heteroscedasticity, potentially leading to incorrect risk assessments. You should investigate incorporating normalized residuals and Conformal Prediction techniques to generate more accurate and adaptive prediction intervals, ensuring your uncertainty estimates are reliable across the entire prediction range.

Key insights

Traditional prediction intervals often fail due to heteroscedasticity, requiring normalized residuals for accuracy.

Principles

• Heteroscedasticity invalidates constant prediction intervals.
• Normalized residuals improve prediction band reliability.

Method

The article advocates for using normalized residuals within a Conformal Prediction framework to construct adaptive prediction intervals that account for varying uncertainty.

In practice

• Implement normalized residuals for prediction bands.
• Apply Conformal Prediction to handle heteroscedasticity.

Topics

• Residual CP
• Normalized Residuals
• Prediction Bands
• Statistical Modeling
• Model Evaluation

Best for: Data Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Conformal Prediction Intervals with Tail-Specific Guarantees — This framework extends Conformal Prediction to offer explicit, calibrated coverage guarantees for both upper and lower tails independently.
• Effects of Training Data Quality on Classifier Performance — Classifier performance degrades universally with training data quality, leading to coincidentally correct but flawed decisions.
• Demystifying Residual Analysis: A Beginner’s Guide to What Your Model Isn’t Telling You — Residual analysis diagnoses model failures by examining prediction errors for hidden patterns and assumption violations.
• The fallacy of predict_proba — `predict_proba` outputs model scores, not calibrated probabilities, impacting downstream decisions and requiring explicit measurement.
• Tight prediction sets under miscalibration — the regularised and sorted refinements of adaptive prediction sets — Regularized and sorted refinements improve adaptive prediction sets by addressing miscalibration and tightening set sizes.
• Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better — Standard Conformal Prediction metrics can be deceptively improved by methods introducing randomness, necessitating new evaluation criteria.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Valeriy’s Substack.