Stop Using Brier Score Wrong

2026-04-04 · Source: Valeriy’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, quick

Summary

A recent analysis revealed that Platt scaling and isotonic regression, two widely used calibration techniques, frequently degrade the performance of strong machine learning models. Across 30 distinct datasets, Platt scaling improved log-loss in only 49.8% of cases, indicating its effectiveness is no better than a random chance. This finding challenges the conventional assumption that these post-hoc calibrators consistently enhance model reliability, particularly for already well-performing models. The study suggests a "calibration paradox" where applying standard calibration methods can be detrimental rather than beneficial.

Key takeaway

For machine learning engineers evaluating post-hoc calibration strategies, you should critically assess the impact of Platt scaling and isotonic regression on your specific models. Do not assume these methods will universally improve performance; instead, rigorously test their effect on metrics like log-loss, especially for models already exhibiting strong predictive capabilities, to avoid unintended degradation.

Key insights

Common calibration methods, Platt scaling and isotonic regression, often degrade strong model performance.

Principles

• Calibration can worsen strong models.
• Platt scaling is often no better than random.

Topics

• Brier Score
• Platt Scaling
• Isotonic Regression
• Model Calibration
• Log-loss

Best for: AI Engineer, NLP Engineer, Computer Vision Engineer, Machine Learning Engineer, Data Scientist

Related on AIssential

• Platt Scaling Destroyed My Model — Blindly applying Platt scaling to well-calibrated models degrades performance by reducing sharpness.
• Calibeating Prediction-Powered Inference — Post-hoc calibration of prediction scores significantly enhances semisupervised mean estimation efficiency without retraining models.
• Your Model Scored 96% Accuracy. Think That’s Success? Think Again. — Model accuracy alone is insufficient; calibrated probabilities and cost-optimized thresholds drive real business value.
• A Deep Dive into Calibration of Language Models: Platt Scaling, Isotonic Regression, Temperature Scaling — LLMs are often miscalibrated; post-hoc methods like Temperature Scaling, Platt Scaling, and Isotonic Regression can align confidence with accuracy.
• The Calibration Paradox: Why Post-Hoc Calibration Hurts Your Best Models — Random Forest `predict_proba` outputs are inherently flawed and not improved by adding more trees.
• Survey Statistics: Individualism and the CV Noise Problem — Individual-level log loss often fails to differentiate substantively important model improvements in binary data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Valeriy’s Substack.