The calibration failure you can predict before training

2026-03-03 · Source: Valeriy’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A surprising calibration failure in logistic regression models is directly predictable from the ratio of feature dimension (d) to sample size (n) before training. An experiment demonstrated that a model with d/n = 0.0025 (50 features, 20,000 samples) was calibrated (Spiegelhalter Z = -0.63, Brier score ~0.033), while an identical model with d/n = 0.10 (200 features, 2,000 samples) failed catastrophically, exhibiting systematic overconfidence (Z = +19.51, Brier score = 0.078). This miscalibration arises because, contrary to classical asymptotic theory, maximum likelihood estimation is systematically biased when d/n is bounded away from zero, inflating coefficients and pushing predicted probabilities towards extremes. This results in overconfident predictions where ranking may remain valid, but the numerical probabilities are not trustworthy.

Key takeaway

For data scientists and ML engineers deploying classification models, you must assess calibration based on your dataset's feature-to-sample ratio. If your d/n ratio is not tiny, assume miscalibration, even with well-specified models. You should compute d/n for production models and run the Spiegelhalter Z statistic on a held-out set to confirm calibration, as high d/n ratios systematically lead to overconfident probability predictions.

Key insights

Logistic regression calibration fails predictably when the feature dimension to sample size ratio (d/n) is not tiny.

Principles

• Maximum likelihood estimation is biased when d/n is bounded.
• Overconfidence is the default failure mode for logistic regression.
• Ranking performance can mask probability miscalibration.

In practice

• Compute d/n for models in production.
• Run Spiegelhalter Z on held-out sets.
• Assume miscalibration if d/n is not tiny.

Topics

• Model Calibration
• Logistic Regression
• Feature-to-Sample Ratio
• Maximum Likelihood Bias
• Overconfidence
• Spiegelhalter Z Statistic

Best for: Machine Learning Engineer, Data Scientist, AI Scientist

Related on AIssential

• Probability Calibration: Turning Raw Model Scores Into Confidence You Can Actually Trust — Model accuracy is insufficient; confidence scores must be calibrated to reflect true probabilities for reliable decision-making.
• I Pitted XGBoost Against Logistic Regression on 358 Matches. The Boring Model Won. — Model complexity must match data availability and structure to avoid overfitting and miscalibration.
• Platt Scaling Destroyed My Model — Blindly applying Platt scaling to well-calibrated models degrades performance by reducing sharpness.
• The fallacy of predict_proba — `predict_proba` outputs model scores, not calibrated probabilities, impacting downstream decisions and requiring explicit measurement.
• Your Model Scored 96% Accuracy. Think That’s Success? Think Again. — Model accuracy alone is insufficient; calibrated probabilities and cost-optimized thresholds drive real business value.
• Measurement noise limits the advantage of nonlinear models over linear models in biomedical prediction — Measurement noise, not model inadequacy, often limits flexible models' advantage over linear models in biomedical prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Valeriy’s Substack.