The C-index illusion: discrimination without calibration in published survival models

2026-07-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

The C-index illusion: discrimination without calibration in published survival models" investigates the limitations of using the concordance index (C-index) alone for evaluating survival models, extending prior normative arguments to real-world, non-clinical applications. The research reproduces three published survival-ML models across hard-drive failure, peer-to-peer credit default, and user disengagement domains. It demonstrates that models achieving high discrimination, such as one with a C-index of 0.9595 (compared to 0.958 reported), can still exhibit severe calibration failures, failing a formal test at p = 2.6e-136. This calibration issue is not attributed to simple shortcuts. Specific findings include a lender's estimated default risk being biased upward by 2-4 percentage points due to misclassifying loan prepayment, and a platform's churn model showing degrading probability estimates with prediction horizon despite a stable global C-index. The study concludes that the problem is often misplaced confidence in a model rather than selecting an incorrect one, and releases a reusable evaluation harness with code.

Key takeaway

For Machine Learning Engineers deploying survival models, relying solely on the C-index for evaluation risks significant miscalibration and biased predictions. You should integrate formal calibration tests and explicitly account for competing risks, such as loan prepayment, to ensure accurate risk estimates. Your models may show high discrimination but still produce degrading probability estimates over time. Consider utilizing the released evaluation harness to thoroughly assess both discrimination and calibration in your survival models.

Key insights

Relying solely on the C-index for survival model evaluation leads to misleading comparisons and misplaced confidence due to ignored calibration.

Principles

• Discrimination metrics alone mislead.
• High C-index doesn't imply calibration.
• Competing risks impact risk estimates.

Method

Reproduce published survival-ML models, validate evaluation instrument against synthetic data, and test pre-registered hypotheses using a Holm-corrected family-wise error rate to assess discrimination and calibration.

In practice

• Formally test model calibration.
• Account for competing risks.
• Assess probability estimates over time.

Topics

• Survival Analysis
• Model Calibration
• C-index
• Competing Risks
• Machine Learning Evaluation
• Predictive Modeling

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• comprisk: A scikit-learn-compatible Python toolkit for competing-risks survival analysis — Correct competing-risks analysis requires specialized methods to avoid biased absolute-risk estimates in time-to-event data.
• Your Model Scored 96% Accuracy. Think That’s Success? Think Again. — Model accuracy alone is insufficient; calibrated probabilities and cost-optimized thresholds drive real business value.
• Unbiased Prevalence Estimation with Multicalibrated LLMs — Multicalibration ensures unbiased prevalence estimation even under covariate shift, outperforming standard calibration methods.
• Conformal Blindness: A Note on $A$-Cryptic change-points — Conformal Test Martingales can be blind to significant data distribution shifts if p-values remain uniform.
• Robust Simulation-Based Inference Under Missing Data via Neural Processes — RISE jointly learns data imputation and parameter inference in SBI to robustly handle missing data and prevent biased posteriors.
• Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study — Perfect internal ML metrics do not guarantee reliability under real-world data distribution shifts.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.