Bayesian credible intervals are not coverage intervals — and the gap matters

2026-06-04 · Source: Valeriy’s Substack · Field: Technology & Digital — Data Science & Analytics, Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

The article highlights a critical distinction between two types of "90%" intervals frequently conflated in statistics: frequentist coverage intervals and Bayesian credible intervals. A coverage interval indicates that the true value will fall within its bounds 90% of the time across repeated experiments, reflecting a property of the procedure itself. In contrast, a Bayesian credible interval states that the posterior probability of the true value being within the interval is 0.9, conditioned on a specific prior distribution and observed likelihood, representing a direct probability statement about the parameter. This fundamental difference, often overlooked in textbooks and production code, has significant implications for statistical inference, decision-making, and the accurate communication of uncertainty, underscoring the importance of understanding the underlying statistical paradigms.

Key takeaway

For Data Scientists interpreting model uncertainty or reporting statistical results, accurately distinguishing between frequentist coverage intervals and Bayesian credible intervals is crucial. You should verify the underlying methodology of any "90%" interval presented or used in your work. Misinterpreting these intervals can lead to incorrect conclusions about parameter certainty or the reliability of your estimation procedures, impacting critical decisions.

Key insights

Frequentist coverage intervals and Bayesian credible intervals are distinct concepts, despite both often being labeled "90%".

Principles

• Coverage intervals describe procedure reliability.
• Credible intervals describe parameter probability.
• Conflating interval types leads to misinterpretation.

In practice

• Review textbook interval definitions.
• Audit production code for interval use.
• Clarify interval type in reports.

Topics

• Bayesian Statistics
• Frequentist Statistics
• Credible Intervals
• Coverage Intervals
• Statistical Inference
• Uncertainty Quantification

Best for: Research Scientist, AI Engineer, NLP Engineer, Data Scientist, Machine Learning Engineer, AI Scientist

Related on AIssential

• Bayesian credible intervals are not coverage intervals — and the apologetics don’t rescue them — Bayesian credible intervals and coverage intervals are distinct, despite sharing "90%" labels, leading to frequent misinterpretation.
• Simulation-based Calibration of Uncertainty Intervals under Approximate Bayesian Estimation — A simulation procedure calibrates approximate Bayesian uncertainty intervals to achieve nominal coverage.
• A data model is not just a “likelihood” — Distinguishing between a data model and a likelihood function is crucial for accurate Bayesian modeling.
• The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection — Current statistical methods for detecting LLM benchmark contamination are unreliable in realistic auditing due to distribution shift and scale.
• Hinge versus margin — the two nonconformity measures practitioners should understand — Conformal prediction yields prediction sets with finite-sample coverage guarantees, not probabilities.
• A consensus–but it’s a consensus of uncertainty. — Consensus in statistical analysis can appropriately include agreement on uncertainty or "not knowing."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Valeriy’s Substack.