Your eval’s confidence interval assumes independent examples. Yours are clustered.

2026-07-27 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

Confidence intervals calculated for model evaluation pass rates, including Wald, Wilson, and Clopper-Pearson methods, often incorrectly assume independent examples. Many evaluation sets, however, contain clustered data, such as multiple questions from the same document, turns from the same conversation, or paraphrases of a single case. This clustering inflates the apparent sample size, causing the computed confidence intervals to be too narrow and leading to false positives where non-significant model improvements are mistakenly declared significant. For instance, a 200-example eval from 40 source documents with an intra-cluster correlation (ICC) of 0.3 effectively yields only 91 independent examples, widening a naive ±3.5 point interval to ±5.2 points. Common clustering types include source, conversation, and template-based examples. The article proposes a cluster bootstrap method to correct this.

Key takeaway

For Machine Learning Engineers evaluating model performance and deciding on shipping improvements, your current confidence intervals are likely too narrow due to clustered evaluation data. This bias risks deploying changes that haven't truly beaten baseline. You should implement a cluster bootstrap method to compute honest confidence intervals, especially for RAG, multi-turn conversation, or template-augmented evals. This ensures you only ship statistically significant improvements, avoiding wasted effort on false positives.

Key insights

Clustered evaluation data invalidates standard confidence intervals, leading to overconfidence in reported model improvements.

Principles

• Correlated examples reduce the effective sample size (n_eff = n / Deff).
• The design effect (Deff) quantifies sample size reduction due to clustering.
• Clustering always biases confidence intervals to be too narrow, increasing false positives.

Method

Implement a cluster bootstrap by resampling entire clusters with replacement, then calculating the confidence interval from the distribution of bootstrap means.

In practice

• Apply cluster bootstrap to RAG evals with source-clustered questions.
• Use for scoring multi-turn conversations where turns are clustered.
• Correct confidence intervals for template-augmented datasets.

Topics

• Confidence Intervals
• Evaluation Metrics
• Clustered Data
• Bootstrap Resampling
• Design Effect
• RAG Evals
• Statistical Significance

Best for: Machine Learning Engineer, Data Scientist, AI Scientist

Related on AIssential

• Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data — Accurate confidence interval reporting for classifier performance metrics, especially with small or nested data, requires specific statistical methods beyond defaults.
• Honesty in Causal Forests: When It Helps and When It Hurts — Honest estimation in causal forests presents a bias-variance trade-off, with its utility dependent on the data's signal-to-noise ratio.
• Evaluating a Regression Model: How Do We Know Our Predictions Are Good? — Effective regression model evaluation requires understanding prediction errors and applying multiple metrics and visualizations.
• How to evaluate clustering with ground truth? — Selecting the right clustering evaluation index depends on whether cluster-level or point-level insights are prioritized.
• Data augmented bootstrap: Unifying confidence interval construction by approximate invariance — Data Augmented Bootstrap (DAB) unifies confidence interval construction by leveraging approximate data invariances, extending classical and modern methods.
• Why Powerful ML Is Deceptively Easy — Part 2 — Spatial ML models require rigorous evaluation frameworks to ensure true generalization beyond observed data.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.