False Sense of Safety in Selective Signal Classification: Auditing Bound Tightness and Exchangeability for Risk Control

2026-06-13 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

False Sense of Safety in Selective Signal Classification" audits selective prediction methods designed to ensure accepted inputs maintain an error rate below a user-defined budget alpha with 1-delta confidence. The study examines four calibration rules—uncertified empirical thresholding (NAIVE), Hoeffding, Clopper-Pearson (CP), and betting (WSR) upper confidence bounds—across anomalous-sound detection (ASD) and AI-generated-image forensics. Findings reveal NAIVE thresholding, common in practice, exceeds its declared budget in 49-73% of synthetic trials (n=200 calibration points) and up to 68% of real-data splits, creating a false sense of safety. Tighter bounds from CP and WSR certify substantial coverage with zero observed budget overruns under exchangeable splits, unlike Hoeffding. However, under grouped deployment with unseen machine types or generators, certified rules overrun in 9-30% of trials, far exceeding delta, indicating a failure of the exchangeability premise. A conservative per-group threshold restores validity but at a severe coverage cost.

Key takeaway

For Machine Learning Engineers deploying selective prediction models, you must rigorously validate the exchangeability premise, especially in grouped deployments involving unseen data types. Relying on uncertified methods like NAIVE thresholding or assuming exchangeability without verification will lead to significant budget overruns, observed at 9-30% in trials, creating a false sense of safety. Prioritize certified bounds like Clopper-Pearson or betting (WSR), and be prepared to implement conservative per-group thresholds to restore validity, even if it means accepting coverage costs.

Key insights

Selective prediction's safety promise is often undermined by uncertified methods or broken exchangeability in grouped deployments.

Principles

• Uncertified methods create false safety.
• Bound tightness is critical for certification.
• Exchangeability is crucial for risk control.

Method

Auditing selective prediction involves testing calibration rules (NAIVE, Hoeffding, CP, WSR) on signal-domain detectors under both exchangeable and grouped deployment scenarios to assess budget overruns.

In practice

• Avoid NAIVE thresholding for safety-critical tasks.
• Prioritize CP or WSR for tighter bounds.
• Validate exchangeability in deployment groups.

Topics

• Selective Prediction
• Risk Control
• Calibration Rules
• Exchangeability
• Anomalous Sound Detection
• AI-Generated Image Forensics

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control — A new joint finite-sample certificate improves risk control and acceptance probability bounds for adaptive selective predictors.
• CauTion: Knowing When to Trust LLMs for Ensemble Causal Discovery — CauTion integrates LLM domain knowledge with statistical ensembles using trust calibration to improve causal discovery accuracy.
• [D] Conformal Prediction vs naive thresholding to represent uncertainty — Conformal Prediction offers theoretical guarantees for uncertainty quantification, unlike naive thresholding.
• Conformal Blindness: A Note on $A$-Cryptic change-points — Conformal Test Martingales can be blind to significant data distribution shifts if p-values remain uniform.
• Selective Prediction from Agreement: A Lipschitz-Consistent Version Space Approach — Selective prediction can be achieved by identifying labels forced by agreement among Lipschitz-consistent classification heads.
• Tight prediction sets under miscalibration — the regularised and sorted refinements of adaptive prediction sets — Regularized and sorted refinements improve adaptive prediction sets by addressing miscalibration and tightening set sizes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.