A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It

2026-07-07 · Source: Machine Learning · Field: Science & Research — Life Sciences & Biology, Health & Medical Research, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent analysis reveals a critical flaw in standard (marginal) conformal prediction when applied to imbalanced datasets in drug discovery. While achieving its global 90% coverage target, the method severely under-covers minority classes, with realized coverage dropping to 64.8% for blood-brain-barrier penetration and a mere 4.2% for clinical-trial toxicity. This failure is consistent across models like random forests, graph networks, and frozen chemical language models (p < 0.001), and is explained by a conservation identity linking minority shortfall to majority surplus and imbalance ratio. The issue is often missed due to high aggregate accuracy and overall coverage. Class-conditional (Mondrian) conformal prediction effectively resolves this, restoring minority coverage to target with only a modest increase in prediction-set size. The study also localizes failures to generic molecular scaffolds and proposes a diagnostic, demonstrating that abstaining on affected compounds can shift screening campaigns from net-negative to net-positive utility.

Key takeaway

For AI Scientists and Research Scientists developing predictive models for drug discovery with imbalanced datasets, you must move beyond standard marginal conformal prediction. Your models, even if globally accurate, are likely severely under-covering minority classes, risking critical failures in areas like toxicity screening. Implement class-conditional (Mondrian) conformal prediction to ensure reliable per-class coverage. Consider using the proposed one-number diagnostic to identify and mitigate these hidden risks, improving overall campaign utility.

Key insights

Marginal conformal prediction critically under-covers minority classes on imbalanced datasets, a flaw fixed by class-conditional methods.

Principles

• Minority class coverage shortfall equals majority surplus amplified by imbalance ratio.
• High aggregate accuracy can mask severe minority class under-coverage.
• Model architecture is less critical than baseline calibration for rare labels.

Method

Class-conditional (Mondrian) conformal prediction restores per-class reliability by adjusting prediction sets based on class, closing coverage gaps on imbalanced datasets.

In practice

• Implement class-conditional conformal prediction for imbalanced data.
• Use a one-number diagnostic to detect minority class under-coverage.
• Abstain from affected compounds to improve screening campaign utility.

Topics

• Conformal Prediction
• Imbalanced Datasets
• Drug Discovery
• Mondrian Conformal Prediction
• Minority Class Coverage
• Virtual Screening

Best for: AI Scientist, Research Scientist

Related on AIssential

• Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification — PFNs' class imbalance issues are best addressed by simple thresholding or downsampling due to their unique calibration.
• Online Conformal Prediction: Enforcing monotonicity via Online Optimization — New online conformal prediction methods ensure nested prediction sets across multiple confidence levels.
• Partial Causal Structure Learning for Valid Selective Conformal Inference under Interventions — Selective conformal prediction under interventions can be robustly maintained by learning partial causal structures.
• Divide et Calibra: Multiclass Local Calibration via Vector Quantization — Vector Quantization enables stable, compositional multiclass local calibration by sharing parameters across sparse latent space regions.
• Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing — The framework unifies data- and algorithm-level balancing to improve imbalanced regression performance.
• Audited Conformal Prediction for Classification under Unknown Distribution Shift — Audited Conformal Prediction improves uncertainty quantification under distribution shift using an auxiliary audit model.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.