A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It

2026-07-09 · Source: stat.ML updates on arXiv.org · Field: Science & Research — Health & Medical Research, Life Sciences & Biology, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

Marginal conformal prediction, widely adopted in drug discovery for its reliability guarantees, exhibits a significant and often invisible failure on imbalanced datasets common in virtual screening. While achieving its global 90% coverage target, it severely under-covers minority classes, with realized coverage dropping to 64.8% on blood–brain-barrier penetration (BBBP), 38.9% on a Tox21 toxicity endpoint, and 4.2% on clinical-trial toxicity (ClinTox). This issue is consistent across diverse model architectures—random forests, graph convolutional networks, and pretrained chemical language models—and is not specific to the nonconformity score used. The problem stems from a conservation identity where the minority's shortfall is amplified by the majority's surplus and the class imbalance ratio. Class-conditional (Mondrian) conformal prediction effectively remedies this, restoring minority coverage to target for a modest increase in prediction-set size. The failure is masked by aggregate metrics and concentrates on common, low-information molecular scaffolds.

Key takeaway

For Machine Learning Engineers deploying models in virtual screening with imbalanced molecular datasets, you must move beyond aggregate coverage metrics. Marginal conformal prediction can dangerously under-cover minority classes, leading to costly false confidence. Implement class-conditional (Mondrian) conformal prediction by default and report per-class coverage to ensure reliable predictions for rare but critical compounds. This prevents significant utility losses from confident errors on generic chemical structures.

Key insights

Marginal conformal prediction under-covers minority classes on imbalanced datasets, an invisible failure remedied by class-conditional calibration.

Principles

• Overall coverage guarantees do not imply per-class reliability.
• Minority coverage shortfall scales with class imbalance ratio.
• Generic chemical scaffolds can be blind spots for models.

Method

Class-conditional (Mondrian) conformal prediction computes separate quantiles for each class from calibration points, enforcing per-class coverage guarantees. This restores minority coverage at a modest increase in prediction-set size.

In practice

• Report coverage per class, not just overall.
• Use Mondrian conformal prediction for imbalanced tasks.
• Identify generic scaffolds as potential confident error sources.

Topics

• Conformal Prediction
• Virtual Screening
• Class Imbalance
• Molecular Property Prediction
• Mondrian Calibration
• Uncertainty Quantification

Code references

• Muhammadjon-crypto/Conformal-Minority-Screen

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It — Marginal conformal prediction critically under-covers minority classes on imbalanced datasets, a flaw fixed by class-conditional methods.
• Online Conformal Prediction: Enforcing monotonicity via Online Optimization — New online conformal prediction methods ensure nested prediction sets across multiple confidence levels.
• Divide et Calibra: Multiclass Local Calibration via Vector Quantization — Vector Quantization enables stable, compositional multiclass local calibration by sharing parameters across sparse latent space regions.
• Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification — PFNs' class imbalance issues are best addressed by simple thresholding or downsampling due to their unique calibration.
• Not all Jensen-Shannon Divergence Estimators are Equal — JSD estimation for synthetic tabular data is protocol-dependent, requiring explicit procedure specification for valid comparisons.
• Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing — The framework unifies data- and algorithm-level balancing to improve imbalanced regression performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.