Finite Resources False Discovery Rate Control in Structured Hypothesis Spaces

2026-06-16 · Source: stat.ML updates on arXiv.org · Field: Science & Research — Mathematics & Computational Sciences, Research Methodology & Innovation, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A new framework is presented for False Discovery Rate (FDR) control in large-scale hypothesis testing, specifically addressing challenges arising from finite null draws, which lead to p-value uncertainty, and the inherent structure within hypothesis spaces. The framework introduces two distinct decision rules: Rule 1, a "Model-Free" approach, guarantees exact FDR control but with lower statistical power. Rule 2, based on "Mirror Statistics," maximizes power by adapting mirror-statistic control into count space, offering FDR control with a quantifiable slack. Utilizing a Reproducing Kernel Hilbert Space (RKHS) framework, the system also provides a policy for the efficient allocation of null distribution samples. Empirical evaluations on 10 ADbench datasets and the AlpacaEval 2.0 LLM-as-judge benchmark demonstrate that both rules effectively maintain FDR control, with Rule 2 consistently achieving higher power. The adaptive allocation policy further enhances decision-making and power per unit of budget.

Key takeaway

For Data Scientists managing large-scale hypothesis testing with limited null samples, this framework offers robust FDR control and improved power. You should consider implementing Rule 2 for higher statistical power, accepting a quantifiable slack, and utilize the adaptive allocation policy to efficiently distribute null sampling budgets, especially in structured hypothesis spaces. This approach can significantly increase actionable discoveries.

Key insights

A framework unifies FDR control for finite-data p-value uncertainty and structured hypothesis spaces, improving power and resource allocation.

Principles

• Finite null draws introduce p-value uncertainty in hypothesis testing.
• Structured hypothesis spaces can be leveraged to boost statistical power.
• Mirror symmetry in count space enables robust FDR control with quantifiable slack.

Method

The framework uses count-based likelihoods and RKHS for structured priors. It offers two decision rules: Rule 1 for exact FDR, Rule 2 for higher power with controlled slack. An adaptive policy allocates null samples.

In practice

• Use count-based likelihoods to handle uncertain p-values from finite null samples.
• Apply RKHS methods to exploit inherent structure in hypothesis spaces.
• Prioritize null sample allocation based on uncertainty and potential impact on decisions.

Topics

• False Discovery Rate
• Structured Hypothesis Spaces
• Reproducing Kernel Hilbert Space
• Adaptive Resource Allocation
• Statistical Power
• Large-Scale Hypothesis Testing

Best for: AI Scientist, Research Scientist, Data Scientist

Related on AIssential

• Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference — Simultaneous FDP bounds enable flexible, data-driven threshold selection with rigorous statistical guarantees.
• A Stabilized Path-Space Approach to Diffusion-Based Posterior Sampling — A stabilized path-space framework improves diffusion posterior sampling by connecting it to stochastic optimal control and removing bias.
• Intrinsic Selection and Particle Resampling for Inference-Time Scaling Beyond Domain Verifiability — Intrinsic statistics provide a robust signal for solution quality in Inference-Time Scaling for non-verifiable domains.
• Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables — PH-CS offers data-adaptive selection with post-hoc FDR control by leveraging e-variables and utility maximization.
• Noise Aggregation Analysis Driven by Small-Noise Injection: Efficient Membership Inference for Diffusion Models — Small-noise injection and noise aggregation analysis enable efficient membership inference against diffusion models.
• How Low Can You Go? Active Learning for Sparse Model Discovery in the Ultra-Low-Data Limit — An active learning strategy using E-SINDy efficiently discovers sparse dynamical models by prioritizing informative data samples in ultra-low data regimes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.