The 80% power lie

2026-02-19 · Source: Statistical Modeling, Causal Inference, and Social Science · Field: Science & Research — Mathematics & Computational Sciences, Research Methodology & Innovation · Depth: Intermediate, short

Summary

The common claim of 80% statistical power in research studies, often required for NIH funding, is mathematically inconsistent with observed scientific outcomes. For a study to truly have 80% power (meaning an 80% probability of a 95% confidence interval excluding zero), the true effect size must be at least 2.8 standard errors from zero. This implies that studies should routinely yield p-values as low as 0.0005 or even 0.0000016. However, such extremely low p-values are rarely seen in practice, suggesting that the actual power of most studies is significantly lower, potentially as low as 6%. This discrepancy arises because assumed effect sizes are often inflated, literature estimates are biased, and systematic errors and variations across conditions are prevalent, making it much harder to achieve 80% power than simple calculations suggest.

Key takeaway

For AI scientists designing experiments or evaluating research, recognize that routine claims of 80% statistical power are often unrealistic. Your experimental designs should account for the common overestimation of effect sizes and the prevalence of systematic errors, rather than relying on simple power calculations that yield inflated N values. Prioritize robust methodology and realistic effect size assumptions over merely meeting a funding requirement for 80% power.

Key insights

Claims of 80% statistical power in research are often mathematically inconsistent with observed p-value distributions.

Principles

• True effect size must be ≥2.8 standard errors for 80% power.
• Inflated effect size assumptions lead to unrealistic power estimates.

In practice

• Scrutinize studies claiming 80% power for implausibly high p-values.
• Recognize that small N studies rarely achieve high power.

Topics

• Statistical Power
• P-values
• Research Funding
• Effect Size
• Replication Crisis

Best for: AI Scientist, Research Scientist, Data Scientist, AI Student

Related on AIssential

• How Far Are We From True Auto-Research? — Current auto-research agents generate polished papers but critically lack experimental rigor and result integrity.
• Recent discoveries on the acquisition of the highest levels of statistical fallacies — Selection bias can create spurious negative correlations among highly selected groups.
• Why Your 99% Accurate Model Might Actually Be Useless — Data leakage inflates model accuracy by providing unavailable information, leading to real-world failure.
• The stupidity, the criminal vandalism, the wanton destruction of information involved in dichotomisation — Dichotomizing continuous outcomes in clinical trials leads to significant information loss and inflated sample sizes.
• Beyond Rational Illusion: Behaviorally Realistic Strategic Classification — Existing strategic classification models often fail by assuming agents are strictly rational, ignoring real-world cognitive biases.
• How I Made $4,000 This Month Fixing My Clients’ “AI Electricity Bill” — AI's unprecedented power demands are creating substantial, often hidden, compute cost burdens for businesses utilizing cloud-hosted models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Statistical Modeling, Causal Inference, and Social Science.