The #1 model on the leaderboard dropped to #14 when I included the benchmarks they didn't report.

2026-05-20 · Source: Machine Learning ML & Generative AI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, quick

Summary

A recent analysis of AI model leaderboards uncovered significant selective reporting, with only 31% of possible benchmark scores actually reported across 133 models and 18 tests. The missing 69% predominantly comprised harder evaluations, leading to inflated rankings for some models. By employing statistical correction to estimate these unreported scores, one model that initially ranked #1 plummeted to #14, while others experienced shifts of up to 78 ranks. This research highlights a systemic issue where companies selectively report favorable benchmarks. To address this, a tool called psycrank.com has been developed, alongside a corresponding paper (arxiv.org/abs/2605.11205) and code (github.com/testofschool/evaluation-failure-scaling-law), to provide a more comprehensive and accurate evaluation of model performance.

Key takeaway

For machine learning engineers and data scientists evaluating AI models, relying solely on published leaderboard rankings can be misleading. You should critically assess the completeness of reported benchmark scores, especially when comparing models for deployment or research. Utilize tools like psycrank.com to access statistically corrected evaluations, ensuring your decisions are based on a comprehensive understanding of a model's true capabilities across all relevant benchmarks, not just selectively reported ones.

Key insights

Selective benchmark reporting distorts AI model leaderboards, requiring comprehensive evaluation for accurate performance assessment.

Principles

• Leaderboard rankings can be misleading due to selective reporting.
• Unreported benchmarks often represent harder evaluation tasks.
• Statistical correction can reveal true model performance.

Method

The method involves statistically estimating missing benchmark scores for AI models by analyzing reported data, then re-ranking models based on these corrected, comprehensive evaluations.

In practice

• Use psycrank.com for comprehensive model evaluation.
• Cross-reference reported scores with full benchmark sets.
• Scrutinize models with unusually high performance on few tests.

Topics

• AI Model Evaluation
• Benchmark Bias
• Leaderboard Rankings
• Statistical Correction
• Model Performance
• Responsible AI

Code references

• testofschool/evaluation-failure-scaling-law

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation — A hierarchical framework quantifies model rank uncertainty on multi-task leaderboards using statistical intervals.
• Benchmarks are broken — Academic benchmarks, while generating headlines, are often flawed and fail to accurately measure real-world AI capabilities, being designed for papers rather than products.
• Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations — Public AI evaluations are selective time series requiring Bayesian inference for accurate interpretation, not terminal leaderboards.
• Arena Leaderboard Dataset — Arena released its historical AI capability leaderboards as a public dataset for community analysis and open science.
• Why High Benchmark Scores Don’t Mean Better AI [SPONSORED] — Current AI benchmarks often neglect human experience, necessitating more rigorous, human-centric evaluation methods.
• I Downgraded My AI and Output Got Better — Newer AI models may score higher on benchmarks but can exhibit reduced reliability in production tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning ML & Generative AI News.