We Built the Hardest Test in Human History to Measure AI. It Lasted 18 Months.

2026-06-26 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Intermediate, quick

Summary

In January 2025, researchers from the Center for AI Safety and Scale AI introduced "Humanity's Last Exam," a benchmark intended to measure AI intelligence and withstand rapid obsolescence. This initiative followed approximately two years of observing AI systems consistently breaking previous benchmarks, often rendering decade-long developments obsolete within just eighteen months. The new test was conceived as a response to this "arms race," aiming to create a robust evaluation method that would challenge AI for years, unlike its predecessors which AI quickly surpassed. The project highlights the escalating difficulty in creating durable metrics for increasingly capable AI.

Key takeaway

For AI researchers and developers focused on evaluating model capabilities, recognize that traditional, long-term benchmarks are increasingly futile. Your evaluation strategies must adapt to AI's rapid progress, shifting towards more dynamic, frequently updated, or challenge-based testing methodologies. Consider investing in adaptive evaluation frameworks that can evolve alongside AI, rather than static tests designed for a generation.

Key insights

AI's rapid advancement consistently breaks benchmarks, creating an "arms race" in evaluation.

Principles

• AI progress outpaces benchmark development.
• Traditional long-term benchmarks are unsustainable.

Topics

• AI Benchmarking
• AI Evaluation
• Model Capabilities
• AI Safety
• Benchmark Obsolescence
• Center for AI Safety

Best for: AI Scientist, Research Scientist, Director of AI/ML

Related on AIssential

• 47 - David Rein on METR Time Horizons — AI's ability to complete long-duration tasks for humans is increasing exponentially, providing a unified metric for long-term progress.
• AI #172: The First Fable — AI capabilities are rapidly advancing, necessitating urgent re-evaluation of safety, economic impact, and regulatory frameworks.
• Misplaced panic over AI progress — AI progress in specific domains like coding does not imply general intelligence or indefinite exponential growth.
• METR’s Joel Becker on exponential Time Horizon Evals, Threat Models, and the Limits of AI Productivity — META evaluates AI capabilities and propensities against threat models to assess societal risks and track continuous progress.
• Scientists built the hardest AI test ever and the results are surprising — New benchmarks are essential to accurately measure AI capabilities beyond pattern recognition and identify gaps in expert-level understanding.
• 47 - David Rein on METR Time Horizons — AI's ability to complete long tasks, measured by human completion time, has increased exponentially over the past five years.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.