The field is underestimating inference compute | Noam Brown

2026-05-20 · Source: ARC Prize · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Expert, extended

Summary

Noam Brown, a prominent AI researcher, discusses the evolving definition and measurement of artificial general intelligence (AGI), highlighting the field's underestimation of inference compute. He initially predicted in 2017 that AI wouldn't write a thought-provoking novel in 10 years, a benchmark he now sees as increasingly challenged. Brown critiques current AGI benchmarks like val perplexity and GSMK for not adequately controlling for prior experience or fine-tuning, advocating for a diversity of evaluations. His "gnomeism" emphasizes inference compute as a key ingredient for intelligence, particularly for reasoning capabilities, arguing that models become more intelligent by "thinking for longer." He also addresses the challenges for academia in competing with industry's vast GPU resources and foresees AI playing a significant role in writing and reviewing research papers.

Key takeaway

For AI Scientists and ML Engineers evaluating model capabilities, recognize that a model's intelligence is increasingly a function of inference compute, not just its base parameters. Your release evaluations, if capped at a certain inference budget (e.g., \$10), may misrepresent a model's true potential, as users can spend more to unlock significantly higher capabilities. Factor this "inference-driven intelligence" into responsible scaling policies and benchmark design to avoid underestimating risks or overstating safety.

Key insights

The field underestimates inference compute's role in AI intelligence, especially for reasoning and AGI benchmarks.

Principles

• Intelligence measurement requires diverse evaluations.
• Priors are crucial for practical AI development.
• AI reviews can surpass average human reviews.

In practice

• Invest in GPU clusters for academic AI research.
• Use AI models for initial paper reviews.

Topics

• Artificial General Intelligence
• Inference Compute
• AI Benchmarking
• Large Language Models
• AI in Academia
• Scaling Laws

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• How Inference Compute Shapes Frontier LLM Evaluation — Frontier LLM evaluation performance is highly sensitive to inference compute and protocol choices.
• The Next AI Bottleneck Isn’t the Model: It’s the Inference System — Misattributing AI system failures to the model rather than inference architecture is a common, costly mistake.
• AI keeps getting smarter, so why does it still fail at obvious things? — AI's apparent intelligence stems from pattern matching, not true comprehension, leading to reliability gaps.
• GPT 5.5 Arrives, DeepSeek V4 Drops, and the Compute War Intensifies — Domain-specific training and cost-efficiency are becoming critical differentiators for new large language models amidst compute scarcity.
• Distinguish between inference scaling and "larger tasks use more compute" — Distinguishing between task size and efficiency is crucial for understanding LLM inference scaling and its economic implications.
• Why AI Models Pause to Think: Test Time Compute Explained — Test Time Compute offers a second scaling axis for AI, enabling models to "think" during inference for improved accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ARC Prize.