Compute Trends

2026-07-14 · Source: Papers & Reports | Epoch AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Emerging Technologies & Innovation · Depth: Expert, medium

Summary

A new dataset and analysis of machine learning model training compute since 1950 reveals three distinct eras: the Pre-Deep Learning Era, the Deep Learning Era, and the Large-Scale Era. This research, based on over 120 milestone ML systems, finds that training compute has grown by a factor of 10 billion since 2010, with a doubling rate of approximately 5-6 months. This growth significantly exceeds Moore's Law. The study also refines previous findings, reporting a slower doubling time of 6 months compared to OpenAI's earlier estimate of 3.4 months. The methodology involved counting operations or estimating GPU-time for milestone systems, focusing on clear importance, relevance, and uniqueness. These findings offer novel insights into compute's significance for AI development and future governance.

Key takeaway

For AI Scientists and Research Scientists planning future model development, you should account for a training compute doubling rate closer to 6 months, not 3.4 months, when projecting resource needs. This revised understanding of compute trends, especially the emergence of a "Large-Scale Era," suggests that scaling models requires substantial corporate resources. Consider contributing detailed hardware and training process information to improve future compute trend analyses.

Key insights

ML training compute growth, while still rapid, is slower than previously thought and defines three distinct historical eras.

Principles

• ML progress is driven by algorithms, data, and compute.
• Training compute growth exceeds Moore's Law.
• Milestone systems reveal key compute trends.

Method

Training compute is determined by counting arithmetic operations or estimating GPU-time based on hardware and training duration for milestone ML systems, focusing on final training runs.

In practice

• Publish GPU model and training iterations for reproducibility.
• Develop tools for automatic compute usage measurement.

Topics

• Machine Learning Compute
• Training Compute Trends
• Deep Learning Era
• Large-Scale Models
• AI Hardware
• Compute Governance

Best for: AI Scientist, Research Scientist, Policy Maker

Related on AIssential

• Can Ai Scaling Continue Through 2030 — AI scaling to 2e29 FLOP by 2030 is feasible, primarily constrained by power and chip manufacturing, not data or latency.
• More open questions about AI — Critical open questions challenge current assumptions about AI's future development, access, and economic impact.
• On the slow death of Scaling (birth of Adaption Labs) | Sara Hooker | HF ML Club India EP2 — Monolithic AI scaling is ending; the future is adaptive, efficient, post-training intelligence.
• The next step towards AGI — AI progress is accelerating due to diverse innovations, shifting the focus from raw scale to sample efficiency and data compression.
• The State of AI Compute — AI compute capacity is undergoing an exponential expansion, signaling a major shift in economic control.
• AI Compute Stopped Being a Training Problem — AI compute costs have structurally inverted, shifting from one-time training capital expenditure to recurring inference operational expenditure.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Papers & Reports | Epoch AI.