Algorithmic Progress In Language Models

2026-07-14 · Source: Papers & Reports | Epoch AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

A comprehensive analysis of algorithmic progress in language models reveals that the compute required to achieve a specific performance level has halved approximately every 8 months, with a 95% confidence interval of 5 to 14 months. This rapid advancement significantly outpaces Moore's Law's 2-year doubling time for hardware improvements. However, a Shapley value analysis indicates that 60-95% of performance gains stem from increased compute and training data, while novel algorithms contribute only 5-40%. The relative importance of algorithmic improvements has decreased since 2018. Notable innovations include the Transformer architecture, accounting for nearly two years of algorithmic progress, and Chinchilla scaling laws, contributing 8 to 16 months. The study acknowledges limitations in disentangling specific innovation impacts and predicting performance at small compute budgets.

Key takeaway

For Machine Learning Engineers optimizing language model performance, understand that while algorithmic innovations like the Transformer architecture are significant, massive compute scaling remains the primary driver of recent gains. Your strategy should prioritize access to substantial computational resources alongside exploring algorithmic efficiencies. This insight suggests that relying solely on novel algorithms without scaling compute may yield diminishing returns for achieving state-of-the-art performance.

Key insights

Algorithmic progress in language models halves compute needs every 8 months but contributes less than compute scaling to overall performance gains.

Principles

• Compute efficiency in language models doubles faster than Moore's Law.
• Compute scaling drives more language model performance than algorithmic innovation.
• Specific architectural innovations can yield years of algorithmic progress.

Topics

• Language Models
• Algorithmic Progress
• Compute Scaling
• Transformer Architecture
• Chinchilla Scaling Laws
• Model Performance

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The Sequence Opinion #868: Recursion Is the New Scaling Law — AI progress is shifting from linear scaling to iterative, recursive processes for improvement.
• The Two Papers That Built the World We Live In — Modern AI's rapid advancement stems from Transformer's parallel processing and predictable scaling laws.
• Can Scale Save Us From Plasticity Loss in Large Language Models? — Plasticity loss persists in LLMs, scaling sublinearly with size, suggesting scale alone won't prevent it.
• Yann LeCun Says Next AI Breakthrough is 2 Years Away — Scaling Large Language Models will not lead to human-level AI, necessitating new approaches for real-world intelligence.
• Compute Optimal Tokenization - AI at Meta — Optimal tokenization for language models depends on compute, with parameter counts scaling by data bytes, not tokens.
• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models — Scaling LLMs by expanding computation along the sequence-length dimension with token streams offers a fixed-backbone improvement path.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Papers & Reports | Epoch AI.