Can Ai Scaling Continue Through 2030

2024-06-12 · Source: Papers & Reports | Epoch AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Data Science & Analytics · Depth: Expert, extended

Summary

Research indicates that AI model training compute, consistently growing at approximately 4x per year, can technically continue scaling through 2030. An analysis of four key constraints—power availability, chip manufacturing capacity, data scarcity, and the "latency wall"—suggests that training runs of 2e29 FLOP will likely be feasible by the decade's end. This represents a significant increase in scale, similar to GPT-4 exceeding GPT-2. The most binding constraints are projected to be power supply, with geographically distributed training potentially accommodating 2 to 45 GW, and chip manufacturing capacity, aiming for 100 million H100-equivalent GPUs. Data scarcity, potentially mitigated by multimodal and synthetic data, and the latency wall are considered less immediate obstacles.

Key takeaway

For AI Architects and Machine Learning Engineers planning future model development, you should anticipate the technical feasibility of training models at 2e29 FLOP by 2030, a 10,000x increase over GPT-2. Focus your infrastructure strategies on securing substantial power resources, potentially through distributed data centers, and monitoring advanced chip manufacturing capacity. Be aware that while data scarcity and latency are less immediate, they will require innovative solutions for scaling beyond this decade.

Key insights

AI scaling to 2e29 FLOP by 2030 is feasible, primarily constrained by power and chip manufacturing, not data or latency.

Principles

• AI training compute scales at ~4x annually.
• Power and chip manufacturing are primary scaling bottlenecks.
• Distributed training can mitigate local power limits.

Method

The analysis models four bottlenecks: power (data center capacity, grid growth), chip manufacturing (CoWoS, HBM), data scarcity (multimodal, synthetic), and latency (kernel, communication, batch size). It projects feasible FLOPs by 2030.

In practice

• Deploy geographically distributed data centers.
• Adopt FP8 training for power efficiency.
• Prioritize advanced chip packaging investments.

Topics

• AI Scaling
• Training Compute
• Data Center Infrastructure
• Semiconductor Manufacturing
• Multimodal Data
• Distributed AI Training

Code references

• mosaicml/examples
• ddupont808/GPT-4V-Act
• NVIDIA/nccl

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Compute Trends — ML training compute growth, while still rapid, is slower than previously thought and defines three distinct historical eras.
• The next step towards AGI — AI progress is accelerating due to diverse innovations, shifting the focus from raw scale to sample efficiency and data compression.
• The AI Scaling Problem — Current AI benchmarks misrepresent intelligence by prioritizing knowledge application over continuous knowledge acquisition.
• The data black hole at the center of AI — AI's primary advancement stems from massive data and compute, not improved sample efficiency.
• BREAKING: Sam Altman concedes that we need major breakthroughs beyond mere scaling to get to AGI — Industry leaders are increasingly questioning pure scaling as the path to AGI, advocating for architectural breakthroughs instead.
• ChatGPT Designs Cancer Vaccine — Semiconductor manufacturing, particularly EUV lithography, is the ultimate bottleneck for AI compute scaling, not power or data centers.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Papers & Reports | Epoch AI.