Data Movement Bottlenecks Scaling Past 1e28 Flop

2024-10-18 · Source: Papers & Reports | Epoch AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, long

Summary

Epoch AI researchers Ege Erdil and David Schneider-Joseph's recent paper identifies data movement as a critical bottleneck for scaling large language model training beyond 1e27 FLOP, assuming a maximum 3-month duration. Their analysis indicates that current training runs are approximately 100 times below this efficiency limit, where data movement costs begin to outweigh arithmetic computation. Notably, NVIDIA's H100 GPUs exhibit a lower efficient training scale than A100s due to disproportionate interconnect and memory bandwidth improvements relative to arithmetic throughput. The study details intra-GPU data movement, primarily HBM bandwidth, and inter-GPU communication patterns for data, pipeline, tensor, and expert parallelism. Beyond 1e29 FLOP, a "latency wall" imposes an absolute upper bound on training, which is considered the most challenging constraint due to slow latency improvements in hardware. Potential solutions include aggressive batch size scaling, with recent research suggesting it could push the limit to 1e29 FLOP, or less plausible model depth reduction.

Key takeaway

For AI Architects planning future large language model infrastructure, recognize that current scaling trends will hit data movement and latency limits around 1e27 FLOP within three years. Your strategy must prioritize aggressive batch size scaling or fundamental interconnect improvements. This is crucial to overcome the "latency wall." Investigate emerging research on critical batch size and model depth scaling laws. This will inform your hardware and algorithmic choices for next-generation models.

Key insights

Data movement bottlenecks, especially network latency, will soon limit large language model training scale and efficiency.

Principles

• Training runs beyond 6 months risk hardware/software obsolescence.
• Chinchilla scaling implies quadratic compute growth with parameters.
• Combining parallelism methods minimizes inter-GPU data movement.

Method

The analysis models intra-GPU (HBM) and inter-GPU (data, pipeline, tensor, expert parallelism) data movement to predict LLM training efficiency limits and the "latency wall."

In practice

• Aggressively scale batch sizes to mitigate latency constraints.
• Investigate alternative optimizers for enhanced batch scaling.
• Research critical batch size and model depth scaling laws.

Topics

• Large Language Models
• Distributed Training
• Data Movement Bottlenecks
• GPU Scaling Limits
• Latency Wall
• Batch Size Scaling

Best for: Research Scientist, MLOps Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• The Real Challenge Limiting AI Models Today — Modern AI performance is increasingly limited by memory bandwidth and data movement, not just computational power.
• CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems — CTA-pipelining reduces LLM serving latency by exploiting Cooperative Thread Array dependencies for concurrent multi-GPU kernel execution.
• GPU Scaling from 1 to 261120 threads Part 2 — GPU performance scaling plateaus due to L2 bandwidth saturation and memory-bound operations, despite latency hiding.
• Scaling Laws, Carefully — Scaling laws predict deep learning loss based on model size, data, and compute, but optimal allocation depends critically on experimental scale and data availability.
• This chip startup just raised $135M on a bet that AI’s biggest bottleneck isn’t compute — it’s memory — AI inference is increasingly a memory scaling problem, not solely a compute challenge.
• Fixing GPU Starvation in Large-Scale Distributed Training — GPU underutilization often stems from data I/O and format transformation bottlenecks, not model complexity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Papers & Reports | Epoch AI.