Is a compute crunch coming?

2026-01-28 · Source: Epoch AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

The article analyzes the world's current AI inference capacity and future demand, modeling prefill and decode runtimes while incorporating efficiency techniques like chunked prefill and speculative decoding. Calibrated against SemiAnalysis's InferenceX data, the analysis estimates that current Nvidia GB200 and GB300 chips (1.9 million and 1.5 million GPUs, respectively, as of Q4 2025) could produce between 500 million and 20 billion output tokens per second from a Kimi K2.6-like model, depending on context length. Global inference capacity is projected to triple annually. Comparing this supply to demand proxies, such as Google's token usage and extrapolated software engineer usage (estimated at 200 million to 4 billion tokens per second, growing 10x per year), the article suggests demand is likely outpacing supply, particularly for long-context agentic AI workloads. This imbalance could lead to a "compute crunch," driving up prices for frontier capabilities and prompting a shift towards smaller, more efficient models for everyday users.

Key takeaway

For AI Architects and Machine Learning Engineers planning infrastructure for large language models or agentic AI, the projected compute crunch necessitates a strategic shift. Your current capacity planning should account for demand growing significantly faster than supply (10x vs. 3.4x annually). Prioritize developing more efficient model serving techniques and consider a tiered model strategy, reserving frontier models for high-value, long-context applications while deploying smaller, cost-effective models for everyday users. This approach mitigates rising token prices and ensures broader accessibility.

Key insights

AI inference demand is rapidly outpacing supply, signaling an imminent compute crunch for large models.

Principles

• Inference capacity grows ~3.4x/year.
• Demand for tokens grows ~10x/year.
• Longer contexts increase compute costs.

Method

Modeled prefill and decode runtimes, incorporating chunked prefill and speculative decoding. Calibrated against SemiAnalysis InferenceX data for Kimi K2.5/2.6 on Nvidia GPUs to estimate global token output.

In practice

• Use chunked prefill to overlap compute/bandwidth.
• Implement speculative decoding for 1.6-2x throughput.
• Optimize for bandwidth in decode-bound stages.

Topics

• AI Inference Capacity
• Compute Crunch
• LLM Performance Optimization
• Speculative Decoding
• NVIDIA Blackwell GPUs
• Agentic AI Workloads

Code references

• MoonshotAI/Kimi-K2.5

Best for: MLOps Engineer, AI Engineer, NLP Engineer, Machine Learning Engineer, AI Architect, AI Scientist

Related on AIssential

• 🔮 Exponential View #568: The labs are rationing. Did you notice? — The AI industry faces a severe compute crunch, impacting growth and resource allocation for major tech companies.
• 2026.22: Luceing Their Mind — The future of AI compute will diverge, with agentic inference prioritizing memory capacity and cost over raw speed.
• AI Compute Stopped Being a Training Problem — AI compute costs have structurally inverted, shifting from one-time training capital expenditure to recurring inference operational expenditure.
• A PhD pivot, a stint in VC and a rumoured Anthropic deal: why Europe is watching Walter Goodwin — AI inference demand is growing exponentially, creating a strategic opportunity for specialized chip development to challenge incumbents.
• AI Is Drying Up Computing Power — Unmet demand for AI compute capacity is forcing companies to prioritize profitable applications and ration access.
• 📈 Data to start your week: AI boom, nowhere near the ceiling — AI compute demand significantly outpaces supply, driving up GPU prices and leading to rationing by providers.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Epoch AI.