Reiner Pope – The math behind how LLMs are trained and served

2026-07-10 · Source: Dwarkesh Podcast · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Rainer Pope, CEO of Maddox and former Google TPU architect, presents a detailed analysis of LLM training and inference, focusing on how batch size, context length, memory bandwidth, and compute performance dictate latency and cost. A roofline analysis demonstrates that larger batch sizes significantly amortize weight fetches, making inference thousands of times more cost-effective, with optimal batch sizes around 300 times model sparsity. The discussion highlights that latency has a lower bound set by memory access. Mixture of Experts (MoE) layers and expert parallelism are examined, revealing rack-to-rack communication as a bottleneck due to scale-out networks being 8 times slower than NVLink. LLM API pricing, like Gemini 3.1's 50% higher cost for context lengths over 200k tokens and 5x higher output token costs, directly reflects these underlying hardware constraints, particularly memory bandwidth limitations. The lecture also touches on memory tiers for KV cache storage and reversible networks for training memory optimization.

Key takeaway

For AI Architects designing LLM inference systems, you must prioritize maximizing batch size to amortize fixed memory costs, as this is the primary driver of cost efficiency. Your system's latency will be fundamentally constrained by memory bandwidth, especially for longer context lengths. Therefore, you should design Mixture of Experts layers to operate within single-rack NVLink domains to avoid significant slowdowns from slower inter-rack communication. Consider reversible networks for training to optimize memory usage.

Key insights

LLM inference and training costs are fundamentally dictated by hardware architecture, particularly memory bandwidth and batch size optimizations.

Principles

• Batch size amortizes fixed costs, drastically reducing per-token inference cost.
• Memory bandwidth often limits LLM performance more than raw compute.
• Rack interconnects (scale-up vs. scale-out) define MoE layer scalability.

Method

The analysis uses a roofline model comparing memory bandwidth and compute performance against model parameters (weights, KV cache) to estimate inference time and cost, quantifying batch size and context length sensitivities.

In practice

• Optimize LLM inference by maximizing batch size to amortize weight fetches.
• Design MoE layers to fit within single rack NVLink domains for speed.
• Consider memory tiering (HBM, Flash, spinning disk) for KV cache based on hold time.

Topics

• LLM Inference Optimization
• Memory Bandwidth
• Batch Size
• Mixture of Experts
• GPU Interconnects
• Reversible Networks

Best for: MLOps Engineer, AI Engineer, CTO, Machine Learning Engineer, AI Architect, Director of AI/ML

Related on AIssential

• Inference Optimization in LLMs: A Systems View — Inference optimization for LLMs is a layered systems problem, not a single technique, where memory bandwidth is the primary constraint.
• How Do Modern LLMs Cheat the Scaling Laws? (In a Good Way). — Mixture of Experts (MoE) enables massive LLMs by activating only a subset of parameters per token, drastically cutting compute costs.
• The Big LLM Architecture Comparison — Modern LLM architectures prioritize inference efficiency and training stability through diverse attention mechanisms and Mixture of Experts.
• assigning Moe to Gpus to reduce inference and memory usage — Optimizing MoE expert-to-GPU assignment using training logs is a critical challenge for LLM inference efficiency.
• What I Learned From Implementing LLM Architectures From Scratch (And How to Get Started) — Coding LLM architectures from scratch reveals critical design nuances and optimization strategies for memory and performance.
• Streaming Faster Made Our LLM Hub Slower — Batching LLM tokens at the agent level significantly reduces network overhead and improves system stability under load.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Dwarkesh Podcast.