Your LLM Isn’t Slow Because of the Model. It’s Slow Because of Physics.

2026-07-12 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

LLM inference performance is primarily limited by physics, specifically the finite resources of GPU compute and memory bandwidth, rather than model size alone. GPUs are designed for massive parallelism, performing many simple calculations simultaneously. The article explains that inference has two distinct phases: prefill, which is compute-bound as it processes prompt tokens in parallel, and decode, which is memory-bound due to repeatedly loading model weights and the growing Key-Value (KV) cache for single token generation. Arithmetic intensity determines whether a workload is compute or memory-bound. Optimizations like batching, quantization (e.g., FP16, INT8, FP8), FlashAttention, PagedAttention, and continuous batching (e.g., vLLM) are not isolated tricks but solutions to efficiently manage data movement and memory bandwidth, particularly for the memory-intensive decode phase and KV cache management.

Key takeaway

For AI engineers optimizing LLM serving performance, understanding the hardware bottleneck is crucial. If your LLM feels slow, ask whether the GPU is compute-bound or memory-bound, especially during the memory-intensive decode phase. Focus your efforts on techniques that improve memory bandwidth utilization, such as efficient batching, quantization, PagedAttention for KV cache management, or FlashAttention. This perspective will guide you toward effective engineering trade-offs, rather than simply blaming model size or GPU power.

Key insights

LLM inference speed is often limited by data movement (memory-bound) rather than raw computation (compute-bound).

Principles

• GPU performance is limited by compute or memory bandwidth, whichever is slower.
• Arithmetic intensity predicts if a workload is compute-bound or memory-bound.
• LLM prefill is compute-bound; decode is memory-bound.

In practice

• Use batching to reuse model weights across multiple requests.
• Apply quantization (e.g., INT8, FP8) to reduce memory traffic.
• Implement PagedAttention for efficient KV cache management.

Topics

• LLM Inference
• GPU Architecture
• Memory Bandwidth
• KV Cache
• Quantization
• PagedAttention

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• I Fine-Tuned an LLM on a 4GB Laptop GPU. Here’s the Roofline Math Big Labs Don’t Publish — LLM inference is fundamentally memory-bound, making memory bandwidth the critical bottleneck, not compute or VRAM capacity.
• How KV Cache Speeds Up LLMs for Faster AI Models on GPUs — KV cache and paged attention in VLLM dramatically boost LLM inference throughput and reduce memory waste on GPUs.
• An Interpretable Latency Model for Speculative Decoding in LLM Serving — A new latency model explains speculative decoding performance in LLM serving under dynamic loads.
• LLM profiling in CI/CD: evaluating inference, chip by chip — LLM inference performance is split between compute-bound prefill and memory-bound decode, requiring distinct profiling.
• How To Reduce Inference Costs While Running LLMs — Optimizing LLM inference costs and performance requires a multi-faceted approach across hardware, software, and architectural techniques.
• Prefill/Decode Disaggregation: Why Your GPU Can’t Do Two Things at Once — LLM inference's prefill and decode phases have conflicting hardware demands, necessitating disaggregation for optimal performance and user experience.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.