How fast does it serve? Throughput, latency, and picking the right GPU

2026-05-07 · Source: Anup Jadhav · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, medium

Summary

LLM inference performance hinges on understanding GPU bottlenecks beyond mere capacity. The article details how inference splits into compute-heavy prefill and bandwidth-bound decode phases, with HBM bandwidth being critical for chat-style workloads. For instance, Llama 3 70B at FP8 achieves roughly 48 tokens/s on an H100 SXM (3.35 TB/s) versus 70 tokens/s on an H200 (4.8 TB/s) due to decode's reliance on moving 70 GB weights. It distinguishes storage precision (e.g., MXFP4) from compute precision (e.g., dequantized to BF16), advocating native precision paths like FP8 on Hopper. Key runtime features like continuous batching, prefix caching, and paged attention significantly boost throughput. GPU selection should align with workload: H200/B200 for batched chat, H100/H200 for latency-sensitive, and A10/L4 for cost-sensitive 7B-13B models. For models exceeding 8x largest GPUs, precision drops or multi-node sharding are necessary. Embedding servers, unlike LLMs, prefer many small GPUs.

Key takeaway

For MLOps Engineers deploying LLMs, carefully evaluate your workload's balance of prefill versus decode to select the optimal GPU and runtime. Prioritize HBM bandwidth for chat-style applications, considering H200 or B200, and ensure your runtime supports continuous batching and prefix caching. If your model exceeds single-node capacity, plan for precision drops or multi-node sharding early in architecture planning to avoid costly rework. Do not apply LLM serving instincts to embedding models; they benefit from many smaller GPUs.

Key insights

LLM inference performance is dictated by balancing compute and bandwidth bottlenecks across distinct prefill and decode phases.

Principles

• Decode is HBM bandwidth-bound; prefill is compute-bound.
• Prioritize HBM bandwidth over FLOPs for chat workloads.
• GPU choice must align with workload shape.

Method

Optimize LLM serving by tracking prefill/decode separately, monitoring p99 time-to-first-token, KV cache usage, and planning for long-context requests.

In practice

• Compare HBM bandwidth when selecting GPUs for LLM serving.
• Use continuous batching for significant decode throughput gains.
• Consider A10 or L4 GPUs for cost-sensitive 7B-13B model inference.

Topics

• LLM Inference
• GPU Selection
• HBM Bandwidth
• Continuous Batching
• Model Quantization
• Embedding Servers
• vLLM

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• LLM Model Serving Best Practices: From a Working Endpoint to a Reliable System — Production LLM serving requires optimizing "goodput" and admission control to manage KV-cache pressure and tail latency.
• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption — Multi-model LLM serving requires schedulers to account for non-linear offloading sensitivity and high preemption costs across diverse models and hardware.
• Running AI on mixed hardware for speed and affordability — llm-d optimizes LLM inference on mixed GPU clusters by intelligently routing requests based on KV cache state, significantly boosting throughput and reducing costs.
• LLM profiling in CI/CD: evaluating inference, chip by chip — LLM inference performance is split between compute-bound prefill and memory-bound decode, requiring distinct profiling.
• Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving — Optimizing CPU LLM serving requires separate execution plans for prefill and decode phases due to their distinct computational and memory demands.
• Serving LLMs to Millions of Users The Production Playbook Part-4 — Scaling LLM systems requires architectural shifts beyond adding replicas, focusing on efficiency, cost, and resilience.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Anup Jadhav.