Why goodput matters more than throughput for LLM serving

2026-07-20 · Source: Cloud Native Computing Foundation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

The article, posted on July 20, 2026, by Graziano Casto and Akamas, argues that "goodput" is a more critical metric than raw throughput for Large Language Model (LLM) serving. Goodput is defined as the number of completed requests per second that also meet specific latency targets, such as Time to First Token (TTFT) and Time Per Output Token (TPOT). The analysis used Qwen2.5-7B with vLLM on a single NVIDIA A10G (24 GB) within an EKS cluster, driving load with GuideLLM and monitoring with Prometheus/Grafana. Experiments varied vLLM settings like `gpu_memory_utilization`, `max_num_batched_tokens`, and `max_num_seqs` across chatbot, reasoning, and agentic workloads. Findings showed that while one configuration achieved 50% higher combined throughput, its p95 TPOT was nearly ten times worse (494 ms vs 50 ms), demonstrating that optimizing for raw throughput can severely degrade user experience if latency targets are not explicitly constrained.

Key takeaway

For MLOps Engineers optimizing LLM inference, prioritize "goodput" over raw throughput by explicitly setting Time to First Token (TTFT) and Time Per Output Token (TPOT) targets. Chasing maximum requests per second without latency constraints can lead to a visibly stuttering service, even if raw throughput numbers look impressive. You should implement automated, data-driven searches to find configurations that meet your specific SLOs, as optimal settings are non-obvious and shift with traffic and model changes.

Key insights

Goodput, defined as completed requests meeting latency targets, is a superior LLM serving metric to raw throughput.

Principles

• Raw throughput alone can mask poor service quality.
• Optimal LLM serving configurations are non-obvious and dynamic.
• Latency targets (TTFT, TPOT) dictate user experience.

Method

Benchmark LLM serving by varying `gpu_memory_utilization`, `max_num_batched_tokens`, and `max_num_seqs` with GuideLLM, monitoring TTFT and TPOT via Prometheus/Grafana, and using a windowing rule for stable throughput measurement.

In practice

• Use GuideLLM for LLM load generation.
• Monitor GPU metrics with NVIDIA DCGM exporter.
• Reproduce benchmarks using the provided GitHub setup.

Topics

• LLM Serving
• Goodput
• Throughput Optimization
• Latency Metrics
• vLLM
• Kubernetes
• Performance Benchmarking

Code references

• vllm-project/guidellm
• graz-dev/vllm-benchmark

Best for: MLOps Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• LLM Model Serving Best Practices: From a Working Endpoint to a Reliable System — Production LLM serving requires optimizing "goodput" and admission control to manage KV-cache pressure and tail latency.
• Optimizing LLM latency — MLC, CTranslate2, and vLLM offer significant latency improvements for LLM inference, with varying trade-offs.
• How fast does it serve? Throughput, latency, and picking the right GPU — LLM inference performance is dictated by balancing compute and bandwidth bottlenecks across distinct prefill and decode phases.
• Designing Systems for LLM Integration: Why Most Advice Assumes Your Latency Budget Is Infinite — Effective LLM integration demands engineering for actual latency mechanics, not merely hiding delays with streaming or UI tricks.
• Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving — Disaggregated LLM serving benefits from deflecting prefill tasks to underutilized decode nodes to reduce tail latency and KV-cache transfer.
• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption — Multi-model LLM serving requires schedulers to account for non-linear offloading sensitivity and high preemption costs across diverse models and hardware.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Cloud Native Computing Foundation.