Serving Large Language Models on Kubernetes at Scale : Connection-Level Overload Protection for GPU…

2026-07-27 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, extended

Summary

Serving a Large Language Model (LLM) is not the same problem as serving a stateless web application. An LLM inference server has a hard, non-negotiable capacity ceiling on a single GPU, and when that ceiling is reached the system does not degrade gracefully: throughput plateaus while tail latency grows without bound. This article focuses on the *systems* problem this creates on Kubernetes : how to protect the service from overload and how to observe it correctly — rather than on the GPU-internals mechanics, which I have covered in earlier work.¹ ² We describe the design and production validation of a connection-level admission-control layer placed in front of a vLLM inference server on a private k8s cluster. We report a full concurrency sweep (8–456 concurrent requests) on an NVIDIA L40S GPU, locate the saturation cliff at which the service stops scaling, and show that a simple *nginx limit_conn* bound chosen from the throughput/tail-latency Pareto frontier :- protects the service while retaining 89% of peak throughput.

Key takeaway

For MLOps Engineers deploying LLM inference on Kubernetes, recognize that GPU capacity is memory-bound, not request-rate-bound. Implement connection-level admission control using tools like "nginx limit_conn" to prevent tail latency divergence and ensure predictable service behavior. Critically, align your autoscaling signals with actual GPU memory pressure or queue depth, not CPU utilization or requests-per-second, to avoid misconfigurations that lead to service degradation.

Key insights

LLM inference capacity is memory-bound, necessitating external admission control to prevent overload and ensure predictable tail latency.

Principles

• LLM capacity is a memory budget, not a request rate.
• Fail fast under overload with HTTP 503 responses.
• Scale on the signal closest to the bottleneck.

Method

Deploy an in-container "nginx" reverse proxy with "limit_conn" to enforce a hard concurrency bound. Determine this limit empirically via a concurrency sweep on the GPU, identifying the throughput/tail-latency Pareto frontier.

In practice

• Set "nginx" "server_name" explicitly for "limit_conn" to function.
• Order all timeouts: client > ALB > nginx > vLLM.

Topics

• LLM Inference
• Kubernetes Deployment
• GPU Overload Protection
• Nginx Admission Control
• vLLM
• Autoscaling Signals

Best for: MLOps Engineer, Machine Learning Engineer, AI Engineer

Related on AIssential

• Meet the Writer: Hacker Noon's Contributor Prakshal Doshi, Site Reliability Engineer — LLM inference scaling requires token throughput metrics, not request counts, for accurate GPU capacity utilization.
• How fast does it serve? Throughput, latency, and picking the right GPU — LLM inference performance is dictated by balancing compute and bandwidth bottlenecks across distinct prefill and decode phases.
• How to Run Llama 3 on Kubernetes Without Crying — Running LLM inference on Kubernetes demands specialized tooling and a distinct approach from traditional microservices to avoid performance and cost issues.
• Serving LLMs to Millions of Users The Production Playbook Part-4 — Scaling LLM systems requires architectural shifts beyond adding replicas, focusing on efficiency, cost, and resilience.
• Reliable LLM Inference at Scale — Reliable LLM inference at scale requires sophisticated capacity management, cost-based routing, and robust runtime failure detection.
• LLM Model Serving Best Practices: From a Working Endpoint to a Reliable System — Production LLM serving requires optimizing "goodput" and admission control to manage KV-cache pressure and tail latency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.