Meet the Writer: Hacker Noon's Contributor Prakshal Doshi, Site Reliability Engineer

2026-07-24 · Source: HackerNoon · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, medium

Summary

Prakshal Doshi, a Site Reliability Engineer at Apple, leads a global SRE team focused on large-scale infrastructure and is particularly interested in AI's impact on engineering. His latest HackerNoon story, "Scaling AI Inference on Kubernetes: The Case for Token-Based Autoscaling," addresses a critical challenge in LLM inference: the inadequacy of request count as a scaling signal. Doshi argues that requests vary significantly in computational load (e.g., 200-token vs. 8,000-token requests), making token throughput a more accurate metric for measuring GPU capacity consumption. He details an implementation using vLLM metrics, Prometheus, and Kubernetes HPA with custom metrics, also emphasizing the importance of redefining Service Level Objectives (SLOs) around time-to-first-token for user experience. Doshi consistently writes about cloud, Kubernetes, reliability, observability, and the practical challenges of deploying AI at scale, aiming to bridge the gap between AI prototypes and robust production systems.

Key takeaway

For MLOps Engineers or AI Engineers scaling LLM inference on Kubernetes, relying solely on request count for autoscaling will lead to inefficient resource utilization. You should implement token-based autoscaling, measuring token throughput to accurately reflect GPU capacity consumption. Consider integrating vLLM metrics with Prometheus and Kubernetes HPA for custom scaling. Additionally, redefine your Service Level Objectives to prioritize time-to-first-token, as this directly impacts user experience and system responsiveness. This approach ensures more reliable and cost-effective production AI deployments.

Key insights

LLM inference scaling requires token throughput metrics, not request counts, for accurate GPU capacity utilization.

Principles

• Request count is an inaccurate LLM scaling signal.
• Scale LLM inference based on token throughput.
• SLOs for LLMs should prioritize time-to-first-token.

Method

Implement token-based autoscaling using vLLM metrics, Prometheus, and Kubernetes HPA with custom metrics to measure actual GPU capacity consumption.

In practice

• Use vLLM metrics for LLM inference workload data.
• Configure Kubernetes HPA with custom token throughput metrics.
• Redefine SLOs to focus on time-to-first-token.

Topics

• AI Inference Scaling
• Kubernetes Autoscaling
• Large Language Models
• Site Reliability Engineering
• Prometheus Monitoring
• vLLM Metrics
• DevOps

Best for: MLOps Engineer, AI Engineer

Related on AIssential

• Reliable LLM Inference at Scale — Reliable LLM inference at scale requires sophisticated capacity management, cost-based routing, and robust runtime failure detection.
• NVIDIA Dynamo on AKS - Autoscaling LLM Inference — Effective LLM inference autoscaling requires specialized tools like NVIDIA Dynamo on AKS to manage variable loads and GPU constraints.
• Serving Large Language Models on Kubernetes at Scale : Connection-Level Overload Protection for GPU… — LLM inference capacity is memory-bound, necessitating external admission control to prevent overload and ensure predictable tail latency.
• Serving LLMs to Millions of Users The Production Playbook Part-4 — Scaling LLM systems requires architectural shifts beyond adding replicas, focusing on efficiency, cost, and resilience.
• Benchmarking inference at scale: coding agents — Production-scale coding agent inference requires full-stack optimization to manage concurrent long-context loads and prioritize Time To First Token (TTFT).
• How to Run Llama 3 on Kubernetes Without Crying — Running LLM inference on Kubernetes demands specialized tooling and a distinct approach from traditional microservices to avoid performance and cost issues.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by HackerNoon.