Production LiteLLM on AWS EKS: High Availability with GitOps

2026-06-16 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

A production-grade deployment of LiteLLM on AWS EKS, managed via ArgoCD for GitOps, addresses the complexities of multi-LLM provider environments. This architecture unifies access to over 100 LLM providers, offering automatic scaling, granular budget controls, and 99.9% uptime. The system, designed to handle 500-1000 requests per second with p95 latency under 150ms, integrates a database-enabled LiteLLM proxy, Horizontal Pod Autoscaler (HPA), PostgreSQL 17 StatefulSet, Redis for caching, and an internal AWS Application Load Balancer (ALB). Key design decisions include pod anti-affinity, equal resource requests and limits for predictable HPA behavior, and tuned health checks to accommodate Prisma migrations.

Key takeaway

For MLOps Engineers or AI Architects deploying an LLM gateway in a multi-provider cloud environment, this production-grade LiteLLM architecture on AWS EKS offers a proven blueprint. You should consider adopting its GitOps-driven approach with ArgoCD for auditable, self-healing deployments and leverage its specific HPA, caching, and database configurations to achieve high availability, cost efficiency, and predictable performance for 500-1000 requests/second.

Key insights

A robust LiteLLM deployment on AWS EKS with GitOps unifies LLM access, ensuring high availability and cost efficiency.

Principles

• HPA scales predictably when resource requests equal limits.
• Pod anti-affinity enhances fault tolerance across nodes.
• GitOps provides declarative state, audit trails, and rollback.

Method

Deploy LiteLLM on AWS EKS using ArgoCD, orchestrating components like HPA, PostgreSQL, Redis, and an internal ALB for high availability and cost optimization.

In practice

• Set LiteLLM liveness probe initial delay to 60s for Prisma migrations.
• Configure HPA CPU target at 60% to maintain p95 latency under 150ms.
• Utilize Redis for API key validation and response caching to reduce database load.

Topics

• LiteLLM
• AWS EKS
• GitOps
• ArgoCD
• Kubernetes
• LLM Gateway
• High Availability

Code references

• devops1952445/argocd-playground
• KnowledgeSharing1/lite-llm-hpa

Best for: MLOps Engineer, AI Engineer, AI Architect

Related on AIssential

• GitOps for Low-Code: Engineering a Production-Grade CI/CD Pipeline for n8n on Kubernetes — Low-code platforms such as n8n are increasingly vital for automating workflows and integrating APIs, but their ad-hoc deployment methods prove unsustainable for mission-critical production environments.
• Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod — RunPod offers flexible, auto-scaling GPU infrastructure for rapid LLM deployment, abstracting away hardware management for developers.
• Clarifai 12.3: Introducing KV Cache-Aware Routing — Intelligent routing and caching optimize LLM inference by leveraging KV cache state and pre-warmed infrastructure.
• Data Science Meets Devops: MLOps with Jupyter, Git, & Kubernetes — Independent, reconciler-based controllers offer a resilient alternative to DAGs for ML CI/CD.
• Breaking free of a single datacenter: Practical geo-distributed AI operations with the k0smos platforms — Geo-distributed AI operations are viable using Kubernetes-native multi-cluster orchestration for heterogeneous hardware and dynamic resource management.
• Building Blocks for Foundation Model Training and Inference on AWS — Foundation model scaling now requires integrated infrastructure for pre-training, post-training, and inference, emphasizing compute, network, and storage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.