Running a self-hosted LLM in Kubernetes with vLLM

2026-07-16 · Source: Cloud Native Computing Foundation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

Michael Troutman of LINBIT, in a post from July 16, 2026, outlines the deployment of a self-hosted large language model (LLM) inference stack within a Kubernetes lab environment. The setup leverages vLLM as a high-performance inference engine and LINSTOR for replicated persistent storage. Specifically, it demonstrates serving the meta-llama/Llama-3.2-1B-Instruct model, a 1B parameter model, by configuring a 50Gi PersistentVolumeClaim with the linstor-csi-lvm-thin-r2 storage class. This ensures model weights are cached persistently, accelerating pod restarts. The vLLM instance exposes an OpenAI-compatible REST API, facilitating integration with existing tools like LangChain or LlamaIndex and supporting hybrid AI strategies. The article provides detailed Kubernetes resource definitions for the PVC, a Hugging Face token secret, and the vLLM Deployment and Service, including testing instructions.

Key takeaway

For MLOps Engineers evaluating LLM deployment strategies, this setup demonstrates a robust pattern for self-hosting. You can achieve predictable costs and data residency by deploying vLLM in Kubernetes with LINSTOR-backed persistent storage. This approach, using an OpenAI-compatible API, allows you to integrate open-source models for sensitive or high-volume workloads while maintaining flexibility for hybrid AI architectures. Consider layering an inference router like llm-d for advanced request dispatching.

Key insights

Self-hosting LLMs in Kubernetes with vLLM and LINSTOR provides control over cost, latency, and data residency.

Principles

• OpenAI API compatibility enables hybrid LLM architectures.
• Persistent storage for model weights speeds up pod restarts.
• Replicated storage enhances fault tolerance for LLM deployments.

Method

Deploy vLLM in Kubernetes using a PersistentVolumeClaim for model caching and a Secret for Hugging Face access, then expose via a Service.

In practice

• Use vLLM for high-performance, concurrent LLM serving.
• Cache model weights on persistent volumes to avoid re-downloads.
• Configure `--gpu-memory-utilization` for CPU-only vLLM deployments.

Topics

• Kubernetes
• LLM Inference
• vLLM
• LINSTOR
• Persistent Storage
• OpenAI API

Code references

• piraeusdatastore/piraeus-operator
• llm-d/llm-d

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• In-House LLM Serving at Netflix — Building an in-house LLM serving platform demands careful engine selection, API standardization, and robust deployment strategies.
• How to Run Llama 3 on Kubernetes Without Crying — Running LLM inference on Kubernetes demands specialized tooling and a distinct approach from traditional microservices to avoid performance and cost issues.
• The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation — LLM agent persistent storage footprint is a critical, overlooked metric with significant variability, requiring dedicated measurement and optimization.
• LLM Observability with Self-Hosted Langfuse and vLLM — LLM observability with traces is critical for debugging and optimizing complex, non-deterministic AI applications.
• I Deployed Local LLMs in Production for a Year. Part 1: The Mental Model — Production LLM deployment requires understanding hidden queues, KV cache memory, and multi-phase model loading beyond basic tutorials.
• Accelerate custom LLM deployment: Fine-tune with Oumi and deploy to Amazon Bedrock — Oumi and AWS services streamline LLM fine-tuning and deployment from experimentation to managed production inference.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Cloud Native Computing Foundation.