In-House LLM Serving at Netflix

2026-07-17 · Source: Netflix TechBlog - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

Netflix's AI Platform team developed an in-house LLM serving platform, integrating the full stack from model deployment to inference within their existing production environment. This system leverages a unified JVM-based serving architecture, with the Model Scoring Service (MSS) handling larger GPU-based models via NVIDIA Triton Inference Server. By summer 2025, vLLM was selected as the primary inference engine over TensorRT-LLM, chosen for its custom model architecture support, extensibility, and debuggability. The platform integrates vLLM using Triton's vLLM backend for dynamic I/O spec generation, and exposes an OpenAI-compatible API alongside gRPC for broad ecosystem compatibility. Deployment strategies include Red-Black for stable interfaces and Versioned for breaking changes, with a preference for embedding variable configurations to enable Red-Black. Operational improvements include model caching on Amazon FSx for faster cold starts and a unified Prometheus metrics endpoint. The team also scaled constrained decoding by migrating to vLLM V1 by Q4 2025, enabling batch-level, C++ multi-threaded processing to overcome CPU bottlenecks.

Key takeaway

For MLOps Engineers building or optimizing LLM serving infrastructure, recognize that in-house solutions demand meticulous design beyond initial benchmarks. Prioritize inference engines like vLLM for custom model flexibility and standardize on OpenAI-compatible APIs for seamless integration. Implement deployment strategies that embed variable configurations to leverage efficient Red-Black rollouts. Proactively address operational challenges such as model caching and unified metrics to ensure robust, scalable production performance.

Key insights

Building an in-house LLM serving platform demands careful engine selection, API standardization, and robust deployment strategies.

Principles

• Select inference engines based on operational fit.
• Decouple model artifacts from frontend upgrades.
• Standardize on ecosystem-compatible API interfaces.

In practice

• Use vLLM for custom model architectures.
• Cache large LLMs on high-performance file systems.
• Merge Triton and vLLM metrics via an HTTP proxy.

Topics

• LLM Serving
• vLLM
• Triton Inference Server
• Model Deployment
• Constrained Decoding
• MLOps

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• 10 Open-Weight Model Deployment Concepts Every MLOps Engineer Must Know — The gap between model weights and production LLM inference requires specialized deployment knowledge.
• Running a self-hosted LLM in Kubernetes with vLLM — Self-hosting LLMs in Kubernetes with vLLM and LINSTOR provides control over cost, latency, and data residency.
• LLM Model Serving Best Practices: From a Working Endpoint to a Reliable System — Production LLM serving requires optimizing "goodput" and admission control to manage KV-cache pressure and tail latency.
• Accelerate custom LLM deployment: Fine-tune with Oumi and deploy to Amazon Bedrock — Oumi and AWS services streamline LLM fine-tuning and deployment from experimentation to managed production inference.
• Native-speed vLLM transformers modeling backend — "transformers" models now achieve native vLLM inference speeds through dynamic runtime optimizations, eliminating manual porting.
• The Complete Technical Guide to Running LLMs Locally in 2026 — A comprehensive guide to local LLM deployment, balancing hardware math, quantization, and inference engine benchmarks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Netflix TechBlog - Medium.