Reliable LLM Inference at Scale

2026-05-27 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, medium

Summary

Databricks has developed a robust inference platform for serving frontier LLMs, including open-source models like Kimi and Qwen, and proprietary ones like OpenAI, Gemini, and Claude, processing over 125T tokens per month. The platform addresses significant challenges in achieving reliability and low latency at scale, particularly with spiky demand and the inherent unreliability of high-bandwidth GPU systems. Key architectural components include an inference runtime, an Axon router, an autoscaler, rate limiting, and a capacity management algorithm. Databricks introduced "model units" to quantify request cost, enabling cost-based load balancing with Dicer and autoscaling that saved over 80% GPU costs compared to static provisioning. Runtime reliability is enhanced by prioritized black-box health checks, reducing false liveness probe failures to zero, and optimizing multimodal request handling by switching to Torchvision-based image processors and configuring OMP_NUM_THREADS, which boosted RPS per server by over 3x.

Key takeaway

For MLOps Engineers managing large-scale LLM deployments, understanding Databricks' strategies for reliability and cost efficiency is crucial. You should consider implementing a "model unit" abstraction to quantify request costs, enabling more precise load balancing and autoscaling. Prioritize health checks in your inference stack to prevent silent hangs and cascading failures. Additionally, optimize multimodal request handling by leveraging efficient image processors like Torchvision and correctly configuring OMP_NUM_THREADS to avoid CPU throttling and achieve significant performance gains.

Key insights

Reliable LLM inference at scale requires sophisticated capacity management, cost-based routing, and robust runtime failure detection.

Principles

• LLM request cost is highly variable and hard to estimate a priori.
• High-bandwidth GPU systems are less reliable than classical CPU systems.
• Overprovisioning for LLM inference is cost-prohibitive due to compute constraints.

Method

Model request cost with "model units" for capacity management. Implement cost-based load balancing and autoscaling. Ensure runtime reliability via prioritized health checks and optimized multimodal processing.

In practice

• Implement "model units" for granular capacity estimation.
• Use cost-based load balancing for LLM workloads.
• Prioritize health checks to prevent cascading failures.

Topics

• LLM Inference
• Scalability
• Databricks Platform
• Capacity Management
• Load Balancing
• Autoscaling
• Multimodal AI

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• How Superhuman and Databricks built a 200K QPS inference platform together — A collaborative engineering approach can achieve high-scale, low-latency AI inference by optimizing infrastructure and runtime.
• Elevate Your LLM Inference: Autoscaling with Ray, ROCm 7.0.0, and SkyPilot — Ray Serve, ROCm 7.0.0, and SkyPilot enable scalable, cost-efficient LLM inference across diverse cloud and cluster environments.
• Optimizing LLM latency — MLC, CTranslate2, and vLLM offer significant latency improvements for LLM inference, with varying trade-offs.
• Inference Scaling (Test-Time Compute): Why Reasoning Models Raise Your Compute Bill — Inference scaling dynamically allocates compute for reasoning, but requires strategic management to avoid excessive costs and performance issues.
• NVIDIA Dynamo on AKS - Autoscaling LLM Inference — Effective LLM inference autoscaling requires specialized tools like NVIDIA Dynamo on AKS to manage variable loads and GPU constraints.
• Everyone Is Scaling AI. Nobody Is Solving Inference. That’s the Real Problem — AI inference costs are escalating due to a fundamental mismatch between model architecture and hardware.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.