How Superhuman and Databricks built a 200K QPS inference platform together

2026-05-08 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, medium

Summary

Superhuman, a productivity platform serving over 40 million daily users, partnered with Databricks to modernize its AI model serving stack for real-time communication assistance. This custom large language model handles peak traffic exceeding 200,000 queries per second (QPS) with P99 latency under 1 second and 99.99% reliability. The migration from a DIY vLLM-based stack to Databricks model serving addressed compounding pain points like manual performance tuning for new model iterations and growing operational burden. The collaboration focused on optimizing load balancing with a power-of-two choices algorithm, accelerating container startup via lazy-loading image formats, and implementing runtime optimizations like FP8 quantization and multiprocessing to boost per-pod throughput by 60% on H100 GPUs, from 750 QPS to 1,200 QPS.

Key takeaway

For MLOps Engineers managing high-QPS, low-latency AI inference, consider a platform partnership that co-invests in engineering to meet strict SLAs. You should prioritize infrastructure optimizations like intelligent load balancing and image acceleration, alongside runtime improvements such as FP8 quantization and multiprocessing, to achieve significant throughput gains and maintain reliability for demanding real-time applications.

Key insights

A collaborative engineering approach can achieve high-scale, low-latency AI inference by optimizing infrastructure and runtime.

Principles

• Asymmetric autoscaling prevents latency spikes.
• Per-channel quantization improves FP8 quality.
• CPU bottlenecks can limit fast GPU models.

Method

Modernized LLM serving involved custom load balancing (power-of-two choices), image acceleration for faster container startup, and runtime optimizations like FP8 quantization and multiprocessing to eliminate CPU bottlenecks.

In practice

• Implement power-of-two load balancing for high QPS.
• Use lazy-loading container images to reduce cold starts.
• Explore FP8 quantization for throughput gains.

Topics

• Databricks Model Serving
• High-QPS Inference
• FP8 Quantization
• Image Acceleration
• Power-of-Two Choices Load Balancing

Best for: MLOps Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Reliable LLM Inference at Scale — Reliable LLM inference at scale requires sophisticated capacity management, cost-based routing, and robust runtime failure detection.
• The AI Industry Spent Billions Chasing Faster Chips. Inference Startup Kog Says They Were Solving the Wrong Problem. — Software optimization can enable standard GPUs to match specialized AI hardware performance for real-time inference.
• Introducing AI Runtime: Scalable, Serverless NVIDIA GPUs on Databricks for Training and Finetuning — AI Runtime simplifies distributed GPU training by offering serverless, on-demand access and optimized tools within Databricks.
• Token-Operations-Oriented Inference Optimization Techniques for Large Models — Token-operations-oriented inference optimization uses a four-layer architecture to make large model services operable.
• Benchmarking inference at scale: coding agents — Production-scale coding agent inference requires full-stack optimization to manage concurrent long-context loads and prioritize Time To First Token (TTFT).
• Inside the Together AI kernels team — GPU kernel optimization, exemplified by FlashAttention, offers substantial performance gains for AI workloads.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.