How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second

2026-07-25 · Source: InfoQ · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, short

Summary

Zalando's engineering team developed an in-process, client-side load balancer to manage approximately 1 million requests per second for its Product Read API. This solution addressed challenges with their shared cluster edge load balancer, Skipper, particularly for high fan-out internal traffic, resulting in more predictable latency, reduced infrastructure costs, and improved failure visibility. The team reimplemented Skipper's xxHash64 algorithm with 100 virtual nodes to ensure consistent routing during migration. Key implementation steps included replacing polling with a watch-based Kubernetes informer, enabling fast, reversible deploys, and gradually ramping the load balancer from 1% to 100%. This initiative reduced Skipper's fleet from over 50 pods to 8, cutting daily deployment costs from \$450 to \$110. They also introduced N-ring fade-in over 30 seconds on a ^2.5 curve to prevent scale-up latency spikes. Zalando notes this custom build was for an extreme edge case, generally recommending mature proxies like Skipper or Envoy.

Key takeaway

For DevOps Engineers or Architects considering custom load balancing solutions for high-throughput APIs, you should generally prioritize mature, off-the-shelf proxies like Skipper or Envoy. Zalando's success with an in-process client-side load balancer was for an extreme edge case of 1 million requests per second with high fan-out. Building your own introduces significant maintenance overhead and complexity that existing solutions already handle, hardened by thousands of users. Evaluate your specific traffic patterns and scale before investing in custom routing logic.

Key insights

Zalando built an in-process client-side load balancer for extreme high-throughput, improving latency and cost by moving routing logic closer to the application.

Principles

• Consistent hashing minimizes cache churn during endpoint changes.
• In-process routing can improve latency for high fan-out internal traffic.
• Gradual rollout with toggles manages risk for critical infrastructure changes.

Method

Reimplement existing load balancing algorithm (e.g., xxHash64, 100 virtual nodes) in-process. Replace polling with watch-based Kubernetes informers for configuration updates. Implement N-ring fade-in over 30 seconds on a ^2.5 curve for new pod warming.

In practice

• Use client-side load balancing for extreme fan-out internal APIs.
• Employ consistent hashing to avoid cache splits during migrations.
• Monitor inter-AZ costs; AZ-aware routing can fragment caches.

Topics

• Client-side Load Balancing
• Consistent Hashing
• Kubernetes Informer
• High-throughput API
• Infrastructure Cost Optimization
• Skipper Load Balancer

Code references

• zalando/skipper

Best for: CTO, AI Architect, MLOps Engineer, Software Engineer, DevOps Engineer, VP of Engineering/Data

Related on AIssential

• Linkerd 2.20 Delivers Smarter Traffic Management and Dramatic Efficiency Gains — Linkerd 2.20 enhances service mesh efficiency and resilience through intelligent rate-limit-aware traffic management and significant memory optimization.
• Offloading CPU Intensive to threads — Offload CPU-intensive tasks to a thread pool via a queue to unblock the main process.
• QCon London 2026: Introducing Tansu.io — Rethinking Kafka for Lean Operations — At QCon London 2026, Peter Morgan introduced Tansu.io, an open-source, Apache Kafka-compatible messaging broker designed with stateless brokers that delegate durability entirely to external storage, c…
• Why WebSockets don’t scale easily — and how AWS changes the game — WebSocket scalability is fundamentally limited by linear resource consumption, necessitating complex distributed systems or managed services.
• The Unexpected Challenges of Building a RAG Ingestion Service for 1 Million Documents — Building RAG ingestion at scale demands robust error handling, cost control, and meticulous data quality validation.
• Why your Kubernetes scheduler can't handle AI workloads — The default Kubernetes scheduler fails AI workloads due to partial-scheduling and lack of fabric topology awareness, necessitating specialized schedulers.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.