Yelp Achieves Zero-Downtime Upgrade of Over 1,000 Cassandra Nodes

2026-04-24 · Source: InfoQ · Field: Technology & Digital — Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, quick

Summary

Yelp successfully executed a zero-downtime upgrade of over 1,000 Apache Cassandra nodes, demonstrating a scalable blueprint for modernizing stateful systems. The company's Database Reliability Engineering team detailed how careful planning, phased execution, and automation facilitated this seamless modernization of critical data infrastructure. This effort tackled the complex challenge of upgrading a live, highly available database without interrupting production workloads, which are essential for many of Yelp's core services. The team employed a rolling upgrade strategy, incrementally updating nodes while maintaining cluster availability and data consistency. This approach minimized cascading failures by upgrading nodes in controlled batches and allowing the cluster to rebalance between steps, aligning with best practices for Cassandra upgrades.

Key takeaway

For CTOs and VPs of Engineering managing critical data platforms, Yelp's successful Cassandra upgrade proves that zero-downtime modernization of large-scale stateful infrastructure is feasible. You should prioritize robust planning, phased execution, and significant investment in automation and observability to ensure continuous availability and seamless change, making traditional maintenance windows obsolete for your systems.

Key insights

Zero-downtime upgrades for large-scale stateful systems are achievable through meticulous planning and phased execution.

Principles

• Prioritize strict compatibility and incremental change.
• Automate orchestration and continuous health checks.
• Understand data replication and node recovery dynamics.

Method

Execute rolling upgrades in controlled batches, allowing rebalancing and repair between steps. Monitor and validate each phase in real time with automation and continuous health checks.

In practice

• Implement rolling upgrades for Cassandra clusters.
• Invest in automation for upgrade orchestration.
• Utilize continuous health checks during upgrades.

Topics

• Apache Cassandra
• Zero-Downtime Upgrades
• Distributed Databases
• Rolling Upgrade Strategy
• Database Reliability Engineering

Best for: CTO, VP of Engineering/Data, Data Engineer, DevOps Engineer, Consultant

Related on AIssential

• How to achieve zero-downtime updates in large-scale AI agent deployments — AI agent "zero-downtime" requires functional uptime, not just system uptime, due to their non-deterministic, stateful nature.
• How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability — Migrating petabyte-scale data ingestion requires staged transitions, continuous validation, and robust rollback mechanisms to ensure zero downtime and consistency.
• Zero-Downtime Patching in Lakebase Part 1: Prewarming — Lakebase is addressing planned database maintenance, which causes more workload disruption than unplanned failures, by aiming for "zero-downtime patching." This initial post details "prewarming," a te…
• How Figma Upgraded Data Pipeline from Multi-Day Latency to Real-Time — Incremental synchronization via Change Data Capture significantly improves data freshness and reduces costs for large-scale data pipelines.
• Uber’s Hive Federation Decentralizes 16K Datasets and 10+ PB for Zero-Downtime Analytics at Scale — Decentralizing monolithic data warehouses via pointer-based migration enhances scalability, security, and operational autonomy.
• Absurd In Production — Absurd demonstrates that durable execution systems can be built effectively and minimally on Postgres alone.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.