Nextdoor’s Database Evolution: A Scaling Ladder

2025-12-15 · Source: ByteByteGo Newsletter · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

Nextdoor's database architecture evolved from a single PostgreSQL instance to a sophisticated distributed system to support millions of users, addressing scaling challenges through a disciplined progression of engineering solutions. Initially, connection limits were resolved with PgBouncer, followed by a Primary-Replica setup and Time-Based Dynamic Routing to manage read traffic and replication lag. A high-performance caching layer using Valkey, optimized with MessagePack and Zstd compression, was implemented for millisecond response times. To ensure cache consistency and prevent race conditions, Nextdoor developed a versioning engine with PostgreSQL Triggers and atomic Compare and set operations via Lua scripts in Valkey. Finally, Change Data Capture (CDC) with Debezium and a background Reconciler provides a self-healing mechanism for eventual consistency, with sharding introduced as the ultimate solution for extreme write volume scaling.

Key takeaway

Nextdoor's database evolution showcases a disciplined scaling ladder, progressing from a single PostgreSQL instance to a sophisticated distributed system. This involved implementing PgBouncer for connection pooling, a Primary-Replica architecture with Time-Based Dynamic Routing to manage read traffic and replication lag, and a Valkey-based versioned cache using MessagePack/Zstd and atomic Lua scripts. A CDC-driven reconciliation system with Debezium ensures eventual consistency, demonstrating that complexity must be earned to maintain data integrity and user trust while scaling to billions of rows, ultimately leading to sharding for write scalability.

Topics

• PostgreSQL Scaling
• PgBouncer
• Primary-Replica Architecture
• Caching Strategies
• Data Consistency

Best for: Software Engineer, Data Engineer

Related on AIssential

• How OpenAI Scaled to 800 Million Users With Postgres — PostgreSQL can scale to massive read-heavy workloads with rigorous optimization, challenging conventional sharding wisdom.
• Enabling Evolutionary Database Development: database branching with Lakebase, continued — Copy-on-write database branching makes per-instance database development feasible, removing key constraints for evolutionary design.
• Eventual Consistency: The Key Trade-Off Behind Modern Databases — Eventual consistency trades immediate data synchronization for enhanced performance, scalability, and availability in distributed systems.
• Zero-Downtime Patching in Lakebase Part 1: Prewarming — Lakebase is addressing planned database maintenance, which causes more workload disruption than unplanned failures, by aiming for "zero-downtime patching." This initial post details "prewarming," a te…
• How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability — Migrating petabyte-scale data ingestion requires staged transitions, continuous validation, and robust rollback mechanisms to ensure zero downtime and consistency.
• How Agoda Scaled Its Feature Store 50X with ScyllaDB — Agoda scaled its feature store 50x using ScyllaDB for high-performance, low-latency data management.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ByteByteGo Newsletter.