How OpenAI Scaled to 800 Million Users With Postgres

2025-12-15 · Source: ByteByteGo Newsletter · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure, Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

OpenAI successfully scaled a single-primary PostgreSQL database to handle millions of queries per second for 800 million ChatGPT users, achieving five-nines availability and low double-digit millisecond latency. This contradicts conventional wisdom that typically recommends sharding for such scale. Over the past year, their database load increased by over 10X, necessitating systematic optimization across the stack. Their strategy focused on minimizing primary database load by offloading read traffic and migrating write-heavy workloads to sharded systems like Azure Cosmos DB, alongside application-level write optimizations. They also implemented query and connection optimizations, including avoiding complex multi-table joins, reviewing ORM-generated SQL, and deploying PgBouncer for connection pooling. Furthermore, OpenAI prevented cascading failures through cache locking, rate limiting, and workload isolation, while addressing PostgreSQL's MVCC and schema change constraints by migrating write-heavy tasks and enforcing strict schema modification rules. High availability is maintained with hot standbys and multiple read replicas, with future plans for cascading replication.

Key takeaway

For MLOps Engineers or Data Engineers managing large-scale, read-heavy applications, consider thoroughly optimizing your PostgreSQL deployment before resorting to complex sharding. Your team should focus on offloading read traffic, migrating write-heavy components, and implementing robust connection pooling and query optimization. This approach can significantly extend PostgreSQL's scalability, potentially delaying or avoiding the operational overhead of sharding, as demonstrated by OpenAI's success with ChatGPT.

Key insights

PostgreSQL can scale to massive read-heavy workloads with rigorous optimization, challenging conventional sharding wisdom.

Principles

• Prioritize read replica scaling for read-heavy workloads.
• Minimize primary database load through offloading and migration.
• Optimize queries and connections at multiple layers.

Method

OpenAI scaled PostgreSQL by minimizing primary load, optimizing queries and connections with PgBouncer, preventing cascading failures via cache locking and rate limiting, and addressing MVCC and schema change constraints.

In practice

• Implement PgBouncer for connection pooling to reduce latency.
• Use cache locking to prevent database overload from cache misses.
• Isolate workloads on dedicated instances to mitigate "noisy neighbor" effects.

Topics

• PostgreSQL Scaling
• Database Optimization
• Read Replicas
• Connection Pooling
• High Availability

Best for: Data Engineer, MLOps Engineer, Software Engineer

Related on AIssential

• OpenAI Scales Single Primary Postgresql to Millions of Queries per Second for ChatGPT — A single-primary PostgreSQL instance can scale to millions of QPS with strategic application and database optimizations.
• Nextdoor’s Database Evolution: A Scaling Ladder — Nextdoor's database architecture evolved from a single PostgreSQL instance to a sophisticated distributed system to support millions of users, addressing scaling challenges through a disciplined progr…
• 10 Data Ingestion Optimization Techniques for Faster and Scalable Processing — Parallelizing data ingestion workloads is critical for achieving faster, more scalable processing and reducing latency by over 50%.
• A Billion Dollar Idea Gone Wrong — Even well-intentioned technical solutions can be misused, requiring robust infrastructure like autoscaling databases.
• How I Engineered a 1.7M Row Product Safety Database to Serve Pages in 66ms on a Single Free VM — Optimizing large-scale database performance and cost on minimal infrastructure requires strategic data management and query pattern alignment.
• EP196: Cloud Load Balancer Cheat Sheet — Cloud-native architectures enhance scalability and efficiency through specialized components like load balancers, CQRS, Docker, and serverless functions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ByteByteGo Newsletter.