Coinbase Postmortem Reveals How a Localized AWS Failure Triggered a Multi-Hour Trading Outage

2026-06-16 · Source: InfoQ · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, short

Summary

Coinbase published a detailed postmortem for its May 7, 2026, outage, revealing a localized AWS cooling failure in the US-East-1 region escalated into a multi-hour disruption, halting nearly all trading. The initial AWS thermal event took EC2 instances and EBS volumes offline. Coinbase's investigation found that architectural dependencies, specifically its Raft-based cluster matching engine operating within a single AWS Cluster Placement Group, lacked automated failover. This design, optimized for ultra-low latency, lost quorum when three of its five nodes went down. Additionally, Kafka workloads for event streaming became stranded, creating backlogs and delaying recovery. The combination of these issues transformed a localized cloud problem into a platform-wide outage, highlighting how architectural assumptions, not just underlying cloud infrastructure, determine real-world availability. Coinbase plans automated cross-zone recovery and improved messaging infrastructure.

Key takeaway

For MLOps Engineers designing high-performance, low-latency systems, you must critically evaluate architectural trade-offs between speed and resilience. Your reliance on single-zone cluster placement for performance can create critical single points of failure, as seen with Coinbase's Raft-based matching engine. Prioritize automated cross-zone failover and robust messaging infrastructure to accelerate recovery from inevitable cloud infrastructure failures, ensuring your systems maintain availability even under localized disruptions.

Key insights

Outages often stem from unexpected interactions between individually manageable failures and architectural assumptions.

Principles

• Performance optimization can compromise resilience.
• Cloud deployment doesn't guarantee resilience.
• Hidden dependencies amplify failure impact.

Method

Recovery involved emergency code changes, manual cluster reconstruction, and rebalancing Kafka partitions to restore quorum and data flow.

In practice

• Implement automated cross-zone recovery.
• Improve quorum restoration procedures.
• Expand disaster recovery testing.

Topics

• AWS Outage
• Cloud Resilience
• Distributed Systems
• Raft Consensus
• Kafka
• Disaster Recovery

Best for: CTO, VP of Engineering/Data, Product Manager, Software Engineer, DevOps Engineer, MLOps Engineer

Related on AIssential

• AWS suffers cloud service disruption after fire at UAE data facility — Physical incidents can severely disrupt cloud infrastructure, even within a single Availability Zone.
• The Pulse: Did Anthropic’s new model just boost rival Codex’s market share? — Vendor policies and infrastructure reliability are critical considerations in AI adoption and operational resilience.
• Discord Reveals How a Hidden Circular Dependency Triggered Its March Voice Outage — Hidden circular dependencies can trigger cascading failures in highly resilient distributed systems.
• How the lakebase architecture stays resilient to cloud failures — Lakebase achieves high availability for agentic workloads through stateless compute, zone-redundant storage, and a refactored control plane.
• How I Built a Commit-Aware Multi-Cloud Reliability Engine (And What Two Patents Taught Me) — Code commits provide early signals for predicting cloud failures and enabling proactive reliability measures.
• AWS Distinguished Eng: Learnings From 3000 Incidents And How Engineering Is Changing | Marc Brooker — Hands-on experience and deep understanding of system failures are paramount for effective distributed systems engineering.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.