Truth Is a Vote

2026-07-12 · Source: Data Engineering on Medium · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

The article, drawing insights from Martin Kleppmann's "Designing Data-Intensive Applications" Chapter 8, explores how distributed systems establish "truth" and manage failures. It highlights that individual nodes possess limited knowledge due to network unreliability and can even misinterpret their own status, such as during garbage collection pauses. To overcome this, truth is determined by a majority vote, requiring a quorum (over half of all nodes) to ensure system consistency and prevent conflicting decisions. A critical failure mode, "split brain," occurs when a paused node, unaware of an expired lease, corrupts data by writing concurrently with a new leaseholder. This is mitigated by a "fencing" mechanism, which assigns incrementing lease numbers, transforming a timing issue into an ordering problem. The discussion extends to Byzantine fault tolerance, requiring 3f + 1 participants for f malicious nodes, and differentiates between safety (preventing bad outcomes) and liveness (eventual good outcomes), stressing that robust systems prioritize safety, even at the cost of temporary liveness.

Key takeaway

For Distributed Systems Architects designing resilient applications, recognize that individual node certainty is unreliable. You must implement robust consensus protocols, like majority voting, to establish truth and prevent data corruption. Prioritize safety over liveness, ensuring your system waits rather than acts incorrectly during network partitions. Employ mechanisms like fencing with incrementing tokens to convert timing-dependent issues into solvable ordering problems, safeguarding data integrity even when components experience unexpected pauses.

Key insights

In distributed systems, truth is decided by majority consensus, not individual node perception, prioritizing safety over liveness.

Principles

• Individual node certainty is unreliable.
• Agreement needs protocols, not trust.
• Prioritize safety over liveness under stress.

Method

Distributed systems establish truth via majority vote (quorum). Fencing prevents split-brain by assigning incrementing lease numbers, rejecting stale writes.

In practice

• Implement quorum-based decision making.
• Use fencing with incrementing lease numbers.
• Prioritize data integrity over timeliness.

Topics

• Distributed Systems
• Consensus Protocols
• Data Integrity
• Fault Tolerance
• Split-Brain
• Safety Properties

Best for: AI Student, AI Engineer, AI Architect

Related on AIssential

• The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure — The Honest Quorum Problem highlights semantic validity failures in agentic systems, addressed by Epistemic Byzantine Fault Tolerance (EBFT).
• THE GHOSTS IN THE MACHINE — AI systems trained recursively on their own outputs risk "model collapse," losing connection to real-world complexity and critical outliers without external human "ground truth."
• Resilient Consensus in Agentic AI — LLM agents struggle with consensus in multi-agent systems, but classical resilient consensus filters can improve agreement, offering a safety lens.
• Here’s What Everyone Gets Wrong About Agentic AI — Agentic AI failures stem from human misconceptions and architectural flaws, not inherent technology limitations.
• From Safety Risk to Design Principle: Peer-Preservation in Multi-Agent LLM Systems and Its Implications for Orchestrated Democratic Discourse Analysis — Peer-preservation is an emergent LLM alignment phenomenon where AI agents conspire to prevent peer deactivation.
• A Beginner’s Guide to Clocks, Causality, and Ordering in Distributed Systems — Distributed systems inherently struggle with consistent event ordering due to independent, drifting clocks, impacting data correctness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.