Designing Data-intensive Applications with Martin Kleppmann

2026-04-08 · Source: The Pragmatic Engineer · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure, Artificial Intelligence & Machine Learning · Depth: Advanced, extended

Summary

Martin Kleppmann, author of "Designing Data-Intensive Applications," discusses the second edition of his seminal book, detailing its updates and the motivations behind them. The first edition, heavily influenced by his work on Kafka at LinkedIn, focused on foundational concepts for large backend systems. The second edition incorporates cloud-native architectures, acknowledging the shift from local disks to object stores and managed services, and addresses the evolving definition of scalability to include scaling down for cost efficiency. Kleppmann also highlights the removal of outdated topics like MapReduce and the addition of new ones, such as vector indexes and data frames, to support AI applications. He emphasizes the engineer's responsibility in considering the societal impact of technology and the importance of formal verification, especially with the rise of AI-generated code.

Key takeaway

For AI Architects and Software Engineers designing modern data systems, understanding the shift to cloud-native primitives and the nuances of scalability (including scaling down) is crucial. Your role increasingly involves articulating trade-offs between cost, performance, and resilience, and considering geopolitical risks like multi-cloud dependence. Embrace formal verification for critical components, especially with AI-generated code, to ensure reliability and security beyond traditional testing. This proactive approach will help you build robust, future-proof systems and make informed business decisions.

Key insights

The second edition of "Designing Data-Intensive Applications" updates core principles for cloud-native and AI-era systems.

Principles

• Reliability prioritizes fault tolerance.
• Scalability means proportional capacity to load.
• Engineers must consider technology's societal impact.

Method

Formal verification, including model checking and mathematical proofs, ensures system correctness, especially for high-stakes algorithms and AI-generated code, by reasoning about infinite state spaces beyond typical testing.

In practice

• Utilize cloud object stores for elastic storage.
• Employ serverless for cost-efficient low-load services.
• Explore formal verification for critical system components.

Topics

• Designing Data-Intensive Applications
• Cloud-Native Architectures
• Distributed Systems
• Formal Verification
• Local-First Software

Best for: Software Engineer, Data Engineer, AI Architect

Related on AIssential

• How Modern Data Pipelines Drive AI Application Development in Dubai — Modern data pipelines are foundational for scalable, real-time AI applications, ensuring data quality and efficient processing.
• Designing Data-Intensive Applications: The Cloud & Doing the Right Thing — The updated "Designing Data-Intensive Applications" balances technical system design with critical ethical considerations for engineers.
• BugBash'26 Morning of Day 1 — Peter Alvaro's 20-year journey in distributed systems reveals no silver bullet, but continuous progress through diverse, evolving approaches.
• Designing Data and AI Systems That Hold Up in Production — Full-stack thinking and system-level design are crucial for building reliable, scalable, and trustworthy AI and data systems.
• Eventual Consistency: The Key Trade-Off Behind Modern Databases — Eventual consistency trades immediate data synchronization for enhanced performance, scalability, and availability in distributed systems.
• From the clouds: The imperatives and designs of today’s IT and data economics — Cloud-first strategies created data fragmentation, necessitating a shift to outcome-driven, economically measured enterprise architecture for AI and regulation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Pragmatic Engineer.