Designing Data-Intensive Applications: The Cloud & Doing the Right Thing

2026-01-06 · Source: The Pragmatic Engineer · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure, Cybersecurity & Data Privacy · Depth: Intermediate, extended

Summary

Martin Kleppmann and Chris Riccomini have released a second edition of "Designing Data-Intensive Applications," updating the 2016 original to reflect the widespread adoption of cloud computing. The refreshed volume, published by O'Reilly Media, Inc. in 2026, includes new content on cloud versus self-hosting tradeoffs, cloud-native system architecture, and the evolving role of operations in the cloud era. It also dedicates a significant portion to the ethical responsibilities of software engineers, addressing issues like predictive analytics, algorithmic bias and discrimination, accountability, feedback loops, and mass surveillance. The book aims to provide foundational principles for building resilient systems, incorporating topics such as AI-driven systems, local-first software, formal methods, and regulatory contexts like GDPR.

Key takeaway

For software engineers designing data-intensive applications, you should critically evaluate the tradeoffs between cloud services and self-hosting based on your team's expertise and workload variability. Beyond technical choices, actively engage with the ethical implications of your systems, particularly concerning predictive analytics and data collection, to ensure accountability and prevent unintended societal harms. Your decisions shape not just technology, but also human experiences.

Key insights

The updated "Designing Data-Intensive Applications" balances technical system design with critical ethical considerations for engineers.

Principles

• Core competencies should be in-house; non-core tasks can be outsourced.
• Cloud services excel with variable loads; self-hosting suits predictable, experienced teams.
• Engineers bear responsibility for software's societal consequences.

Method

When deciding between cloud and self-hosting, evaluate organizational skills, workload predictability, and the system's core competency alignment. For ethical design, engage in iterative reflection and accountability, considering system-wide impacts.

In practice

• Evaluate cloud services for systems with highly variable loads.
• Consider self-hosting for predictable workloads if operational expertise exists.
• Analyze potential feedback loops and biases in predictive analytics.

Topics

• Cloud Computing Tradeoffs
• Data-Intensive Systems
• Cloud-Native Architecture
• Software Engineering Ethics
• Algorithmic Bias

Best for: Software Engineer, AI Architect, MLOps Engineer

Related on AIssential

• Designing Data-intensive Applications with Martin Kleppmann — The second edition of "Designing Data-Intensive Applications" updates core principles for cloud-native and AI-era systems.
• System Design for Data Engineers: A Complete Guide (with Real Walkthroughs) — Data engineers require a unified system design approach integrating both data and infrastructure scaling concepts.
• 8 Strategies a Data Engineering Services Provider Uses to Optimize Data — Effective data engineering transforms raw data into actionable insights through structured architecture, quality, and integration.
• Data Residency Is Not a Legal Problem. It Is an Infrastructure Design Problem — Data residency is an infrastructure design problem, not just a legal one, requiring full data lifecycle awareness.
• Reflections on Anthropic’s Self-Service Data Analytics Playbook — Data infrastructure, not AI, is the bottleneck for self-service analytics, requiring robust data foundations and governance.
• EP212: Data Warehouse vs Data Lake vs Data Mesh — Effective system design balances data storage, API architecture, server communication, and quality assurance for optimal performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Pragmatic Engineer.