JSON, Parquet, Iceberg, Blob: Where the Data Actually Sits

2026-07-28 · Source: Data Engineering on Medium · Field: Technology & Digital — Cloud Computing & IT Infrastructure · Depth: Intermediate, quick

Summary

This article introduces a "four-floor" model to clarify the architecture of modern data platforms, explaining how terms like S3, Parquet, Iceberg, and data lake fit together without being competing answers. It details the foundational "ground floor" as object storage, exemplified by Amazon S3, Azure Data Lake Storage, and Google Cloud Storage. This layer serves as an almost infinite, cost-effective repository for raw, unprocessed data, accepting chunks of bytes with a name and ensuring their safe storage. Its simplicity is highlighted as a key design principle, distinguishing it from higher-level data organization. The piece aims to illustrate this stack by tracing customer orders from an online store to their representation as actual bytes on disk.

Key takeaway

For data engineers designing or managing data lakes, understanding the distinct layers—object storage, file formats, table formats, and processing engines—is crucial. Clearly differentiating these "floors" prevents architectural confusion and ensures optimal data management strategies. You should prioritize robust, cost-effective object storage like S3 for raw data, then build higher-level structures. This clarity will streamline troubleshooting and improve system scalability.

Key insights

Modern data platforms are structured in four distinct layers, from object storage to processing engines.

Principles

• Object storage provides cheap, simple, infinite data retention.
• Each data platform layer builds upon the one below it.
• Confusion arises from mixing data platform layers.

In practice

• Utilize S3, ADLS, GCS for raw data storage.
• Separate raw data storage from file/table formats.
• Understand data location from bytes to engine.

Topics

• Object Storage
• Data Lake Architecture
• Cloud Storage
• Amazon S3
• Azure Data Lake Storage
• Google Cloud Storage

Best for: Data Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Twenty years in, Amazon S3 finds itself at the center of AWS’ push beyond storage — Amazon S3 has evolved into a foundational data substrate for AI and analytics, moving beyond basic object storage.
• The Invisible Layer: Why the success of your data pipeline starts in the File System — Understanding file systems is crucial for optimizing data pipelines and diagnosing performance issues in data engineering.
• Data Lake vs Data Warehouse vs Lakehouse vs Data Mesh: What’s the Difference? — Data architectures evolve from structured reporting to flexible raw storage, then to unified platforms and decentralized organizational models.
• JSON and Beyond: A Developer’s Guide to Avro, Parquet, ORC, Protobuf, and More — Different data serialization formats offer distinct trade-offs, making format selection dependent on specific use case requirements.
• Main Data Warehouse Components Explained (2025) — Data warehouses organize vast datasets into actionable insights using a layered architecture and dedicated processing.
• Unfreezing The Data Lake: The Future-Proof File Format — F3 is a future-proof file format using flexible layouts and embedded WebAssembly for efficient, extensible data handling.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.