JSON and Beyond: A Developer’s Guide to Avro, Parquet, ORC, Protobuf, and More

2026-07-13 · Source: Data Engineering on Medium · Field: Technology & Digital — Software Development & Engineering, Data Science & Analytics, Artificial Intelligence & Machine Learning · Depth: Intermediate, long

Summary

A developer's guide explores various data serialization formats beyond JSON, detailing their characteristics, use cases, and trade-offs. It covers JSON for REST APIs and configuration, CSV for tabular data, and XML for enterprise systems. More specialized formats include Apache Avro for streaming and schema evolution in Kafka and Flink, Apache Parquet and ORC for columnar storage and efficient analytics in data lakes and Spark, and Protocol Buffers (Protobuf) for compact, high-performance service communication in gRPC. MessagePack offers binary JSON-like efficiency, while YAML is primarily for human-readable configuration. The article emphasizes that each format solves specific problems related to storage, transfer, compression, validation, and querying.

Key takeaway

For data engineers or distributed systems architects evaluating data storage and transfer solutions, understanding the specific strengths of each format is crucial. Your choice impacts performance, storage efficiency, and schema evolution capabilities. Prioritize Avro for evolving streaming data, Parquet for analytical workloads in data lakes, and Protobuf for high-performance inter-service communication, reserving JSON for human-readable APIs and configurations.

Key insights

Different data serialization formats offer distinct trade-offs, making format selection dependent on specific use case requirements.

Principles

• Serialization converts in-memory objects for external use.
• Schema evolution is crucial for distributed systems.
• Columnar storage optimizes analytical queries.

In practice

• Use JSON for public REST APIs.
• Implement Avro for Kafka event streams.
• Choose Parquet for data lake analytics.

Topics

• Data Serialization
• Apache Avro
• Apache Parquet
• Protocol Buffers
• Data Engineering
• Schema Evolution

Best for: Software Engineer, Data Engineer, Machine Learning Engineer

Related on AIssential

• YAML vs XML vs JSON: History, Trade-offs, and Where Each Wins in the Age of Agentic AI — The choice between XML, JSON, and YAML depends on the job, with JSON Schema emerging as the universal validation layer.
• Building Your Modern Data Analytics Stack with Python, Parquet, and DuckDB — Combining Parquet for columnar storage and DuckDB for direct querying offers a fast, efficient data analytics stack.
• JSON, Parquet, Iceberg, Blob: Where the Data Actually Sits — Modern data platforms are structured in four distinct layers, from object storage to processing engines.
• Evolutionary Data Through Schemaboi: Achieving Forward, Backwards, and Sideways Compatibility — A durable, self-describing schema embedded in data files enables robust, independent evolution for local-first applications.
• QCon London 2026: Introducing Tansu.io — Rethinking Kafka for Lean Operations — At QCon London 2026, Peter Morgan introduced Tansu.io, an open-source, Apache Kafka-compatible messaging broker designed with stateless brokers that delegate durability entirely to external storage, c…
• AWS Glue, S3, and Athena — a beginner’s real-world workflow — AWS S3, Glue, and Athena form a cohesive serverless pipeline for storing, understanding, and querying cloud data efficiently.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.