Evolutionary Data Through Schemaboi: Achieving Forward, Backwards, and Sideways Compatibility

2026-07-14 · Source: InfoQ · Field: Technology & Digital — Software Development & Engineering, Emerging Technologies & Innovation · Depth: Intermediate, quick

Summary

Seph Gentle, in a closing talk at a local-first software conference, proposed a robust data format to enable interoperable applications and user data ownership, moving beyond the limitations of traditional local or cloud software. He introduced "schemaboi", an experimental Rust implementation, which embeds a self-describing, statically typed schema directly into file headers. This approach allows any tool to interpret data without external schema files, contrasting with Protocol Buffers or JSON. "schemaboi" supports primitives and complex structures, preserving unknown fields and filling missing ones with defaults. Gentle defined "sideways compatibility," allowing independent field additions via globally unique identifiers, alongside standard backward and forward compatibility. He outlined safety guidelines for schema evolution, noting that adding optional fields is safe, while renaming or type changes are high-risk. Benchmarks indicate "schemaboi" is significantly smaller and faster than JSON, though the project remains experimental.

Key takeaway

For software engineers designing local-first applications, consider adopting a data format that embeds its schema directly within files. This approach, exemplified by "schemaboi", ensures long-term data durability and "sideways compatibility," allowing your team to evolve schemas independently without central coordination. Prioritize globally unique field identifiers and incremental schema changes, like adding optional fields, to maintain data integrity across diverse application versions and prevent costly migrations.

Key insights

A durable, self-describing schema embedded in data files enables robust, independent evolution for local-first applications.

Principles

• Data formats must be self-describing and statically typed.
• Preserve unknown data fields for future compatibility.
• Use globally unique field identifiers for sideways compatibility.

Method

The "schemaboi" system embeds the schema in the initial bytes of every file, merging it with the application's internal schema upon opening to ensure data interpretation and compatibility.

In practice

• Embed schemas directly into data files for self-description.
• Implement globally unique field IDs to prevent schema collisions.
• Prioritize adding optional fields over type changes for safety.

Topics

• Local-first Software
• Data Compatibility
• Schema Evolution
• Schemaboi
• Protocol Buffers
• Cap'n Proto
• Data Formats

Code references

• josephg/schemaboi

Best for: CTO, VP of Engineering/Data, AI Architect, Software Engineer, Research Scientist

Related on AIssential

• Orchestrating Black-Box Schema Converters: An Empirical Study of Automated, Quality-Ranked Conversion Across Heterogeneous Schema Languages — Orchestrating black-box schema converters via a graph model enables automated, quality-ranked data model consistency across heterogeneous languages.
• Why the Best Data Engineers Never Memorize Tools. — Mastering data engineering patterns, not just tools, ensures adaptability and robust pipeline design.
• Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework — Pinterest's framework automates schema evolution in CDC pipelines using a phased convergence model for safe, auditable updates.
• JSON and Beyond: A Developer’s Guide to Avro, Parquet, ORC, Protobuf, and More — Different data serialization formats offer distinct trade-offs, making format selection dependent on specific use case requirements.
• Eventual Consistency: The Key Trade-Off Behind Modern Databases — Eventual consistency trades immediate data synchronization for enhanced performance, scalability, and availability in distributed systems.
• From Legacy to AI-Ready: How MongoDB AMP Accelerates Modernization — MongoDB's unified platform and document model accelerate AI-driven application modernization by integrating operational data, vectors, and search.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.