QCon London 2026: Introducing Tansu.io — Rethinking Kafka for Lean Operations

2026-03-21 · Source: InfoQ · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure, Data Science & Analytics · Depth: Advanced, short

Summary

At QCon London 2026, Peter Morgan introduced Tansu.io, an open-source, Apache Kafka-compatible messaging broker designed with stateless brokers that delegate durability entirely to external storage, contrasting with Kafka's replicated stateful design. Tansu brokers are "cattle" that scale to zero and back up rapidly, running in minimal memory, and support pluggable storage backends like S3, SQLite, and Postgres, with optimized Postgres integration using the COPY FROM protocol. It features broker-side schema validation for Avro, JSON, and Protobuf, guaranteeing data consistency and enabling direct writing of validated data to open table formats such as Apache Iceberg, Delta Lake, or Parquet. Tansu can also function as a Kafka proxy, handling high throughput with low latency, though it currently lacks features like throttling and ACLs. The project, written in asynchronous Rust and Apache-licensed, is actively seeking contributors.

Key takeaway

Tansu.io introduces a stateless, Kafka-compatible messaging broker that offloads durability to external storage, enabling 10ms scale-to-zero and a 20MB memory footprint. This design, featuring pluggable storage (S3, Postgres with COPY FROM for high-throughput) and broker-side schema validation, directly writes Avro/JSON/Protobuf data to Iceberg/Delta Lake/Parquet. It significantly simplifies ML data pipelines by ensuring data consistency and reducing operational overhead from ingestion to analytics.

Topics

• Messaging Broker
• Apache Kafka Protocol
• Stateless Architecture
• Pluggable Storage
• Schema Validation

Best for: CTO, VP of Engineering/Data, AI Architect, Software Engineer, Data Engineer, MLOps Engineer

Related on AIssential

• Apache Kafka for Data Engineers — Beginner Guide — Apache Kafka provides a scalable, fault-tolerant, and decoupled architecture for real-time data streaming.
• Introducing Storage Buckets — Hugging Face Storage Buckets provide affordable, deduplicated, and CDN-enabled storage for AI workflows.
• From Natural Language to PromQL: A Catalog-Driven Framework with Dynamic Temporal Resolution for Cloud-Native Observability — A catalog-driven framework translates natural language into PromQL, simplifying cloud-native observability with dynamic temporal resolution.
• How Reddit Migrated Petabyte-Scale Kafka from EC2 to Kubernetes — Reddit's Engineering Team successfully migrated its entire Apache Kafka fleet, comprising over 500 brokers and more than a petabyte of live data, from Amazon EC2 virtual machines to Kubernetes with ze…
• Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance — Moving Kafka schema IDs to message headers decouples schema metadata from data payloads, simplifying governance and evolution.
• Message Brokers 101: Storage, Replication, and Delivery Guarantees — Message brokers enable asynchronous communication and temporal decoupling in distributed systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.