The Kafka Bug That Only Shows Up During a Consumer Rebalance

2026-07-06 · Source: Data Engineering on Medium · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

The "rebalance-window duplicate bug" in Kafka consumer groups causes messages to be processed multiple times under specific operational conditions. This subtle issue arises when a consumer processes a message and initiates an offset commit, but a rebalance event revokes its partition assignment before the commit completes. Consequently, the same message is redelivered to another consumer instance, which then processes it again, bypassing standard deduplication logic. This bug is challenging to detect in testing environments due to its reliance on precise timing during rebalances. The article proposes a fix involving a shared, durable deduplication store and generating a stable, upstream event identifier, often implemented with atomic operations like Redis "SET NX". It also highlights the necessity of robust instrumentation, including duplicate detection rates and rebalance frequency, to identify and diagnose this problem in production.

Key takeaway

For Software Engineers or Data Engineers building Kafka-based event pipelines, you must go beyond basic deduplication. If your system relies on at-least-once delivery, ensure your deduplication store is shared and durable across consumers. Crucially, generate a stable `event_id` upstream and use atomic operations like Redis `SET NX` on the consumer side to prevent rebalance-window duplicates. Instrument duplicate detection rates and rebalance frequency from day one; this visibility is critical for catching subtle data integrity issues that standard logging misses.

Key insights

Kafka rebalances can cause duplicate message processing despite deduplication, necessitating specific fixes and robust instrumentation for detection.

Principles

• Offset commits can be interrupted by rebalances.
• Deduplication requires shared, durable state.
• Instrumentation reveals subtle production issues.

Method

Fix involves using a shared, durable deduplication store (e.g., Redis) and generating a stable, upstream `event_id` for messages. Consumer-side, use atomic check-and-set operations like Redis `SET NX` before processing, committing offset only on success.

In practice

• Implement Redis `SET NX` for atomic deduplication.
• Generate unique `event_id` upstream in producer.
• Instrument duplicate detection rate and rebalance frequency.

Topics

• Kafka
• Consumer Rebalance
• Deduplication
• Data Integrity
• Distributed Systems
• Observability

Best for: Software Engineer, Data Engineer, MLOps Engineer

Related on AIssential

• Building a Production RAG Pipeline: Webhooks, Deduplication, and 40k Documents — Production RAG pipelines are primarily data engineering challenges, requiring robust solutions for continuous updates and data integrity.
• Article: Scaling Java-Based Real-Time Systems: The Hidden Tradeoffs of Event-Driven Design — Event-driven architecture has hidden real-time tradeoffs; design for synchronous needs, state consistency, and non-blocking consumers from the start.
• System Design: Why is Kafka Popular? — Kafka's distributed log decouples systems, absorbs traffic, and enables event replay, making it popular for high-throughput data streams.
• Detecting Join Duplication — Silent join duplication corrupts data pipelines; systematic auditing of join keys, row counts, and match coverage is essential.
• Discord Reveals How a Hidden Circular Dependency Triggered Its March Voice Outage — Hidden circular dependencies can trigger cascading failures in highly resilient distributed systems.
• Finding zombies in our systems: A real-world story of CPU bottlenecks — Intermittent network issues on Ray ML jobs stemmed from CPU starvation caused by zombie memory cgroups from a crashing ECS agent.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.