Rebalancing: Everyone Stops So the Partitions Can Move

Partition assignment must be mutually exclusive: two members processing the same partition means duplicate work and incoherent offsets. To guarantee exclusivity while the mapping changes, the classic protocol takes the simplest safe route — everyone releases everything, then everyone gets a new assignment. No member holds a partition during the transition, so no overlap is possible.

The cost is that the entire group stops, including the members whose assignments will not change at all. A twelve-member group where one pod restarts pauses all twelve. Depending on protocol and configuration this is typically seconds; with large groups, many partitions, or slow revocation handlers it can be tens of seconds, and every second is backlog accumulating on every partition.

This is a real distributed-agreement cost, and it is exactly what a work queue does not have. A queue worker dying affects only its own in-flight messages; a group member dying affects everyone. That difference is the price of the assignment guarantee, and it is the strongest argument for a queue when ordering and replay are not required.

Rebalancing produces duplicates by construction, and understanding exactly where they come from removes a great deal of confusion. A member has processed records up to offset 500 on partition 2, but has only committed offset 450 — commits are periodic, not per record. The rebalance revokes partition 2, and a new owner starts from 450. Records 451 to 500 are processed a second time.

The size of that window is the distance between "processed" and "committed", which is directly under your control through commit frequency — and is precisely the trade-off in Commit Before or After: There Is No Third Option. Commit more often and the window shrinks at the cost of throughput; commit less often and the window grows.

The revocation callback is where you shrink it deliberately: on revoke, stop fetching, finish or abandon in-flight work, and commit the true last-completed offset before releasing the partition. A group that does this well has a duplicate window of nearly zero on planned restarts. A group that ignores the callback reprocesses up to a full commit interval on every deploy, forever, and blames the broker.

Note the harder case the callback cannot fix: a member that was declared dead because it was *slow* never runs its revocation handler in time. It is still processing records for a partition the coordinator has already given away, and it may commit an offset for a partition it no longer owns. Some implementations reject that commit — which is a fencing check, and it is the only thing standing between you and two members writing incoherent positions.

Here is the loop, and it is worth memorising because it explains a large fraction of "our stream processor fell over and we do not know why" incidents.

A member is considered alive only if it keeps talking to the coordinator. In protocols where liveness is tied to how often the member asks for records, a member that spends too long processing a batch stops asking, is declared dead, and its partitions are given away. It then rejoins, triggering another rebalance. Meanwhile the pause has increased everyone’s backlog, so the next batch each member fetches is larger, so processing takes longer, so another member exceeds the interval. Now two members are cycling. Then three.

The signature is unmistakable once you have seen it: rebalances every few seconds, lag climbing steadily, CPU high, and completed-work throughput near zero. The group is spending all its time agreeing on who owns what and none of its time processing. It does not recover on its own, because the mechanism that would let it catch up is the mechanism being interrupted.

The fixes are all about decoupling liveness from processing duration: heartbeat on a separate thread so liveness does not depend on the handler; cap the number of records per fetch so a batch cannot take too long; raise the maximum processing interval to something safely above your p99 batch time; and hand very long work to a side queue rather than blocking the consumer loop. This is the same conclusion as Visibility Timeout: The Message Is Hidden, Not Yours reached by a different route — never let "how long the work takes" be the signal for "is this process alive".

Two mitigations are worth understanding structurally, because they attack different halves of the problem.

Cooperative / incremental rebalancing removes the stop-the-world property. Rather than everyone revoking everything, the assignment is computed as a diff and only the partitions that actually move are revoked. A member whose assignment is unchanged never stops. This turns a group-wide pause into a per-partition handover and is a large improvement for big groups — at the cost of a more complex protocol, sometimes two rounds, and behaviour that is harder to reason about during an incident.

Static membership removes many rebalances entirely. Each member carries a stable identity across restarts, and the coordinator holds its assignment for a grace period rather than reassigning immediately. A pod that restarts within the window gets its own partitions back and no rebalance occurs. This is close to free for rolling deploys and is the highest-value setting most teams have not turned on. The trade is that a genuinely dead member’s partitions are unowned for the whole grace period, so you have chosen slower failure detection in exchange for fewer spurious rebalances — the same dial as every failure detector in No Heartbeat Does Not Mean Dead.

Neither removes the duplicate window. That belongs to the commit point, and no rebalancing protocol can fix a decision made in Commit Before or After: There Is No Third Option.