The question this answers
How do several consumer processes share a topic without duplicating work — and how does another team read the same data without affecting mine?
Within a consumer group, each partition is assigned to exactly one member at a time, so each record is delivered to exactly one member of that group per assignment epoch. Across groups, assignment and position are entirely independent: every group observes every record. Neither guarantee survives a Rebalancing: Everyone Stops So the Partitions Can Move boundary without duplicates.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A member knows which partitions it has been assigned and its position in each. It does not know how many other members exist, which partitions they hold, or whether its own assignment has already been revoked by a rebalance it has not yet learned about. That last gap is the source of the duplicate processing at rebalance boundaries: a member can be working on a partition it no longer owns and cannot tell.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The two axes
The group id is the whole mechanism. Two consumers with the same group id split the partitions between them; two consumers with different group ids each get all of them. That is the entire API surface of the fan-out decision, and it is a string in a config file — which is why a copy-pasted config that reuses another service’s group id produces one of the most confusing outages in this domain: two unrelated services silently stealing half of each other’s records.
Inside a group you have a work queue with a bonus: work is distributed, no record is processed twice in the steady state, and adding a member increases parallelism — up to the partition count. Unlike a real work queue you also keep per-key ordering, because the unit of assignment is a whole partition.
Across groups you have pub/sub with a bonus: each group reads independently at its own pace, and a group that is hours behind does not slow anyone else. Unlike real pub/sub there is one stored copy rather than one per subscriber, and a new group can start in the past.
The ceiling: members beyond partition count do nothing
The unit of assignment is a partition, and a partition goes to exactly one member. So a group on a 6-partition topic can usefully have at most 6 members. The seventh joins, participates in the rebalance, receives no assignment, and idles — consuming a pod, appearing healthy, and contributing nothing.
This makes autoscaling on a log fundamentally different from autoscaling a queue-backed worker pool. With a queue, capacity is continuous: add a worker, get more throughput. With a group, capacity is quantised at the partition boundary, and scaling past it is a no-op that costs money and adds a rebalance. Scaling policies written for queues therefore behave badly here — they observe lag, scale up, observe no improvement, and scale up again.
When the group is at the ceiling and still behind, there are only three moves. Add partitions (which breaks per-key ordering — see A Topic Is Not One Log: Ordering Lives Inside a Partition). Make the consumer faster, usually by parallelising *inside* the member across the partitions it owns. Or split the work into a second topic with a different key. None of these is a config change, which is why partition count deserves thought at design time rather than at incident time.
| Members | Assignment | Effect of adding one more |
|---|---|---|
| 1protocol | All 6 partitions to one member | Halves the load, doubles throughput |
| 4typical | Uneven: two members get 2, two get 1 | Improves balance and throughput |
| 6protocol | One partition each — the ceiling | No throughput change; a rebalance for nothing |
| 10protocol | 6 working, 4 idle | Another idle member, another rebalance |
| 6, one partition hotassumption | One member saturated, five idle | Nothing — the bottleneck is one partition, not member count |
Group state is durable, and that is the interesting part
A group is not just a runtime arrangement. Its committed offsets are durable state stored outside the members, which produces properties that surprise people arriving from queue-land.
Stop every member of a group and the group still exists, holding its positions. Restart tomorrow and it resumes where it left off — with a day of backlog, but no gap. Delete the group and its positions are gone; recreate it and it starts from wherever its configured reset policy says, which is typically the earliest or latest available offset. That reset policy is a correctness setting: latest on a group recreated after an outage silently skips everything that arrived while it was gone, and reports zero lag immediately afterwards, which looks exactly like a healthy recovery.
The durable position is also the mechanism behind the log’s best operational trick: you can reset a group’s offsets deliberately. Reprocess the last hour after fixing a bug, rebuild a derived view from the beginning of retention, or move a group forward past a poison record. These are ordinary operations rather than emergencies, and they exist only because position is external to the data.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG MEMBER billing orders 0 9,910,442 9,910,451 9 m1-7f3a billing orders 1 4,420,118 4,420,120 2 m1-7f3a billing orders 2 8,003,900 8,004,011 111 m2-c19b billing orders 3 2,110,004 2,110,004 0 m2-c19b search orders 0 8,100,300 9,910,451 1,810,151 s1-44de search orders 1 - 4,420,120 - s1-44de <-- NO COMMITTED OFFSET A partition with no committed offset has never been processed by this group, or its commit was lost. On restart it will use the reset policy, not resume.
One member, several partitions: the concurrency question inside the process
A member assigned four partitions receives records from all four in a single fetch loop. The naive consumer processes them one at a time in whatever interleaving the fetch produced, which is correct but caps throughput at one record at a time even though four independent ordered streams are available.
The correct optimisation is to parallelise by partition: one worker per assigned partition, each processing its partition strictly in order. Ordering is preserved because ordering was only ever per partition, and throughput multiplies by the number of partitions the member owns. This is the single highest-value consumer optimisation available and it is frequently missed.
What must not happen is parallelising *within* a partition — handing several records from the same partition to a thread pool. That discards the only ordering guarantee the system offers, and it makes offset commits incoherent: you cannot commit offset 105 while 103 is still in flight without claiming to have processed something you have not. Which is precisely the subject of Commit Before or After: There Is No Third Option, and the reason per-partition workers with per-partition offset tracking is the shape that works.
1// CORRECT: N ordered streams, N workers, order preserved per key2onAssign(partitions):3 for p in partitions:4 spawn worker(p):5 for record in stream(p): // strictly sequential within p6 handle(record)7 trackOffset(p, record.offset)8 9// WRONG: destroys the only ordering guarantee you have, and makes the10// commit point meaningless -- committing 105 while 103 is in flight11// claims progress that has not happened.12for record in fetch():13 pool.submit(() => handle(record))14 15// The revoke path matters as much as the assign path: on revocation you16// must stop the worker and commit what it finished, or the next owner17// reprocesses from the last committed offset.18onRevoke(partitions):19 for p in partitions:20 stopWorker(p); commit(p, lastCompletedOffset(p))Key points
- Same group id splits the partitions; different group ids each read everything. That string is the entire fan-out decision.
- A group cannot usefully have more members than the topic has partitions; extra members idle and add rebalances.
- Committed offsets are durable group state that outlives every member, which is what makes deliberate replay an ordinary operation.
- A recreated group uses its reset policy —
latestsilently skips everything missed and then reports zero lag. - Inside a member, parallelise across assigned partitions and never within one; within-partition concurrency destroys ordering and makes commits incoherent.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Each member joins the group by contacting the group coordinator with its group id and subscription.
- • The coordinator selects a leader among the members, which computes a partition-to-member assignment using the configured strategy.
- • The assignment is distributed; each member begins fetching from its assigned partitions at its last committed offset, or at the reset policy if none exists.
- • Members periodically send heartbeats to signal liveness and periodically commit their positions.
- • A member joining, leaving or failing to heartbeat triggers a rebalance and a new assignment epoch.
- • Two unrelated services share a group id by accident and each processes half the records.
- • A group is scaled past partition count and the extra members idle while lag continues to climb.
- • A group is deleted and recreated with a
latestreset policy, silently skipping the backlog. - • A member holds an assignment it has already lost because it has not yet processed the rebalance.
- • A member parallelises within a partition and commits an offset ahead of records still being processed.
- • Split-stream from a shared group id: two services each report processing roughly half the expected records. Broker metrics show perfect delivery and zero loss, and the discrepancy is only visible by comparing the two services’ output counts.
- • Autoscaling futility: lag climbs, the scaler adds members, throughput does not move, the scaler adds more. The operator sees a group with 30 members, 6 assignments, rising cost and unchanged lag.
- • Silent gap after group recreation: lag drops to zero immediately after an incident and stays there. The recovery looks perfect; a window of records was never processed and nothing reports it.
- • Uneven assignment with a hot partition: five members idle at 5% CPU while one is pinned at 100%. Group-level lag looks moderate because it is summed across partitions.
- • Offset ahead of processing: after a crash, records are missing from the output. The consumer had committed positions from a thread pool that had not finished the work, so restart resumed past them.
- • Group membership and partition assignment require agreement among members via a coordinator — genuine distributed coordination, with a genuine cost paid on every membership change.
- • That coordination is what a plain work queue does not have, and it is the source of the pause described in Rebalancing: Everyone Stops So the Partitions Can Move.
- • Offset commits are coordination between a member and durable group state; their timing determines whether failure produces duplicates or gaps.
- • A member failure costs only its partitions, which are reassigned; the rest of the group keeps working.
- • Committed offsets survive the loss of every member, so a total group outage costs time, not data — provided retention holds.
- • Exclusive assignment is guaranteed only within an assignment epoch; across a rebalance boundary two members may briefly process the same partition.
- • Detect: per-partition lag per group, plus a check that active member count matches expected and does not exceed partition count.
- • Contain: never delete a group to "reset" it during an incident — set offsets explicitly instead, which preserves the ability to choose where to resume.
- • Recover: restart members and let the assignment settle; verify every partition has an owner and a committed offset before declaring recovery.
- • Reconcile: if a reset policy was used, identify the skipped offset range and reprocess it explicitly rather than assuming zero lag means completeness.
- • Verify: every partition assigned, every partition with a committed offset advancing, and no member holding zero partitions.
- • Lag per partition per group, and the count of partitions with no committed offset.
- • Member count versus partition count, alerting when members exceed partitions.
- • Assignment distribution across members — evenness of partition count, and evenness of throughput, which differ when a partition is hot.
- • Rebalance frequency per group; a healthy group rebalances on deploys and almost never otherwise.
- • Commit rate per member; a member with flat lag and no commits is stalled, not idle.
- • Parallel consumption with per-key ordering preserved — the combination a work queue cannot offer.
- • Multiple independent teams reading one stream at their own pace from one stored copy.
- • Deliberate replay and reprocessing, which durable external positions make routine.
- • Workloads needing continuous scaling granularity; the partition-count ceiling makes capacity quantised and autoscaling ineffective past it.
- • Highly variable per-record cost, where whole-partition assignment cannot balance load even when partition counts are even.
- • Small deployments where the coordination machinery (coordinator, heartbeats, rebalances) is pure overhead over a simple queue.
- • A work queue with competing consumers, when ordering is unnecessary and continuous scaling matters more than replay.
- • Manual partition assignment, skipping group coordination entirely: a member is pinned to specific partitions, so there are no rebalances and no automatic failover — you own both.
- • One consumer per partition as separate deployments, making assignment a deployment concern rather than a runtime negotiation. Rigid, predictable, and sometimes exactly right.
- • Parallelism inside a single member across its assigned partitions, when the ceiling has been reached and adding partitions would break ordering.
Queue semantics inside a group, pub/sub semantics across groups
What people believe, and what is true
Adding consumers always increases throughput.
Only up to partition count. Beyond it, members idle and each addition costs a rebalance.
A consumer group is like a queue subscription.
It is durable, externally stored position over immutable data. You can move it backwards, which no queue subscription allows.
Deleting and recreating a group resets it safely.
It applies the reset policy. With latest you skip everything that arrived meanwhile, and the resulting zero lag looks like success.
Even partition counts mean even load.
Load follows records and their cost, not partition count. One hot partition saturates one member while the rest idle.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Consumers sharing a group id split the topic’s partitions between them, one owner per partition. Consumers with different group ids each get the whole topic. More members than partitions does nothing.
Practical
Give every service its own group id and treat it as an identity, not a config detail. Size the group to partition count, not to lag. Set the reset policy deliberately and never delete a group to reset it. Parallelise across assigned partitions inside each member, never within a partition. Alert on per-partition lag, member-versus-partition count, and rebalance frequency.
Advanced
A consumer group is a distributed assignment problem solved by consensus on membership plus a deterministic assignment function. That framing predicts its properties: the exclusivity guarantee is scoped to an assignment epoch, because that is what the agreement covers, so anything that spans epochs — an in-flight record, an uncommitted offset — is outside the guarantee and is where duplicates live. It also explains the ceiling: the assignment function maps partitions to members, and a function cannot give a partition two owners without breaking the exclusivity that makes the group a queue in the first place.
Apply it
- 🔧 Run two services with the same group id on purpose and observe each receiving a subset. Then find the metric that would have made it obvious.
- 🔧 Implement per-partition workers inside one member with correct per-partition offset tracking, including the revoke path.
- ⚡ Autoscaling has grown a group to 30 members on a 6-partition topic and lag is unchanged. Explain, then give the three real options.
- ⚡ After an incident, lag drops to zero instantly and a downstream report is missing four hours of data. Diagnose from the group state alone.
- 💬 Six partitions, ten consumers in one group. What happens?
- 💬 Two teams want the same stream and neither should slow the other. How do you set it up, and what could go wrong?
- 💬 What happens if you delete a consumer group and recreate it?