The Consumer Group That Spent Its Life Rebalancing
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Consumer 4 | Assigned 4 partitions, began processing a batch, was revoked 20 seconds later, reassigned 4 different partitions. | The group is scaling; carry on. | ✕ wrong |
| Group coordinator (broker) | A member missing its heartbeat deadline, then rejoining. Every time, correctly. | A member left; trigger a rebalance to redistribute its partitions. | ✕ wrong |
| Deployment system | All 12 pods Ready, no restarts, no OOM kills. | The consumers are healthy. | ✓ right |
| On-call engineer | Lag climbing, CPU idle, no errors in consumer logs. | The brokers are throttling us. | ✕ wrong |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
The consumers are healthy, the brokers are healthy, and CPU is idle. Commit before reading on: name the mechanism by which a group with no failing member rebalances sixty times a minute — and why idle CPU is a *clue*, not a contradiction.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.