Stream Processing
7 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
A topic is split into partitions, each an independent append-only log with its own offsets. Order is total inside a partition and undefined across them. Choosing a partition key is therefore choosing which ordering guarantee you get — and simultaneously choosing where your hot spots will be.
Q · My topic promises ordering. Ordering of what, exactly?
A consumer group is a set of processes that share the partitions of a topic between them, each partition assigned to exactly one member. Inside a group that behaves like a work queue; across groups, every group reads everything independently. One stored copy, two models, and a parallelism ceiling set by partition count rather than by how many machines you own.
Q · How do several consumer processes share a topic without duplicating work — and how does another team read the same data without affecting mine?
When a member joins or leaves a consumer group, partitions are redistributed. In the classic protocol every member gives up everything and waits for a new assignment, so the whole group stops processing. Worse, a member that is merely slow is declared dead, which triggers a rebalance, which makes everyone slower — a loop that produces the rebalance storm.
Q · A consumer restarted and my whole group stopped for 20 seconds. Why does one member leaving affect everyone?
A consumer must record how far it has got. Commit the offset before processing and a crash loses records. Commit after and a crash reprocesses them. There is no arrangement that does neither, because the offset store and the output store are different systems — and that, precisely, is where exactly-once actually lives or dies.
Q · When should my consumer commit its position, and why can I not have both no loss and no duplicates?
Every record carries two timestamps, whether or not you record both: the moment the thing occurred, and the moment your system observed it. They are never equal and sometimes differ by days. Every windowed aggregation is computed against one of them, and choosing without noticing is how a dashboard becomes confidently wrong.
Q · My hourly counts are wrong after an outage. Which clock was I counting by?
An event occurred at 10:00 and arrives at 10:05. The 10:00 window closed four minutes ago and something downstream has already acted on its result. You have exactly three options — drop it, restate the result, or route it aside — and each one hands a different problem to somebody else.
Q · The window already emitted its answer and a record for it just showed up. Now what?
A watermark is the processor saying "I believe I have now seen everything that happened up to time T". It is an estimate, not a fact — nothing can tell it that a source is finished. It is nonetheless the mechanism that lets an event-time window ever close, and understanding it as a heuristic rather than a guarantee is the difference between a pipeline you can debug and one you cannot.
Q · If I can never know an event-time window is complete, what makes it fire?