The question this answers
Everyone calls Kafka a message queue. What is actually different, and why does it change my design?
Records appended to a log are assigned a monotonically increasing offset and are immutable. Any consumer may read from any retained offset, any number of times, without affecting any other consumer. Reads are non-destructive; a record is removed only when it falls outside the configured retention, never because it was read.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A consumer knows its own position and the highest offset it has been told exists. It does not know what other consumers have read, does not affect them by reading, and — crucially — cannot tell from the log itself whether it has *processed* what it has read. Processing state lives in the consumer, not in the data, which is the inversion the whole model rests on and the source of Commit Before or After: There Is No Third Option as a separate problem.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two data structures, not two products
A work queue is a mutable collection with a claim protocol. Messages are added, claimed (hidden), and destroyed on acknowledgement. The consumption state lives *in the message*: available, in-flight, acked. Because the message is destroyed, there is no history, no second reader, and no replay. These are not missing features; they are excluded by the structure.
A log is an immutable ordered sequence with external cursors. Records are appended and never modified. Consumption state lives *in the consumer* as an offset. Because the data is not touched by reading, any number of readers can hold independent positions, a reader can move its position backwards, and a reader that does not exist yet can start anywhere in the retained history.
Everything else follows mechanically from where consumption state lives. Replay is possible because the data is still there. Fan-out is cheap because there is one copy and N cursors. Per-key ordering is possible because the sequence is stable and a key can be routed to one sequence. And per-message retry is awkward because a cursor is a single position that can only move forward — it cannot step around a record. That last one is the price, and it is not small.
| Work queue | Append-only log | |
|---|---|---|
| Consumption state lives inprotocol | The message (available/in-flight/acked) | The consumer (an offset) |
| Effect of readingprotocol | Claims and hides; ack destroys | Nothing — data is untouched |
| Data removed whenprotocol | Acknowledged | Retention (time or size) expires |
| Second independent readerprotocol | Impossible — they compete | Free — another cursor |
| Replayprotocol | Impossible | Move the offset backwards |
| Orderingprotocol | Lost with concurrency | Total within a partition, always |
| Skip one bad recordprotocol | Natural — DLQ that message | Requires copying it elsewhere and committing past it |
| Storage grows withprotocol | Backlog only | Throughput × retention, regardless of consumption |
| Parallelism ceiling per reader groupprotocol | Number of workers | Number of partitions |
Retention: the clock that is always running
In a queue, storage is a function of your backlog: process everything and the queue is empty. In a log, storage is a function of throughput times retention, and it is entirely independent of whether anyone has consumed anything. A topic at 50 MB/s with seven days of retention holds about 30 TB whether it has zero consumers or fifty.
This flips the operational question. The queue question is "are we falling behind?". The log question is "is the slowest consumer’s position still inside the retention window?" — because when a consumer’s offset falls off the end of retention, the records it had not read are gone, and what happens next is product-specific and always bad: either the consumer resets to the earliest available offset and silently skips a gap, or it errors out. Both are silent data loss from the business’s point of view, and neither shows up as an error in the log itself.
Retention is therefore a correctness setting disguised as a cost setting, and it is routinely tuned by someone looking at a storage bill. The number you need is the maximum time a consumer might be down and still be expected to catch up: a long weekend, a holiday freeze, the time to notice and fix a broken consumer. Seven days is the common default because it covers a weekend plus a working day of response.
Compaction is the other retention mode and answers a different question. A compacted log retains the *latest* record per key indefinitely and discards superseded ones, so it stops being a history of events and becomes a durable snapshot of current state per key — the thing you rebuild a cache or a Materialized Views: A Read Model That Lags from. History and snapshot are different products of the same structure; choosing the wrong one loses either your audit trail or your ability to bootstrap.
partition 3 earliest retained offset : 8,140,220 (records before this are deleted) latest offset : 9,002,551 group=billing position : 9,002,540 lag 11 margin 862,320 records group=analytics position : 8,145,900 lag 856,651 margin 5,680 records <-- 40 min group=archive position : 8,140,224 lag 862,327 margin 4 <-- ABOUT TO LOSE DATA "Lag" says how far behind you are. "Margin" says how long until data you have not read is deleted. Only the second one is an emergency, and almost nobody graphs it.
Immutability changes what the log is for
Because records are never modified and are retained independently of consumption, the log stops being purely a transport and becomes a shared source of history. Several capabilities that are impossible with a queue become routine, and they are the real reason log-based systems reshaped event-driven architecture.
You can add a consumer in month six and have it build its entire view from the beginning of retention, with no involvement from the producer. You can fix a bug in a stream processor, reset the offset, and recompute a derived view from scratch — replacing a data migration with a replay. You can run a new version of a consumer alongside the old one, reading the same records, and compare outputs before switching. And you can debug by reading exactly what the consumer saw, rather than reconstructing it from logs.
That is also the connection to Architecture’s event-sourcing and cqrs, and the boundary worth being precise about: event sourcing is a *modelling* decision — the event stream is the system of record and current state is derived. Using a log as transport is not event sourcing; you can have a perfectly ordinary database as the source of truth and a log carrying notifications about it. Conflating the two leads teams to treat a seven-day-retention transport topic as an audit trail, and to discover the difference during an audit.
What the log makes worse
It is worth being blunt about the costs, because the log is frequently adopted for prestige rather than for a requirement it satisfies.
Per-record failure handling is genuinely awkward. A cursor cannot skip. Every log-based pipeline eventually builds retry topics or a DLQ topic to get failures out of the sequence — reinventing a queue at the edge. Parallelism is capped by partition count, so scaling a consumer group past that number does nothing, and changing the count reshuffles key-to-partition mapping and breaks the per-key ordering you were relying on. Storage costs are unconditional and paid whether or not anyone reads. Operational weight is higher: partitions, replication, consumer-group coordination, rebalancing, and lag monitoring per partition rather than one depth number.
And the model demands more of the consumer. Because processing state lives in the consumer rather than in the data, *you* now own the decision about when to commit a position — which is the trade-off in Commit Before or After: There Is No Third Option and, ultimately, the concrete root of why Exactly-Once Is a Scope, Not a Guarantee is hard. A queue answered that question for you with an ack; a log hands it back.
Key points
- A log is an immutable ordered sequence read by advancing a cursor; a queue is a mutable set with a claim-and-destroy protocol. Every other difference follows.
- Reads are non-destructive, so replay, multiple independent readers, and starting a new consumer in the past are all natural.
- Retention is time- or size-based and independent of consumption: storage equals throughput times retention whether or not anyone reads.
- A consumer whose position falls out of retention loses data silently — track margin to the earliest retained offset, not just lag.
- The costs are real: no per-record skip, parallelism capped by partition count, unconditional storage, and the offset-commit decision handed to you.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A producer appends a record; the log assigns it the next offset in that partition and makes it durable.
- • The record is immutable from that moment; nothing rewrites or removes it except retention.
- • A consumer reads sequentially from its current position, which is state the consumer holds (usually persisted back to the broker or to its own store).
- • After processing, the consumer advances and commits its position. The record itself is untouched by any of this.
- • Other consumers hold their own positions over the same records and are entirely unaffected.
- • Retention deletes records older than the configured age or beyond the configured size, from the tail, regardless of who has read them.
- • A slow or stopped consumer falls behind retention and the records it had not read are deleted.
- • A consumer commits a position it has not actually finished processing, silently skipping records.
- • A record that cannot be processed blocks the cursor, halting everything behind it in that partition.
- • Retention is lowered to save storage, and a consumer that was safely behind is now beyond the window.
- • A topic is treated as an audit trail when it has a seven-day retention, so the history assumed to exist does not.
- • Silent gap after retention loss: a consumer that was down over a long weekend resets to the earliest available offset and resumes normally. Lag reads zero, processing is healthy, and a block of records was never seen. Nothing in the pipeline reports it.
- • Storage bill shock: a topic nobody consumes still costs full throughput times retention. The operator sees cluster storage growth uncorrelated with any consumer activity, and the cause is a topic added months ago.
- • Partition stall: one unprocessable record halts a partition. Per-partition lag climbs; aggregate lag looks normal; users whose keys live in that partition see stale data and nobody is paged.
- • False audit trail: a compliance question arrives about an event from four months ago and the topic retained seven days. The team discovers the difference between a transport log and a system of record during the audit.
- • Parallelism ceiling: a consumer group is scaled from 6 to 24 instances and throughput does not change, because the topic has 6 partitions. Eighteen instances sit idle holding no assignment.
- • Ordering broken by a partition-count change: partitions are increased to raise throughput, and keys begin hashing to different partitions. Per-key ordering silently breaks at the moment of the change, and events for a key appear interleaved across two partitions.
- • Between readers: none. Independent cursors over immutable data is coordination avoidance in its purest form, and it is why fan-out is nearly free.
- • Inside the log: appends within a partition must be totally ordered, which requires a leader per partition and replication before acknowledgement — real coordination, paid on every write.
- • Between members of a consumer group: partition assignment must be agreed, which is Rebalancing: Everyone Stops So the Partitions Can Move and is the coordination cost a queue does not have.
- • Committed records are immutable and survive consumer failure entirely; a consumer crash costs position, not data.
- • A consumer that restarts resumes from its last committed position, which is why the commit point determines whether the failure produces duplicates or gaps.
- • Data outside the retention window is gone unconditionally — no consumer state, backup or replay can recover it from the log.
- • Detect: alert on margin to the earliest retained offset per consumer group and per partition, in addition to lag.
- • Contain: raise retention immediately when a consumer is badly behind. It is usually a live setting and it buys time that nothing else can.
- • Recover: restart or scale the consumer group; a log tolerates a consumer being down far better than a queue tolerates it, provided retention holds.
- • Reconcile: after any suspected retention loss, rebuild the affected derived view from the source of truth rather than from the log — the log no longer has the data.
- • Verify: every group inside the retention window with comfortable margin, and per-partition lag flat rather than merely aggregate lag being acceptable.
- • Per-partition lag, and per-group margin to the earliest retained offset — the second is the one that predicts data loss.
- • Log-end offset growth rate per partition, which reveals hot partitions before they become a latency problem.
- • Storage per topic against retention setting, so the cost of an unconsumed topic is attributable.
- • Consumer position commit rate; a group whose lag is flat but whose commit rate is zero is stalled, not caught up.
- • Number of active consumers versus partition count per group, to catch the idle-instance ceiling.
- • Multiple independent consumers of the same stream, especially when the set grows over time.
- • Any requirement to replay: rebuilding a derived view, reprocessing after a bug fix, or bootstrapping a new service from history.
- • Per-key ordering requirements, which partitioning gives you and a work queue cannot.
- • High sustained throughput, where sequential append and sequential read are exactly what disks and page caches are good at.
- • A single consumer doing independent tasks with no ordering requirement. You have paid for partitions, rebalancing and retention to get nothing a queue would not have given you.
- • Workloads dominated by per-message retry, where the cursor model actively fights you.
- • Small teams without capacity to operate partitioned, replicated storage and to monitor lag per partition.
- • Low-volume streams with long-tail latency requirements, where the polling and batching behaviour adds latency for no throughput benefit.
- • A work queue, when there is one consumer, no ordering requirement and no replay use case. Simpler in every dimension and the right answer more often than fashion suggests.
- • A database table with an auto-incrementing id read by cursor — a log with worse throughput and dramatically better queryability, joins and retention control. Excellent at modest volume.
- • Change data capture from the database’s own write-ahead log, which gives you a log without a separate publish step and without the dual-write problem, at the cost of coupling consumers to your schema.
- • Object storage plus a manifest for high-volume, high-latency-tolerance streams — much cheaper per byte, entirely unsuitable for low latency.
The log is not a queue
What people believe, and what is true
Kafka is a message queue.
It is a partitioned append-only log. It can be used to build queue-like behaviour with consumer groups, but reads are non-destructive and position is per partition, which is a different structure with different algebra.
Once consumed, the message is removed.
Nothing is removed on consumption. Records leave only when retention expires, which is why a stopped consumer loses nothing until the window closes, and everything after it does.
The log is our audit trail.
Only if retention is set to the audit period and compaction is off. A seven-day transport topic is not an audit trail, and the difference is discovered at the worst possible moment.
Lag is the metric to watch.
Lag tells you how far behind you are. Margin to the earliest retained offset tells you how long before unread data is deleted. The second is the one that becomes an incident.
Adding partitions scales an existing topic safely.
It scales throughput and breaks key-to-partition mapping. Existing keys move, so per-key ordering is not preserved across the change.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A log stores records in order and never deletes them on read. Consumers keep a position and can move it backwards. Old records disappear on a retention timer, not when someone reads them.
Practical
Pick retention for the longest plausible consumer outage, not for the storage bill. Monitor margin to the earliest retained offset per group, not only lag. Expect to build retry topics for per-record failures. Size partitions for the parallelism you will need, because changing the count later breaks key ordering.
Advanced
The log is the same structure a database uses for its write-ahead log, exposed as a public interface. That is the deep reason it composes so well with derived state: a replicated state machine is exactly "apply an ordered log of deterministic operations", so any consumer that applies records in order and is deterministic will converge to the same view as any other consumer that did the same — with no coordination between them. The log becomes the single point of ordering, and every downstream view is a fold over it. That is why Total Order Broadcast Is Consensus Wearing a Different Hat, leader-based replication and log-based streaming are the same idea wearing three different job titles.
Apply it
- 🔧 Take a topic and compute storage as throughput times retention. Compare that to the backlog you would have held in a queue, and articulate what the extra bytes bought.
- 🔧 Reset a consumer group to the earliest offset, rebuild a derived view from scratch, and diff it against the incrementally maintained one.
- ⚡ Finance asks for all events from five months ago. Retention is seven days. What do you tell them, and what should have been designed differently?
- ⚡ A team scales a consumer group from 6 to 24 instances and sees no throughput change. Explain, then explain why adding partitions is not a free fix.
- 💬 What is the difference between a log and a queue, structurally? Do not name products.
- 💬 A consumer group has been down for nine days and retention is seven. What happened, and what will the dashboards show?
- 💬 Why can a log support replay when a queue cannot? Answer from the data structure, not the feature list.