Kafka as a Log, Not a Queue
A partitioned, durable, append-only log with independent consumer groups and time-based retention. Records survive being read, which is the property everything else in a data platform is built on.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Why does calling Kafka a message queue lead teams to build the wrong thing?
A data platform that needs the same records more than once: a streaming aggregation now, a warehouse load hourly, a search index rebuild on demand, and a reprocess of six weeks of history the day someone finds a bug in the transformation. None of those is possible against a broker that deletes on consume, and all four are routine against one that does not.
One record in one partition at one offset. That triple is the address, and every operational conversation about Kafka — lag, ordering, replay, rebalancing — is a statement about it. A record is never addressed by topic alone, and forgetting the partition is where most reasoning about ordering goes wrong (Topics and Partitions).
Treat Kafka as a faster, more durable message queue: producers publish, one consumer service subscribes, messages are processed and forgotten. The mental model is familiar, the client library cooperates, and for a single point-to-point integration it produces a working system that behaves acceptably.
Analytics asks for the same events. Under the queue model the answer is to have the existing consumer re-publish them to a second topic — a second integration with its own gaps and its own lag. Under the log model the answer is one line of configuration: a new consumer group (Consumer Groups and the Parallelism Ceiling).
- Analytics asks for the same events. Under the queue model the answer is to have the existing consumer re-publish them to a second topic — a second integration with its own gaps and its own lag. Under the log model the answer is one line of configuration: a new consumer group (Consumer Groups and the Parallelism Ceiling).
- A transformation bug is found six weeks late. Under the queue model the records are conceptually gone and the fix is a reconstruction from whatever the warehouse still holds. Under the log model it is an offset reset, provided retention was set as a recovery window rather than as a disk-space setting (Retention and Replay).
- Throughput is short, so the team adds consumer instances. Nothing changes, because the topic has six partitions and there are already six active instances; every instance beyond that is assigned nothing at all (Consumer Groups and the Parallelism Ceiling).
- One malformed record cannot be parsed. The consumer retries it forever and every subsequent record in that partition waits behind it, while five other partitions look completely healthy and overall lag looks merely elevated.
- A "clean-up" reduces retention to save disk. Two months later an incident needs a three-week replay and the window is four days, and there is no version of this problem that can be solved after the fact (Replay from the Log).
What is actually happening
- A topic is a set of partitions; a partition is a sequence of segment files on a broker's disk; a record is bytes at an offset in one of those files. Appending is a sequential write to the end of the active segment, which is the access pattern storage hardware is best at and the reason the log sustains write rates a random-access structure could not. Nothing is indexed on write, nothing is rebalanced, nothing seeks.
- Reading is a sequential scan from a requested offset. Because the broker does not track per-consumer message state, serving a second consumer group is the same work as serving the first, and the broker can hand bytes from the page cache to the network with very little copying (Zero-Copy: Serving a File Without Touching It).
- Retention is a property of the topic, not of consumption. A record is deleted when it ages out of the retention window or the partition exceeds a size bound — never because someone read it. This single design decision is what separates the log from every queue (Message Brokers: Log-Shaped and Queue-Shaped).
- A consumer group's position is stored as a committed offset per partition, held by the broker in an internal topic. That is why a consumer can crash and resume, why a new group starts wherever you tell it to, and why "where are we" is one small integer per partition rather than a per-message ledger (Offsets and Commits).
- Durability comes from partition replication: each partition has a leader and followers, and a producer chooses how many replicas must acknowledge before the write is considered done. That knob is the actual durability guarantee — the word "durable" on its own is a description of the architecture, not a promise about your data.
- Log compaction is a second, optional retention mode that keeps the latest record per key and discards earlier ones. It produces a compacted changelog, which is a snapshot of current state, and it is emphatically not history (Retention and Replay).
The record stays; only the reader moves
The clearest way to see the difference is to draw what happens after a consumer processes a record. In a queue, the record is gone: the broker deleted it because the consumer said it was done. In a log, nothing at all happened to the record — a single integer moved.
That integer is the whole of a consumer's state, and it is why every capability this module cares about exists. Rewind it and you reprocess. Create a second one under a different group name and you have an independent consumer that the first one cannot see or affect. Never advance it, and the broker still keeps the records until retention says otherwise.
It is also why the log offers no per-message failure handling. There is no per-message state to record "this one failed" in, so a consumer that cannot process record N has exactly two honest options: stop, or move past it and put it somewhere else. Choosing neither — retrying forever — is the default, and it is how one bad record stops a partition (A Dead-Letter Queue Is a Workflow, Not a Bin).
What is actually on disk
A partition is a directory of segment files. Records are appended to the newest segment until it rolls; older segments sit there until retention removes them whole. An index alongside each segment maps offsets and timestamps to byte positions, which is how a consumer can seek to an arbitrary offset — or to a timestamp — without scanning.
Two consequences of this layout are worth internalising. First, deletion happens at segment granularity, not record granularity, which is why retention is approximate at the edges and why "delete this one record" is not an operation the structure supports. Second, a consumer reading recent records is reading pages the operating system already has cached from the writes, so fan-out to several current consumers is far cheaper than it looks (CPU Cache Is Not the Page Cache).
The compaction mode replaces this picture: instead of dropping old segments by age, a background process rewrites them keeping only the most recent record per key. What you end up with is a compact snapshot of current state per key — useful as a changelog for a state store or a lookup table, and useless for any question about history (Stateful Stream Processing).
topic "orders" — 4 partitions, each an independent log
/kafka-logs/orders-0/
00000000000000000000.log <- oldest retained segment
00000000000000000000.index <- offset -> byte position
00000000000000000000.timeindex<- timestamp -> offset (how "seek to a time" works)
00000000000000524288.log
00000000000000917504.log <- active segment, appends land here
Retention deletes WHOLE SEGMENTS from the head:
[ expired ][ retained ......................... ][ active ]
^ earliest readable offset ^ log end
A consumer group is just:
(group, topic, partition) -> committed offset
Compaction is a different mode entirely:
before: k1=a k2=b k1=c k3=d k1=e k2=f
after: k3=d k1=e k2=f
-> current state per key, and no history at allFile names, index formats, compaction scheduling and the availability of tiered storage that offloads old segments to object storage all change between versions and between managed offerings. The structural facts — segment-granularity deletion, offset and timestamp indexes, compaction keeping the latest record per key — have been stable for years; verify anything more specific against current documentation.
Retention is a recovery window, and it is the cost line that pays for it
Almost every argument about a Kafka cluster's cost eventually lands on retention, and it is usually framed as a storage question. Framing it that way guarantees the wrong answer, because the benefit of retention is not storage — it is the maximum age of a mistake you can fix by replaying instead of by reconstructing.
The drivers below are ordered by how much they typically move the total for an analytics-facing cluster. The point of the ordering is that the two largest are structural decisions made once — replication factor and retention window — while the one teams reach for first, partition count, is comparatively small unless it has been set absurdly.
The honest way to have the retention conversation is to state the recovery window as a commitment: "we can reprocess any bug found within N days without touching the raw layer". Then the cost is being spent on something a business can evaluate, rather than on a number in a config file that somebody halved during a cost review (What Actually Drives Data Platform Cost).
Retention window times append rate times replicas. Every byte is paid for by every replica for the whole window, and this line accumulates whether or not anyone reads it.
Producers, replicas and consumers spread across zones pay for every hop. Frequently the single largest line and completely invisible in the architecture diagram.
Scales with fan-out rather than with data. Recent-record reads are comparatively cheap because they are served from cache; a replay from the earliest offset is not.
A replay costs read bandwidth plus the full downstream compute of the window replayed. Proportional to the window, never to the size of the bug.
Open files, replication streams and metadata per partition. Small unless the partition count was chosen by wishful thinking, at which point it stops being small.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for an analytics-facing cluster, shown to establish an ordering rather than a magnitude. They are not measurements and do not transfer to a specific deployment. The teaching is the ordering: retention and network placement dominate, and the knob people reach for first is near the bottom.
How to build it
Most important first.
- Set retention from the recovery window you want, then size storage for it. Doing it the other way round means the recovery window is whatever fell out of a disk budget, and nobody has ever written that number down (Retention and Replay).
- Give every distinct consumer purpose its own consumer group. Groups are the unit of independent progress, and sharing one between two purposes couples their lag, their failures and their replays permanently (Consumer Groups and the Parallelism Ceiling).
- Make the first consumer of every topic a raw landing writer that persists arrivals untouched to object storage. It converts the broker's finite retention into history you control and costs almost nothing (The Raw Landing Zone).
- Choose the producer acknowledgement setting deliberately and record the choice next to the topic's contract. Waiting for replicas costs latency and is the difference between "we do not lose data" being true and being aspirational.
- Handle poison records explicitly: catch, publish to a quarantine topic, commit past. Without that code a single unparseable record halts a partition, and no amount of retry configuration turns that into progress (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Register schemas and enforce compatibility at the boundary. Records live for the whole retention window and replays read old ones, so the compatibility question is not about deployment ordering — it is about whether a replay next quarter still works (Schema Registry, Contract Enforcement).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Durable once acknowledged at the replication level the producer requested — and only at that level. A producer that does not wait for in-sync replicas is trading durability for latency, whatever the cluster is capable of.
- Ordered by offset within a partition. Not ordered across partitions of the same topic, and not ordered across topics (Topics and Partitions).
- Retained for the configured window regardless of who has consumed, which is the property that makes replay and multi-consumer fan-out possible at all.
- At-least-once delivery in the default configuration. Producer idempotence and transactions can eliminate producer-side duplicate *appends* and can tie an offset commit to a write within the same cluster — that is a narrow, useful, well-scoped guarantee and it is not end-to-end exactly-once (Offsets and Commits, Exactly-Once: Input Consumption, State Update, Output Write).
- No completeness guarantee relative to the producing system. A change that was never published is invisible to every Kafka metric (Reconciliation).
- No guarantee that a consumer that committed an offset actually did the work correctly. The offset records intent, not success.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Reconcile per closed hour of event time: records in the topic versus changes in the producing system. Run it against a closed window so in-flight records are not counted as losses, and alert on any divergence rather than on a threshold.
- It misses duplicates that offset losses, and it cannot see records that a producer failed to publish while also failing to record the failure — which is why producer error rate belongs on the same dashboard.
- A second check worth having is a replay equivalence test: reprocess a closed historical window into a scratch location and compare the result with what is in production. If they differ, the pipeline is not deterministic, and every recovery plan that assumes replay works is fiction (Reprocessing vs Retrying).
- Records are readable as soon as they are appended and replicated, so the broker removes scheduling from the freshness budget entirely. What remains is producer batching, replication acknowledgement, and however long the consumer takes.
- Producer batching is an explicit freshness-versus-throughput knob: larger batches mean fewer, bigger writes and more delay before a record is visible. It is one of the few places in a data platform where you can buy freshness with cost directly (Cost vs Freshness).
- The log makes stale consumers survivable rather than fast. A consumer hours behind is still going to read every record, in order, and end up correct — which is a different and more valuable property than low latency (Consumer Groups and the Parallelism Ceiling).
- The retention window is the schema-compatibility horizon. Any record still retained can be read by a replay, so a change is safe only if today's consumer code can read the oldest retained record (Backward Compatibility).
- Adding a partition is a schema change in disguise for anything that depends on per-key ordering, because the key-to-partition mapping changes for future records (Event Keys and Partition Assignment).
- Compacted topics evolve differently again: a key's last record persists indefinitely, so a field removed from the schema today may still be present in records written years ago and never superseded.
- Reset a consumer group's committed offsets to an earlier position and let it reprocess. This is the primary recovery mechanism of a log-based platform and it is bounded entirely by retention (Replay from the Log).
- Replay into a scratch destination and validate before swapping it in. Replaying directly over a live serving table means consumers read a half-rebuilt state, and the pipeline reports success throughout (Atomic Publish, Validating a Backfill Before You Publish).
- Replay is only correct if the transformation is a pure function of the records. Any reference to
now(), to a mutable dimension, or to a non-idempotent merge makes the replay produce a different answer than the original run, and neither is reproducible (Idempotent Data Pipelines). - When retention has already expired, the recovery path is the raw landing zone, not the broker. That is the entire argument for having one.
What can go wrong
- Retention expiring beneath a stalled consumer group, followed by an offset reset to earliest or latest. Both silently skip records and both look like a normal restart.
- A poison record blocking one partition indefinitely, with aggregate lag looking merely elevated because the other partitions are fine.
- A producer configured for speed rather than durability, so an unluckily-timed broker failure loses acknowledged writes that no downstream check can identify.
- Consumer instances added past the partition count, producing no throughput improvement and a plausible-looking deployment that changed nothing (Consumer Groups and the Parallelism Ceiling).
- A partition count increase applied to fix throughput, which fixes throughput and quietly breaks per-key ordering for every key from that moment forward (Event Keys and Partition Assignment).
- A replay pointed at a live topic instead of a scratch consumer group, so the reprocessing consumer competes with the production one and both make partial progress.
- "Kafka guarantees exactly-once." It does not, and the phrase is meaningless without saying which of input consumption, state update and output write it applies to. Kafka can make an offset commit and a write to another Kafka topic atomic within one cluster; the moment your sink is a warehouse or an object store, that transaction does not extend to it (Exactly-Once: Input Consumption, State Update, Output Write).
- "Kafka is a message queue." It is a log. A queue deletes on consume; Kafka deletes on a timer. Every difference in this lesson descends from that (Message Brokers: Log-Shaped and Queue-Shaped).
- "Retention is a storage setting." Retention is the maximum age of a bug you can fix by replaying. It is a recovery-window decision that happens to be spent in storage (Retention and Replay).
- "Kafka is ordered." Ordered within a partition. A topic with twelve partitions has twelve independent orderings and no relationship between them (Topics and Partitions).
- "Adding consumers adds throughput." Only up to the partition count. Beyond that they idle, and the deployment looks entirely successful (Consumer Groups and the Parallelism Ceiling).
- "Log compaction gives us history in less space." Compaction keeps the latest record per key and discards the rest. It gives you current state cheaply and destroys exactly the history you would replay.
- Records in a topic are immutable for the whole retention window, so a deletion request cannot be satisfied by editing them. The workable designs are keeping personal data out of payloads, holding it behind a token resolved elsewhere, or setting retention short enough that deletion is achieved by waiting (Deletion Requests).
- A compacted topic keyed by a person's identifier retains their last record indefinitely, which turns a convenient state store into an unbounded personal-data retention decision that nobody consciously made (Data Retention).
- Every consumer group is another system holding a copy of whatever the payload contains. Fan-out multiplies the classification obligation, and the broker records who is reading but not what they then did with it (PII in Pipelines).
Operating it
- Consumer lag in records, per partition, per group. Per-partition is the resolution that matters — a single blocked partition is invisible in a topic-level total (The Backlog Arithmetic: Four Levers and a Drain Time).
- Oldest retained record age per partition, on the same chart as the largest consumer lag. When those converge you are about to lose data and every other panel will look fine.
- Offset-reset and rebalance events, treated as data incidents rather than as informational log lines.
- Under-replicated partition count, which is the leading indicator for the one failure in this lesson with no recovery path.
- Producer error and retry rate, the only visible proxy for records that never entered the log at all (Retry Storms: The Load You Generated Yourself).
- At 10x append rate the log itself is usually unbothered — sequential appends and sequential reads are the friendliest possible workload — and the constraint moves to partition count as the consumer parallelism ceiling.
- At 100x, retention becomes the binding constraint and the temptation is to shorten it. That trades an invisible recovery capability for a visible storage line, which is exactly the trade that looks correct on a cost review and is wrong (Retention and Replay).
- Consumer group count scales read bandwidth and, more importantly, the number of parties who now depend on the schema. Technical scaling is easy here and organisational scaling is not (Data Contracts).
- Key cardinality decides skew. A log accepts a hot key happily; the consumer assigned to that partition is the one that falls behind, and nothing rebalances it away (Event Keys and Partition Assignment, Data Skew).
- Bytes retained, multiplied by replication factor, multiplied by the retention window. Replication factor is a multiplier on every byte for the whole window and is routinely left out of capacity conversations.
- Bytes read, which scales with consumer group count rather than with data volume. Fan-out is the reason to use the log and the clearest driver of its read cost.
- Cross-zone network transfer between producers, brokers and consumers, which is frequently the largest single line and is invisible in the architecture diagram (Multi-Zone Deployment).
- Partition count itself carries fixed overhead per partition — open files, replication streams, metadata — so a topic with an extravagant partition count costs whether or not it carries traffic (Topics and Partitions).
- Reprocessing: a replay re-reads and re-computes an entire window, so its cost is set by the size of the window rather than by the size of the mistake.
- The log buys replay, fan-out and independent consumer progress. It costs a stateful distributed storage system to operate, a partition count that must be planned in advance, and schema discipline that lasts as long as retention.
- Retention buys a recovery window and costs storage continuously, including for the overwhelming majority of records nobody will ever re-read. That is insurance, and pricing it as though it were working storage is how recovery windows get cut.
- No per-message acknowledgement means no built-in dead-lettering, so every consumer must implement poison-record handling itself or accept that one bad record can stop a partition.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka: partitions are the unit of ordering and parallelism, offsets are consumer-controlled and seekable, retention is time or size based per topic. Kinesis orders per shard and addresses positions with shard iterators over a bounded window; Pub/Sub has no consumer-visible offset at all and replays are a subscription seek to a timestamp; Pulsar separates serving brokers from storage segments, so scaling and retention behave differently.
- GENERALThe primitive — append-only partitions, offset-addressed reads, retention independent of consumption — is what to carry to any other system. The specific configuration names are the part that will be different or gone in a few years.
- SIMPLIFIEDThis lesson describes a single cluster with local disks. Tiered storage, cross-cluster replication and stretched clusters change the cost curve and the failure modes substantially, and are deliberately out of scope here so that the structural argument stays visible.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns partition replication itself — leader election, in-sync replica sets, and what an acknowledgement from a quorum actually promises when the network partitions. Every durability claim in this lesson bottoms out there.
- — DevOps / Production Engineering owns running the cluster: capacity planning, rolling upgrades, broker replacement and the runbook for under-replicated partitions.