The Event Log
An append-only, immutable, ordered sequence of facts that each reader moves through at its own pace — the primitive underneath brokers, replication, CDC and stream processing.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What can an append-only log of events answer that a table of current state cannot?
Every downstream that needs history rather than the present: a stream processor computing windows, a warehouse loader rebuilding a fact table, an audit process asking what the value was in March, and a new service that did not exist when the events were produced and still needs all of them. What they need from the log is a stable position they can return to and a promise that the record at that position has not changed.
One record: an immutable statement that something happened, stored at a position — an offset — that is assigned once and never reused. A correction is not an edit of an earlier record; it is a later record. This is the single most important grain shift in the domain, because it means counting records is not counting entities (What a CDC Event Contains).
Write each event into a table with an auto-incrementing id and let consumers poll WHERE id > :last_seen ORDER BY id. It is one system instead of two, it is queryable with SQL, it is transactional with the write that produced it, and for a single consumer at modest volume it is genuinely the right answer for a surprisingly long time.
Two transactions get ids 101 and 102, but 102 commits first. A consumer polling id > 101 reads 102, advances its cursor, and never sees 101 — the row exists, is correct, and is invisible forever. The same gap-under-concurrency bug that ruins timestamp-based extraction (Incremental Extraction) ruins sequence-based extraction.
- Two transactions get ids 101 and 102, but 102 commits first. A consumer polling
id > 101reads 102, advances its cursor, and never sees 101 — the row exists, is correct, and is invisible forever. The same gap-under-concurrency bug that ruins timestamp-based extraction (Incremental Extraction) ruins sequence-based extraction. - A second consumer appears. Now two pollers are scanning the same table on their own schedules, the table is also being written to by the application, and the operational database is paying for analytical fan-out it was never sized for (Workload Isolation).
- Someone adds
DELETE FROM events WHERE created_at < now() - interval '30 days'to keep the table small. A consumer that was down for a long weekend now has a hole in its history and no way to detect it — the rows are not late, they are gone. - A consumer needs to reprocess the last six months after a logic bug. The table is still there, so this looks possible — until you notice the transformation reads a
customerstable that has since been updated in place, so the replay produces different output than the original run did and neither is reproducible (Keeping Raw History: The Recovery Position and the Liability). - The application team, entirely reasonably, adds an
UPDATE events SET status = ...to fix a bad record. Every consumer that already read that record has a different version of history than every consumer that reads it now, and nothing anywhere records that this happened.
What is actually happening
- A log is the simplest durable data structure there is: bytes appended to the end of a file, never modified in place. Position in that file is the entire addressing scheme. That is why an offset is meaningful, monotonic and cheap — it is not an index into a mutable structure, it is a byte position in an immutable one (Write-Ahead Logging).
- Immutability is what makes a position a durable contract. If a record could change, an offset would only tell you where to look, not what you would find. Because it cannot, "I processed up to offset 4,812" is a complete, restartable statement of progress (Offsets and Commits).
- The log stores changes; a table stores the result of applying them. You can always derive the table from the log by folding the records forward. You can never derive the log from the table, because the table forgot. A log is strictly more information than the state it produces — that asymmetry is the entire argument for this module.
- Reading is non-destructive. A reader holds its own position; the log does not know or care how many readers exist, and one reader falling behind changes nothing for the others. This is what makes the same log serve a real-time alerting consumer, an hourly warehouse loader and a batch reprocess simultaneously (The Event-Driven Data Platform).
- Ordering is a property of a *single* log. Two records in one log have an unambiguous before-and-after. Two records in two different logs have none, and no amount of timestamping recovers it, because the clocks belong to different machines (Topics and Partitions).
A log of changes contains a table; a table does not contain the log
Take one order and follow it. It is placed, it is paid, the shipping address is corrected, and it ships. In an operational orders table this is one row that has been updated three times, and at the end it holds exactly one thing: the final state. Three of the four facts have been overwritten by the fourth.
In a log it is four records, appended in that order, none of them modified. The final state is still available — fold the records forward and you have the row. But so is the address that was corrected, the length of time between payment and shipping, and the fact that a correction happened at all. Every one of those is a question an analyst will eventually ask, and only one of the two structures can answer them.
This is why the log is the primitive and the table is the derivative. The fold from log to table is a pure function; there is no inverse. Everything expensive about data engineering — slowly changing dimensions, snapshot tables, reconstructing history nobody kept — is the cost of having only run that fold and thrown away the input (Event vs Snapshot Modeling).
One row per order, updated in place as the order progresses. Consumers poll the table for rows whose `updated_at` moved, and reconstruct what changed by diffing against what they stored last time.
One immutable record per change, appended in order. Consumers read forward from a position they own, and the current state is whatever the fold produces at their position.
The table is lossy by construction — an in-place update destroys the prior value, and no consumer polling interval is short enough to catch two updates that happened between polls. The log is lossless within its retention window, so a consumer that was down for an hour reads exactly the changes it missed rather than a diff that silently merges them.
Log (append-only, offsets never reused)
offset event payload
------ -------------------------- -----------------------------------
4810 OrderPlaced {order:7731, total:120.00}
4811 PaymentCaptured {order:7731, amount:120.00}
4812 ShippingAddressCorrected {order:7731, city:"Leipzig"}
4813 OrderShipped {order:7731, carrier:"DHL"}
Fold forward -> table row (one row, three facts destroyed)
order_id total status city updated_at
-------- ------ -------- -------- ----------
7731 120.00 shipped Leipzig ...
Questions the table cannot answer:
- what city was this shipping to before the correction?
- how long did payment-to-ship take?
- did a correction happen at all?Reading does not consume, and that changes what the log is for
The property that turns a message-passing mechanism into data infrastructure is that a read does not remove anything. The log holds records for its retention period regardless of who has read them; each consumer holds nothing but an integer saying how far it has got.
The consequences compound. Many consumers with different purposes and wildly different speeds read the same records without coordinating. A consumer that falls behind does not slow anyone else down. A consumer with a bug rewinds its own position and reprocesses without asking the producer for anything. And a consumer that did not exist yesterday can start at the oldest retained record and build up a complete derived dataset from scratch — a capability with no analogue in a queue.
That last one is the argument for the log as a platform boundary rather than a service-to-service connector. Every new analytical use case is a new consumer group starting from the beginning, and the producing team is not asked, is not affected, and does not need to know (The Event-Driven Data Platform).
What the log does not promise
A log is a strong primitive with a narrow set of promises, and almost every production incident in this module comes from believing it promises something adjacent. It guarantees that what was appended is durable, immutable and ordered within one log. It does not guarantee completeness, uniqueness, global ordering, meaning, or that history is available for as long as you assumed.
Read the table below as a list of things you must build yourself. None of them is exotic; all of them are somebody's job, and when nobody has been assigned they are nobody's.
The failure that deserves special attention is the last one. A consumer whose committed position has aged out of retention is not slightly behind — it has a hole. Depending on configuration it will resume from the oldest available record (silently skipping the gap) or from the newest (silently skipping much more). Both look like a successful restart in the logs.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Record count in the log per closed hour of event time, versus the source system's count for the same hour. | Completeness: everything that happened was published. | Producer-side drops, a filter that excluded an event type, a topic that stopped receiving one producer's traffic. | Duplicates that offset losses exactly; any open period, where late arrivals are still expected; every value-level error in records that are all present. |
| Distinct event ids divided by record count, per hour. | Uniqueness: one record per real-world occurrence. | Producer retry storms, a redelivering connector, a replay that was pointed at a live topic by mistake. | Redeliveries that generate a fresh event id each time — those are two records that are genuinely distinct and describe one event. |
| Age of the newest record per topic, compared with the topic's stated freshness commitment. | The stream is alive and current. | A producer that stopped, a broker partition with no leader, a deployment that silently disabled publishing. | A producer that is publishing on schedule and publishing wrong data; and it fires falsely on any topic with genuinely bursty traffic. |
| Oldest retained offset age versus maximum committed consumer lag, per partition. | Every consumer can still reach every record it has not read. | The approach of unrecoverable data loss for a lagging consumer, while it is still recoverable. | A consumer that is not registered with the broker at all — an external system checkpointing offsets in its own store is invisible to this check. |
The first three checks are about the records; the fourth is about the *window*, and it is the one almost nobody has. It is also the only one whose failure is unrecoverable.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A producer's async send fails after the application has already committed its own transaction. | The order exists in the database and no event for it exists in the log. Every downstream count is short by one, forever. | The database write and the log append are two separate systems with no shared transaction — the log cannot promise completeness on the producer's behalf. | Use a transactional outbox so the event is committed with the business data and relayed afterwards, or drive the log from the database's own change log instead (Change Data Capture, The Transactional Outbox). |
| A producer retries after an acknowledgement times out, though the first write actually succeeded. | Two records describing one real-world event. Downstream sums are inflated and the pipeline is green. | At-least-once is the honest default for any network write with an ambiguous outcome. Duplicates are expected traffic. | Carry a stable event id and deduplicate at the first place that owns a keyed store, or make the sink idempotent by upserting on that key (Deduplication). |
| Two related streams — orders and payments — are joined downstream by arrival order. | A payment appears to precede its order; the join drops rows or produces a state that never existed. | Ordering exists inside one log. There is no ordering relationship between records in two different logs, and their timestamps come from different clocks. | Join on event time with an explicit tolerance for lateness rather than on arrival, and accept that the join is a windowed approximation (Stream Joins, Late Events). |
| A consumer is paused for maintenance longer than the topic's retention window. | It restarts, reports healthy, and its lag is near zero within minutes. | Its committed offset no longer exists, so the reset policy chose a valid position. The records between the two positions were never read and nothing recorded that. | Alert on oldest-retained-record age approaching maximum consumer lag, and treat an offset-reset event as a data incident rather than a startup log line (Retention and Replay). |
How to build it
Most important first.
- Model events as facts in the past tense —
OrderPlaced,PaymentCaptured,AddressChanged— not as instructions. A fact stays true no matter how many consumers read it or when; a command implies exactly one recipient who is supposed to act, which is a different shape with different delivery needs (Commands vs Events). - Put enough in each record that a consumer can act without calling back into the producing system. A record carrying only
{order_id}forces every consumer to query the source at read time, which re-couples them to it and makes replay meaningless — you would be replaying against today's state, not the state at the time. - Give every record a stable business identity — an event id, or a natural key plus a source sequence — so downstream deduplication has something to work with when redelivery happens, and it will (Deduplication).
- Carry the event time on the record, assigned by the producer, distinct from whenever the log or a consumer sees it. Every correctness question in stream processing turns on having that field (Event Time).
- Write the schema down and version it at the boundary, because the log outlives every consumer that reads it and a record written today will be read by code that does not exist yet (Schema Registry).
- Land the raw log output immutably before transforming it. The log's retention is finite; your raw layer's does not have to be, and the two together are what make a six-month reprocess possible (The Raw Landing Zone).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Durability of an appended record, once acknowledged under whatever replication setting the producer asked for. A producer that does not wait for replication has a weaker guarantee than the log is capable of, and this is a producer-side decision, not a broker property.
- Ordering within one log, by offset, and nothing across logs. This is the guarantee people most often over-read: "the log is ordered" is true and much narrower than it sounds (Topics and Partitions).
- Immutability of a written record. This is the strongest thing a log promises and the one everything else is built on.
- Delivery is at-least-once by default in every real system — producers retry on ambiguous acknowledgement, consumers restart from their last committed position. Duplicates are normal traffic, not an incident (At-Least-Once Delivery).
- No completeness guarantee. The log promises that what was written is there; it promises nothing about whether everything that happened was written. That gap is the producer's, and only reconciliation against the source closes it (Reconciliation).
- No guarantee that the record means what the consumer thinks. A schema check verifies shape; nothing verifies that
amountis still gross rather than net (Semantic Changes).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Reconcile counts per closed period between the producing system and the log: for a bounded window of event time, count records in the log and compare with the count of the corresponding entities or changes in the source. It catches producer-side drops, a misconfigured filter, and a topic that silently stopped receiving one event type.
- It misses duplicates that offset losses, anything in a period still open (late arrivals have not landed yet), and every value-level error — a log with exactly the right number of records can carry the wrong amount in every one of them.
- A second cheap check is a uniqueness rate on the event id per period. Rising duplication is usually a producer retrying more, which is itself a signal about the producer's health before it is a data problem (Duplicate Rows).
- A log removes the polling interval from the freshness budget: a record is readable as soon as it is appended and acknowledged, so end-to-end delay becomes producer latency plus consumer processing rather than producer latency plus a schedule.
- It does not make consumers fast. A log fed continuously and drained by an hourly batch job gives hourly data, and describing that platform as "real-time" because one hop is continuous misleads everyone downstream (Batch vs Streaming Ingestion).
- The freshness a log genuinely offers is per-consumer, because each consumer holds its own position. One consumer can be seconds behind while another is hours behind, on the same records, and a single platform-level freshness number hides exactly the one that is broken (Freshness Monitoring).
- The log is the hardest place in a platform to change a schema, because records already written cannot be rewritten and consumers read old and new records in the same pass. Compatibility is therefore not a nicety here, it is a physical constraint (Backward Compatibility).
- Adding an optional field with a default is safe for readers that ignore unknown fields. Renaming, retyping or removing a field breaks every consumer that reads it — usually at replay time rather than at deploy time, which is much later and much more confusing (Breaking Schema Changes).
- The change nothing catches is a meaning change with a stable shape:
statusgaining a new value,amountswitching currency basis,user_idstarting to refer to accounts instead of people. The records validate perfectly and every derived metric moves (Semantic Changes).
- The log *is* the recovery mechanism for everything downstream of it. If a transformation was wrong, you reset a consumer's position and reprocess — provided the records are still retained and the transformation is deterministic (Replay from the Log).
- That makes retention a recovery-window decision and not a storage one: retention is literally the maximum age of a bug you can fix by replaying rather than by reconstructing (Retention and Replay).
- A replay is only safe if the sink tolerates re-writing the same records — upsert on a business key, or an atomic swap of a rebuilt partition. Replaying into an append-only sink duplicates everything, and the pipeline will report success while doing it (Idempotent Data Pipelines, Upserts and Merges).
- Recovery of the log itself is a replication question, not a data-engineering one: a partition with an under-replicated write can lose the tail of its history in a broker failure, and no downstream check will tell you which records they were.
What can go wrong
- A producer that fails to write and does not surface it — a full local buffer, an exception swallowed in an async send callback. The log looks healthy; it is simply missing records nobody counted.
- Retention expiring beneath a stalled consumer. The consumer restarts, finds its committed offset no longer exists, and — depending on configuration — silently jumps to the earliest or latest available record. Both outcomes are wrong and only one of them is loud.
- A record whose payload is only an id, so every consumer calls back to the source and replay silently produces present-day answers for historical events.
- A schema change that is compatible for live traffic and incompatible for the historical records a replay reads, so the pipeline works until the day you need it not to.
- Two logically-related event streams written to separate logs, joined downstream on the assumption that their ordering is comparable. It is not, and the resulting state depends on which consumer happened to run first (Stream Joins).
- "A log is a queue with better marketing." A queue hands a message to one consumer and forgets it; a log stores the record and lets any number of readers move through it independently. Everything else in this module follows from that difference (Message Brokers: Log-Shaped and Queue-Shaped).
- "Events are ordered." Records in one log are ordered. Across logs — or across partitions of the same topic — there is no order at all, and building on an assumed global order is the most expensive mistake in this module (Topics and Partitions).
- "The log is the source of truth." It is the source of truth for *what was published*. If the producer has a bug, the log faithfully preserves the bug, and the operational database still owns the actual state (Source of Truth).
- "Event sourcing and using an event log are the same thing." Event sourcing means an application's state is *defined* as a fold over its events. Publishing events from a service that stores its state normally is a different, much more common and much cheaper design (Event Sourcing).
- "Immutable means we cannot get it wrong." Immutability protects the record, not the meaning. A permanently preserved record of the wrong number is still the wrong number.
- An immutable log and a deletion request are in direct conflict. You cannot erase a record from an append-only structure on demand, so personal data in event payloads needs a strategy decided before the first record is written — keys held externally, tokenised identifiers, or a retention window short enough that deletion is achieved by waiting (Deletion Requests).
- Every consumer of a topic inherits the classification of its most sensitive field. Fan-out means one incautious payload field is now in as many downstream systems as there are consumers, and no one of them recorded that it received it (PII in Pipelines).
Operating it
- End-to-end offset lag per consumer: the difference between the newest offset in the log and the offset that consumer has committed. It is a count of unprocessed records, and it is the single most informative number about a log-based platform (Consumer Groups and the Parallelism Ceiling).
- Append rate per topic against its own history by weekday. A stream that quietly stops is one of the most common data incidents and one of the easiest to detect (Volume Anomalies).
- Age of the oldest retained record per partition, next to the largest consumer lag on the same partition. The moment those two lines approach each other you are about to lose data, and nothing else on the dashboard will say so.
- Producer error and retry counts, which are a data-completeness signal disguised as an application metric (Retry Storms: The Load You Generated Yourself).
- At 10x append rate the log itself is usually undisturbed — sequential appends are the workload storage is best at — and the pressure moves to consumers and to partition count as the parallelism ceiling (Consumer Groups and the Parallelism Ceiling).
- At 100x, retention becomes the binding constraint long before throughput does: keeping a week of history is a very different proposition at a hundred times the volume, and the recovery window is what gets quietly shortened to pay for it.
- Consumer count scales fan-out reads and, more importantly, the coordination problem: every new consumer is a new contract against a schema you can no longer change unilaterally (Data Contracts).
- Cardinality of keys scales the skew problem rather than the volume problem — a log is happy with a hot key and the consumer reading that partition is not (Event Keys and Partition Assignment).
- Bytes retained, which is retention window multiplied by append rate multiplied by replication factor. Replication factor is the multiplier people forget, and it applies to every byte for the whole window.
- Bytes read, which scales with the number of independent consumers rather than with the data. Fan-out is the log's best feature and its clearest cost driver — ten consumers read the same records ten times.
- Cross-zone or cross-region traffic when producers, brokers and consumers are not co-located. This is frequently the largest line and it is invisible in every architecture diagram (Multi-Zone Deployment).
- Reprocessing: a replay re-reads and re-computes history, so its cost is proportional to the window you replay, not to the size of the bug.
- A log is a second durable copy of your event history with its own operational burden, its own retention policy, its own security surface and its own on-call. It buys replay, fan-out and decoupling; it costs a system that must never lose writes.
- Immutability buys reproducibility and costs correction ergonomics: fixing a bad record means publishing a compensating one and teaching every consumer to apply it, which is strictly harder than an
UPDATE. - Decoupling producers from consumers means the producer no longer knows who depends on the shape of its events, so a change that is locally safe becomes globally unsafe. The coupling did not disappear; it became invisible, and a schema registry is what makes it visible again (Contract Enforcement).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALAppend-only, immutable, offset-addressed, non-destructive read: these four properties define the primitive and hold for Kafka, Pulsar, Kinesis, a database WAL and a plain file. What differs between them is retention control, partition semantics and how a reader's position is stored.
- BROKER-SPECIFICKafka and Pulsar let a consumer seek to an arbitrary retained offset; Kinesis addresses positions by shard iterator with a bounded window; Pub/Sub has no offset a consumer can reason about at all and replays are configured as a subscription seek to a timestamp. "Replay" therefore means three different things.
- SIMPLIFIEDTreating the log as one ordered sequence is a teaching shape used here deliberately. Every production topic is several independent logs and the ordering story changes completely — that is the whole subject of Topics and Partitions and it is separated out on purpose.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a log is the natural shape for replicating state across machines at all — state-machine replication, total order broadcast, and the consensus protocol that decides which append wins when two brokers disagree. Everything this lesson calls "durable" rests on that.
- — Distributed Systems also owns the impossibility results behind the completeness gap here: no protocol makes a database commit and a separate log append atomic without a distributed transaction, which is why the outbox pattern exists.