CDCGENERALSOURCE-SPECIFICSIMPLIFIED

What a CDC Event Contains

An operation, a before image, an after image and source metadata. Which of those you actually receive is decided by the source's configuration, not by CDC.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

A CDC record arrives describing an update to one order. What is inside it, what is deliberately absent, and which parts can you rely on being there?

Who needs this

Everyone who has to turn a change record back into something meaningful: the transformation that reconstructs current state, the audit table that needs the previous value, the stream job that computes time-in-status, and the debugging engineer at 02:00 who needs to know which transaction produced this row (Debugging a Data Incident).

What one row is

One event is one committed change to one row of one table. A transaction that touched three tables produces at least three events, and a statement that updated two hundred thousand rows produces two hundred thousand events. The transaction is metadata on the event, not a container for it (Grain: What Does One Row Represent?).

The obvious build

Take the after object out of each event, write it into a table, and treat that table as a copy of the source. It is the shortest path from a change stream to something queryable, and for an append-only table with no updates and no deletes it is entirely correct.

Why it breaks

Deletes carry no after — the row is gone, so there is nothing to write. Taking after and ignoring the operation means every delete becomes a no-op and the downstream table keeps rows the source removed (Missing Rows).

How it breaks with real data

Deletes carry no after — the row is gone, so there is nothing to write. Taking after and ignoring the operation means every delete becomes a no-op and the downstream table keeps rows the source removed (Missing Rows).
Updates arrive as full row images, so an append of after values produces one row per change rather than one row per order. COUNT(*) on the resulting table counts changes and is presented as orders (Duplicate Rows).
A consumer needs the previous value — to compute a delta, to audit a price change, to detect a status transition — and discovers that before is null on every event because the source was never configured to supply it.
Two events for the same key arrive in the opposite order after a partition rebalance, so a naive last-write-wins by arrival writes the older value and leaves it there permanently (CDC Ordering and Transaction Boundaries).
The connector's own metadata fields — position, transaction id, snapshot flag — are dropped during a "clean the payload" step, and the pipeline loses the only fields that could have proved ordering or completeness.

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

A decoded change event is an envelope: an operation code, an optional before row image, an optional after row image, and a source metadata block. Different connectors name the fields differently, but every log-based CDC system produces this shape because it is what the log records (Write-Ahead Logging).
The operation is the discriminant, and there are more than three. Insert, update and delete are universal; a truncate may or may not be represented; and a snapshot read is a synthetic event with no corresponding log record, marked as such so consumers can tell bootstrap from live change (Snapshot and Stream: the Bootstrap Problem).
The source metadata is the load-bearing part. Log position orders the stream, transaction id groups changes that committed together, commit timestamp gives event time, and the table and schema names say what this is a change to (Event Time).
The before image exists in the storage engine because multi-version concurrency keeps prior row versions for concurrent readers. Whether it reaches the connector is a source configuration — how much of the old row the log is told to record (MVCC: Multi-Version Concurrency Control, UPDATE, DELETE and Dead Tuples).
A delete event always carries the key and, where configured, the full previous image. Without the previous image a delete is still actionable — you know which key to remove — but an audit trail built from it will record that something vanished without recording what it was.
Columns the source cannot represent in the log come through degraded: large out-of-line values may arrive as an unchanged-placeholder rather than a value, which reads as "this column is null now" to a consumer that does not check (Nullability & Defaults).

The envelope

SOURCE-SPECIFICThe lsn, txid and commit_ts names are Postgres-flavoured. MySQL exposes a binlog file plus position and a server-assigned transaction identifier; MongoDB exposes an opaque resume token and a cluster time. The roles are identical and the field names are not, so a parser written against one source does not port.

Written out, a change event is unremarkable — and that is the point. There are four parts, and knowing which of the four you may rely on is most of what this lesson teaches.

The operation says what happened. The two row images say what the row looked like on either side of it. The source block says where in the log it happened, which transaction it belonged to, and when that transaction committed. Consumers spend most of their attention on after and most of their incidents on the other three.

The metadata is not decoration. lsn is the ordering key for the entire pipeline and the guard that makes an idempotent sink possible. txid is the only evidence that two events belonged to one atomic change. commit_ts is the event time, and the only timestamp that describes the business rather than the plumbing.

A change event, conceptually

1{
2  "operation": "UPDATE",
3  "before": {
4    "order_id": 88124,
5    "status": "paid",
6    "amount_minor": 4990,
7    "currency": "EUR",
8    "updated_at": "2026-03-11T09:12:03Z"
9  },
10  "after": {
11    "order_id": 88124,
12    "status": "refunded",
13    "amount_minor": 4990,
14    "currency": "EUR",
15    "updated_at": "2026-03-11T09:41:57Z"
16  },
17  "source": {
18    "table": "public.orders",
19    "lsn": "0/1A2F4C88",
20    "txid": 918273,
21    "commit_ts": "2026-03-11T09:41:57.318Z",
22    "snapshot": false
23  }
24}

before is the field to be suspicious of. On a delete it is the only row image there is; on an update it may be the full previous row, only the key columns, or nothing at all — decided by the source's configuration and not by CDC. Design nothing that depends on it until you have inspected real events from your own source.

What survives when the grain changes

The most expensive mistake made with CDC events is not a parsing error. It is treating a stream of changes as a table of entities, which is a grain confusion, and grain confusions produce numbers that are precise, confident and wrong (Grain: What Does One Row Represent?).

Follow one order through the table below. It exists once in the source, produces three change records, lands as three rows in raw, becomes one row in a staging model, and contributes one row to a fact table. Every one of those transitions is legitimate. Every one of them is also a place where a COUNT(*) means something different from what the person writing it believes.

The breaksIf column is the useful one during review. When someone proposes a model over CDC data, the question to ask is not "is the SQL right" but "what does one row of your input represent, and what does one row of your output represent" — and if those two answers are the same, something is wrong.

One order, followed through a CDC pipeline

Stage	One row is	Breaks if
Source `orders` row	One order, in its current state, with no memory of its previous ones.	You assume it holds history. It holds the latest value of every column and has forgotten the rest (Slowly Changing Dimensions).
CDC event	One committed change to one row — created, paid, refunded are three events.	You count events and call the result orders. Three changes to one order are three records and one order.
Raw change file	One delivered event, possibly delivered more than once.	You treat the file as a deduplicated set. At-least-once delivery makes duplicates normal, not exceptional (At-Least-Once Delivery).
Change history table	One distinct change, deduplicated on key plus log position.	Deduplication is done on the payload rather than on the position, so two genuine identical-value updates collapse into one and a transition disappears.
Current-state model	One order, holding the values from its highest-position change, with deleted keys absent.	"Highest" is chosen by arrival time rather than log position, so an out-of-order update wins and the row holds the second-newest value (CDC Ordering and Transaction Boundaries).
`fct_orders`	One order at a declared grain, with measures and dimension keys.	It is built from the change history rather than the current-state model, multiplying every measure by the number of times each order was touched.
Revenue tile	One number, with the grain now entirely invisible.	Any of the above went wrong. The tile will render the result with total confidence either way (The Pipeline Succeeded. The Data Is Wrong.).

Three grain changes in seven hops. The dangerous one is the fourth: collapsing a change history into current state is where ordering, deletes and duplicates all have to be handled at once, and it is usually written as a single window function nobody reviews.

Handling the operation, and handling it in the right order

A consumer of a change stream is a switch. Writing that switch in the wrong order — or leaving a branch off — is the difference between a table that tracks its source and a table that slowly diverges from it in a way no count will reveal.

The comparison below is the whole lesson in two paragraphs of SQL-shaped thinking. Appending after values is what everyone writes first; it is simple, it is fast, and it produces a table whose row count grows with change volume rather than with the business, and which never loses a row the source deleted.

The better version does three things at once: it keys on the source primary key, it guards on log position, and it acts on the operation. Those three together give effectively-once state at this sink — not because the delivery became exactly-once, but because reapplying an already-applied change is now a no-op. The assumption that buys it is that the sink can store the applied position alongside the row (Exactly-Once: Input Consumption, State Update, Output Write).

Checks on a CDC-derived current-state table, and what each still misses

Check	Expresses	Catches	Still misses
Primary key uniqueness on the current-state table	One row per source entity — the grain the model claims.	An append-instead-of-merge sink, a fan-out join, a re-run that reinserted history.	A table that is unique and stale, holding the second-newest value for every key because ordering was taken from arrival.
Applied position is non-decreasing per key	Changes were applied in source commit order.	Out-of-order application after a rebalance or a parallel consumer; a replay that overwrote newer state with older.	Changes that never arrived at all — a gap leaves the sequence non-decreasing and simply short (CDC Failure Modes and the Retention Deadline).
Key-set difference against the source for a closed period	Completeness including deletions.	Ignored delete events, a table silently dropped from capture, rows lost during a re-snapshot.	Entities created and deleted entirely within the period; any error in column values, since it compares keys only.
Null rate of `before` on update events, tracked over time	The source is still configured to supply previous images.	A replica-identity or row-format change on the source that silently removed the previous image.	A previous image that is present but partial — key columns only reads as non-null and carries almost nothing.

The second and third rows are the pair that matters. Ordering checks catch application bugs and cannot see gaps; key-set reconciliation catches gaps and cannot see ordering. Running only one of them is the usual state of a CDC pipeline.

Append the `after` image

For every event, insert `after` into the target table. Deletes are skipped because there is nothing to insert; ordering is whatever the consumer read; duplicates are inserted again.

Merge on key, guarded by position, branching on operation

For every event: if the operation is a delete, remove the key (or mark it deleted with the position and commit timestamp). Otherwise upsert `after` on the primary key, applying only when the event's log position exceeds the position already stored for that key. Record the applied position with the row.

Delivery is at-least-once and post-partitioning arrival order is not guaranteed, so the sink has to be both idempotent and order-insensitive on its own. Keying on the primary key handles duplicates, the position guard handles reordering, and the operation branch handles the deletes that an append can never represent — and all three are needed, because each fixes a failure the others do not.

How to build it

Most important first.

Branch on the operation first, always. Every consumer of a CDC stream should be a switch over operation whose delete branch is written before the update branch, because the delete branch is the one people forget.
Persist the whole envelope in raw, metadata included. The fields that look like connector noise are exactly the fields an incident needs, and they cannot be reconstructed later (The Raw Landing Zone).
Reconstruct current state with an upsert keyed on the source primary key, guarded by log position: apply the change only if its position is greater than the position already recorded for that key. That single guard makes the sink both idempotent and order-insensitive (Upserts and Merges).
Decide explicitly whether the platform keeps history. Collapsing to current state is cheaper and answers fewer questions; keeping every change is the raw material for slowly changing dimensions and for any transition metric (SCD Type 2 in Practice).
Verify before availability by inspecting real events from your source before designing anything that depends on it. It is a configuration question with a per-source answer and it is the most common assumption failure in this module.
Treat unchanged-value placeholders as a first-class case. A transformation that cannot distinguish "this column is now null" from "the log did not carry this column" will null out real data during a merge.

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

The envelope guarantees the operation and the key for every committed change, at least once, ordered by the log position it carries. Those three fields are the ones you may rely on.
It does not guarantee before. Availability is source-specific and configuration-specific, and a null before is indistinguishable from a configured-off before unless you check the connector's settings (CDC vs Polling).
It does not guarantee that after is a complete row. Sources may omit unchanged large values, and some emit only the changed columns plus the key.
It does not guarantee transactional atomicity: the transaction id groups events that committed together, but nothing downstream applies them together unless a consumer buffers by transaction and commits at the boundary (Distributed Transactions).
It does not deduplicate. The same event, with the same log position, may be delivered more than once after a restart — which is exactly why the position belongs in the payload and in the sink's guard (Deduplication).
Ordering holds per source log position and, after publication, per partition. Across partitions there is no order at all (Topics and Partitions).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

Assert the envelope's shape at the boundary: operation present and in the known set, key present and non-null, log position present and greater than the previously seen position for that partition (Contract Enforcement).
Assert per-table operation mix against its own history. A table that suddenly produces zero deletes, or nothing but deletes, is a signal long before anyone notices the downstream number (Volume Anomalies).
These miss the case where every field is present and the after image is truncated — the shape is valid, the operation is right, and one column has quietly become null for every row that carried a large value.
They also miss semantic drift entirely: an amount_minor that switched currency conventions produces perfectly valid envelopes forever (Semantic Changes).

Freshness

The commit timestamp in the metadata is the event time and it is the only trustworthy one. Arrival timestamps measure the pipeline, not the business, and using them for windowing produces results that shift when the pipeline lags (Processing Time).
The gap between commit timestamp and arrival timestamp is the pipeline's freshness, measured per event rather than per run — which is the useful shape, because a lag spike is a tail phenomenon and an average hides it (Tail Latency: Why p50 Being Fine Does Not Help).
Snapshot events carry a commit timestamp that is the moment of the snapshot read, not the moment the row was originally written. A consumer that treats them as event time will see the entire history of the source appear to have happened during the bootstrap (Snapshot and Stream: the Bootstrap Problem).
A consumer that must reconstruct state can only answer "as of the newest position applied", which is a stronger and more honest freshness statement than a wall-clock one.

When the schema or meaning changes

The event's payload shape is the source table's shape. Adding a column adds a field to after — usually harmless for consumers that name their columns, and quietly breaking for anything that positionally unpacks (Schema Evolution).
The envelope itself also has a version, defined by the connector. A connector upgrade can change field names, nesting or the representation of decimals and timestamps, which is a schema change to every consumer that no source migration caused (CDC and Schema Drift).
Registering the payload schema and rejecting incompatible versions at the boundary is what turns a silent reshape into a loud failure (Schema Registry).
A column dropped upstream disappears from after on new events while old events in raw still carry it — so any model reading across the boundary must tolerate both shapes (Backward Compatibility).

How to re-run this safely

The whole envelope, retained raw, is what makes recovery possible: current state can be rebuilt from a change history, but a change history cannot be rebuilt from current state (Keeping Raw History: The Recovery Position and the Liability).
A replay is safe precisely because the position guard is in the payload — reapplying an event whose position is not greater than what is recorded is a no-op, so re-reading a topic converges rather than corrupts (Replay from the Log).
Recovering a mis-collapsed table means reprocessing the change history with a corrected reduction, not patching rows. Patching produces a table that disagrees with its own source of truth at the next replay (Reprocessing vs Retrying).
If raw kept only after values, the recovery of any question involving previous values is impossible, and no amount of reprocessing brings it back.

What can go wrong

Failure modes

Delete events silently ignored because the consumer only handled inserts and updates — the most common single bug in CDC consumption.
before assumed present, found null, and the audit or delta logic quietly producing nulls rather than failing.
Large out-of-line values arriving as unchanged-placeholders and being merged in as real nulls, wiping a column across the table.
Snapshot events mistaken for live changes, so a bootstrap looks like every row in the source having been updated at once (Volume Anomalies).
The mitigation fails too: a schema assertion at the boundary that checks only field presence will pass a payload whose decimal representation changed from a number to a scaled byte string, which types cleanly and means something different.

Misreads

"before is part of CDC." It is part of *some* CDC configurations. Assuming it is available is the single most common design failure in this module (CDC vs Polling).
"The event is the order." The event is a change to a row. Reconstructing the order from a sequence of changes is a modelling decision with a right answer and several wrong ones (Event vs Snapshot Modeling).
"Null in after means the column is null." It may mean the log did not carry an unchanged large value. Those are different facts and merging them is how a column gets wiped.
"We can drop the metadata, it is connector noise." Log position and transaction id are the only fields that carry ordering and atomicity information. Dropping them makes every future ordering question unanswerable.
"A delete event deletes the data." It deletes a row downstream if a consumer acts on it. The value remains in raw, in the broker until retention expires, and in every table built before the delete (Deletion Requests).

Privacy, retention and access

The before image is a privacy amplifier: a stream configured for full previous images retains the pre-correction value of every field, which defeats the point of a correction and extends the retention of data someone asked to have changed (PII in Pipelines).
Column filtering belongs at the connector, not at the transformation. A column excluded at capture never enters the broker, the raw layer or any copy taken from them (Data Minimization).
A delete event is an instruction, not an erasure. Honouring a deletion request means propagating it through raw, the broker, every derived table and every extract — which is a designed capability, not a consequence (Deletion Requests).
Envelope metadata identifying transactions and source hosts is useful for debugging and is itself operational data that belongs under the same access controls as the payload (Audit Logs for Privileged Actions).

Operating it

How you see it in production

Event count by operation by table, per interval. The mix is a stable fingerprint per table, and a shift in it is a genuine signal (Pipeline Metrics).
Null rate of before on update events. A jump to one hundred percent is a source configuration change nobody announced.
The count of snapshot-flagged versus streamed events, so a connector that silently re-snapshotted is visible immediately rather than as a duplicate-row incident a day later.
Distribution of commit-timestamp to arrival-timestamp gap per table (Freshness Monitoring).

What changes at 10x and 100x

At 10x change volume, payload width becomes the dominant broker cost and the argument for column filtering at the connector stops being about privacy and starts being about bytes.
At 100x, per-transaction buffering by a consumer stops being viable — a bulk statement produces a transaction too large to hold in memory before applying (Stateful Stream Processing).
Table count multiplies envelope variety: every captured table is a distinct payload schema that has to be registered, versioned and validated (Schema Registry).
Consumer count is free at the envelope level and expensive at the interpretation level — every consumer that re-implements the collapse-to-current-state logic will implement it slightly differently (Model Layering).

What drives cost here

A full row image on every update means the stream carries the whole row for every change, so a wide table with a hot narrow column dominates retained bytes out of proportion to its business importance (What Actually Drives Data Platform Cost).
Carrying before roughly doubles the payload for updates. That is the price of being able to answer questions about previous values, and it should be a deliberate purchase per table rather than a global default.
Envelope metadata is small per event and large in aggregate at high change volume — and it is the last thing to strip, because it is the part that makes recovery possible.
Retained bytes downstream scale with change volume rather than table size, which is the opposite of the polling cost shape (Storage Lifecycle).

What this approach costs

Keeping the full envelope buys auditability, replayability and the ability to answer transition questions. It costs storage proportional to change volume and a modelling step that a plain table copy would not need.
Enabling full previous images buys delta and audit capability and costs payload size and a larger privacy surface — the previous value of a corrected field survives the correction.
Collapsing to current state on the way in is cheaper to query and destroys the history that made the stream worth capturing. Doing it in raw is the version of this trade that cannot be undone (Raw, Staging, Curated: Layers by Purpose).
A strict boundary schema catches reshapes loudly and will also reject valid data during an unannounced upstream migration, turning a quality problem into an availability one (Contract Enforcement).

CDC envelope explorer

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

CDC envelope explorer

A change record is not a row. It is a statement about a row: what it was, what it became, and where in the log that happened.

Operation

{
  "op": "u",
  "ts_ms": 1756166460000,
  "source": { "lsn": 84219, "table": "orders", "txid": 9913 },
  "before": { "id": 100731, "status": "placed", "amount": 4250 },
  "after":  { "id": 100731, "status": "shipped", "amount": 4250 }
}

Field	What it is for
op	Which of the three things happened. A consumer that ignores it and upserts every record silently resurrects deleted rows.
ts_ms	When the source committed it — the closest thing to an event time this record has. It is not the arrival time and the two will diverge.
source.lsn	The position in the source log. This, not the timestamp, is what orders two changes to the same row — timestamps tie and clocks move.
source.txid	The transaction. Two rows changed atomically at the source arrive as two records, and only this field says they belonged together.
before	The row as it was. Optional in most sources, and the option people discover they needed after a delete they cannot explain.
after	The row as it now is. Null for a delete.

A row changed. With both images a consumer can tell which columns changed, not merely that something did.

SOURCE-SPECIFICThe field names are Debezium-shaped. A Postgres WAL, a MySQL binlog and a Mongo oplog expose different amounts of this: whether a before image exists at all is a source setting (`REPLICA IDENTITY FULL`, `binlog_row_image`), not a connector feature.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALEvery log-based CDC system produces the same conceptual envelope — operation, optional before, optional after, source metadata — because that is what a change record in a transaction log contains. Field names, nesting and encoding differ completely between connectors.
SOURCE-SPECIFICAvailability of the previous row image is the sharpest difference in this module. Postgres logical replication supplies as much of the old row as the table's replica identity dictates — key columns only by default, the full row only when configured; MySQL row-based binlog carries both a before and an after image for updates as a property of the row format; MongoDB change streams deliver the change description plus optionally the post-image, with the pre-image available only where it has been enabled on the collection.
SIMPLIFIEDThe JSON shown is a teaching envelope, not any connector's literal wire format. Real payloads add schema descriptors, connector-specific nesting and encodings for decimals, dates and binary that differ enough to matter when you write a parser.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasemvcc update-delete-under-mvcc write-ahead-logging transactions-internally replication-internals

Backendat-least-once-delivery duplicate-detection idempotency-keys event-consumers

Architectureevent-sourcing cqrs

Securitydata-classification audit-logs

Domains that do not exist yet

— Distributed Systems owns what it means for two events to be ordered at all once they leave a single log, and why a per-key guard is the practical substitute for a global order. The position guard in this lesson is a local answer to a problem that domain treats properly.