CDCGENERALBROKER-SPECIFICSOURCE-SPECIFIC

CDC Ordering and Transaction Boundaries

The source log has a total order and a transaction boundary. Publishing splits both, and every consumer that joins two tables inherits the consequences.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

One transaction updated orders and order_items together. Downstream, why can a consumer see one of them and not the other — and for how long?

Who needs this

Any consumer that reads more than one CDC-derived table and expects them to agree: a join in a staging model, a stream job enriching an order with its items, a search indexer building one document from two tables, a reconciliation that compares a parent total against the sum of its children (Stream Joins).

What one row is

Ordering is a property of a sequence, so the unit here is the pair: two events and the question of which came first. Per key, per table, per transaction and per partition are four different sequences with four different guarantees, and conflating any two of them is this lesson's entire failure surface.

The obvious build

Assume the order events arrive in is the order things happened. It is true inside a single partition of a single topic fed by a single connector, which describes most development environments perfectly, so the assumption survives every test you write and fails the first time production rebalances.

Why it breaks

A consumer group rebalances mid-stream. Two events for the same key are processed by two different consumers, the second finishes first, and a last-write-wins upsert leaves the older value in place permanently (Consumer Groups and the Parallelism Ceiling).

How it breaks with real data

A consumer group rebalances mid-stream. Two events for the same key are processed by two different consumers, the second finishes first, and a last-write-wins upsert leaves the older value in place permanently (Consumer Groups and the Parallelism Ceiling).
The partition count of a topic is increased. Keys are rehashed, a key that lived on partition 3 now lives on partition 7, and its future is ordered against a past it can no longer see (Topics and Partitions).
An order and its items are published to two different topics. The transformation joins them ten seconds later, the items have not landed, and the order is written to the fact table with a total of zero (Missing Rows).
A consumer parallelises for throughput and processes a partition's events across a worker pool. Per-partition order was the last guarantee standing, and it has just been discarded for a latency improvement nobody measured (Ordering Guarantees: Four Levels, Four Prices).
A retry sends an event a second time after a later event for the same key already landed, so the stream contains the sequence new, old — which is valid at-least-once behaviour and looks exactly like corruption.
A backfill republishes historical changes onto the same topic as live traffic, interleaving old positions with new ones and rewinding every key it touches (What Backfills Break).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

The source log is totally ordered. Every committed change has a position — LSN, binlog offset, oplog timestamp — and those positions define a single sequence across every table in that database. This is the strongest ordering guarantee anywhere in a data platform and it exists for free (Write-Ahead Logging).
The log also carries transaction boundaries. Changes that committed together are adjacent and share a transaction id, and a reader of the log can therefore see the atomic unit the database saw (A Transaction, Inside the Engine).
Publication destroys both, and it destroys them for a reason. A broker partitions to parallelise, and parallelism is precisely the absence of a total order. What remains is order within a partition, which for a key-partitioned topic means order per key (Kafka-Style Logs: Topics, Partitions, Offsets).
Per-key order is usually enough for state reconstruction and never enough for a join. Two keys in two tables land on two partitions, are consumed by two workers, and arrive at the join at two unrelated moments (What a CDC Event Contains).
The transaction boundary survives only as metadata. A transaction id in each event tells a consumer which events belonged together; nothing makes the consumer wait for the rest of them, and a consumer that does wait must buffer until it sees the boundary, which requires the connector to emit one (Stateful Stream Processing).
The practical substitute for global order is a position guard at the sink: apply an event only if its log position exceeds the position already recorded for that key. That converts an ordering requirement into a monotonicity check, which survives reordering, duplication and replay (Upserts and Merges). The one case it does not save you from is a key that changed partition, because the guard is per key and that key's *history* is now split across two partitions with no relative order at all (Event Keys and Partition Assignment).

What the log knows that the topic does not

SIMPLIFIEDLog positions are shown as short hex labels for readability and the interleaving is idealised. Real logs contain records for pages, indexes and background work that logical decoding filters out, and a single logical row change may span several physical records.

Start from the source. The log below is one transaction and two ordinary updates, written out as the database recorded them. Every line has a position, positions increase, and the two BEGIN/COMMIT markers say precisely which changes were atomic.

That is a remarkable amount of information, and it is all free — the database wrote it for crash recovery whether or not anyone was reading (Crash Recovery). A consumer of this log can answer "which happened first", "did these commit together" and "have I seen everything up to here" without any additional machinery.

Now look at what a consumer receives after publication. The positions survive, because a good connector puts them in the payload. The relative order between orders and order_items does not, because they went to different partitions. And the transaction boundary survives only as a shared txid that nothing acts on. Two of the three properties were lost in transport, and the third was downgraded from a mechanism to a label.

SOURCE LOG (totally ordered, transaction boundaries intact)

  0/1A2F4C10  BEGIN            txid=918273
  0/1A2F4C28    UPDATE orders       order_id=88124  status: paid -> refunded
  0/1A2F4C44    UPDATE order_items  item_id=51188   refunded: false -> true
  0/1A2F4C60    UPDATE order_items  item_id=51189   refunded: false -> true
  0/1A2F4C88  COMMIT           txid=918273
  0/1A2F4D02  UPDATE orders         order_id=88125  status: pending -> paid
  0/1A2F4D40  UPDATE orders         order_id=88124  note: NULL -> 'partial'

AFTER PUBLICATION (order survives only inside a partition)

  topic orders.p3      : 0/1A2F4C28 (88124)   0/1A2F4D40 (88124)
  topic orders.p7      : 0/1A2F4D02 (88125)
  topic order_items.p1 : 0/1A2F4C44 (51188)
  topic order_items.p5 : 0/1A2F4C60 (51189)

  within orders.p3            : ordered, same key, guaranteed
  between orders.p3 and .p7   : no order
  between orders and items    : no order, no boundary, no bound on the gap

Where the transaction goes

The diagram traces one atomic source transaction to the point where a consumer tries to reassemble it. Nothing here is a bug: every component is doing exactly what it was designed to do, and the atomicity is lost anyway.

This is worth stating plainly because it is often treated as a tooling deficiency to be fixed with configuration. It is not. Publishing an ordered stream into partitions is a deliberate exchange of ordering for parallelism, and any system that reassembled the transaction would be paying that parallelism back with latency and buffering (Distributed Transactions).

The consumer at the end has three honest options and no fourth. Tolerate the inconsistency and design for a missing counterpart. Buffer by transaction id and apply at the boundary, accepting the latency. Or stop joining streams and join materialised tables at a common position instead — which is what most platforms should do and few explicitly decide to.

One atomic transaction, four independent arrival paths

UserLLMAgentToolDataDecisionHumanGuardrail

Four sequences, four different promises

BROKER-SPECIFICThe per-partition row assumes a broker that orders within a partition at all, which Kafka and Kinesis do and which Pub/Sub does only when an ordering key is configured. On a broker with no ordering primitive, rows three and five collapse to "nowhere" and the position guard at the sink becomes the only ordering mechanism in the system.

Most ordering arguments are really vocabulary failures: two people say "ordering" and mean two different sequences. The table separates them, and the useful column is the last one — what a consumer may actually assume.

The row to internalise is the third. Per-key ordering is the guarantee people believe they have, and it holds only while the key stays on one partition. Partition counts are changed for throughput, by people optimising a different metric, and the effect on ordering is invisible until a key that rehashed is updated twice (Event Keys and Partition Assignment).

The fourth row is the one nobody has. Cross-table ordering does not exist after publication under any configuration, and every design that assumes it is a design that will produce a state the database never held (Trusting Data).

Sequence	Where it exists	What destroys it	What a consumer may assume
Global commit order	The source log only. One position sequence across every table in the database.	Publication into more than one partition, and any second source shard.	Nothing after the connector. Available in the payload as a position, usable for comparison but not for arrival order.
Transaction atomicity	The source log, as explicit boundaries between shared-txid changes.	Publication. The id survives as metadata; the boundary does not.	Only what a consumer reconstructs by buffering on txid until it sees the boundary.
Per-key order	One partition of a key-partitioned topic.	A partition count change that rehashes the key; a consumer that parallelises inside a partition; a republished backfill.	Order for that key, provided the key has not moved partition and the consumer processes the partition serially.
Cross-table order	Nowhere after publication.	It never survives — two tables are two topics with independent lag.	Nothing. Two CDC streams may be arbitrarily far apart, and the gap is unbounded during an incident.
Per-partition order	Every partition, always.	Only a consumer that processes one partition concurrently.	The one guarantee the broker genuinely makes. Everything else in this table is derived from it or lost.

How to build it

Most important first.

Partition by the source primary key of the captured table, not by anything derived. It is the only key for which per-partition order is the same thing as per-entity order (Event Keys and Partition Assignment).
Carry the log position in every event and guard every write with it. This is the highest-value single line of code in a CDC pipeline and it costs one column (What a CDC Event Contains).
Never join two CDC streams and expect consistency at a point in time. Join CDC-derived tables after both have been materialised to a common position, or accept eventual consistency explicitly and design the consumer to tolerate a missing counterpart (Eventual Consistency in Practice).
If a consumer genuinely needs transactional atomicity, buffer by transaction id and apply at the commit boundary — and size the buffer for the largest bulk statement your source can produce, because that is what will overflow it.
Treat a partition count change as a migration, not a config tweak. Drain, stop, repartition and re-materialise, or accept that every key touched has lost its ordering against its own history (Reprocessing vs Retrying).
Publish a single "materialised as of position P" marker per table so downstream models can join tables at a common position rather than at a common wall clock (Atomic Publish).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

CDC guarantees ordering by source log position, at-least-once, for every committed change — a total order at the point of capture and nowhere after it.
After publication, ordering is guaranteed per partition only. For a key-partitioned topic that means per key, and only while the key stays on the same partition.
There is no ordering across partitions, no ordering across topics, and therefore no ordering between two tables of the same source once they are published separately (Topics and Partitions).
Transactional atomicity is not preserved downstream. The transaction id survives as metadata; the atomic application does not, unless a consumer implements it.
Nothing deduplicates, so a stream may legitimately present the sequence new, old, new for one key after a retry. Order-insensitivity at the sink is required, not optional (Deduplication).
A consumer that processes a single partition with a worker pool has no ordering guarantee at all and has given up the only one the broker offered.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

Assert monotonicity: for every key, the applied log position never decreases. This catches out-of-order application from rebalances, parallel consumers and republished backfills in one check (Data Tests).
Assert referential agreement across the pair of tables at a common position: every order in the fact table has its items, and the sum of items matches the parent total (Reconciliation).
The monotonicity check misses gaps entirely — a sequence with a hole in it is still non-decreasing, so completeness needs a separate check (CDC Failure Modes and the Retention Deadline).
The referential check misses the case where both tables are consistently stale, and it produces false alarms during any period where one stream legitimately lags the other, which is why it belongs on closed periods only.

Freshness

Ordering and freshness pull against each other directly. Buffering to restore transaction boundaries adds latency exactly equal to the time to see the boundary, and that time is unbounded for a long transaction.
The freshness statement a CDC-derived table can honestly make is positional, not temporal: "complete as of log position P". A wall-clock freshness claim over an unordered multi-table join is not meaningful (The Freshness SLO).
Two tables materialised to different positions are not merely stale relative to each other — they are inconsistent, and a join between them can produce a state the source never held (Trusting Data).
The gap between the two positions is the real freshness metric for a join, and almost nobody measures it (Pipeline Metrics).

When the schema or meaning changes

Changing the partition key is the most disruptive change available here. Every key's history splits across the old and new partitions and no ordering exists between the halves (Event Keys and Partition Assignment).
Increasing partition count has the same effect for whichever keys rehash, and it is usually done for throughput by someone who does not know it is a data-consistency change (Kafka-Style Logs: Topics, Partitions, Offsets).
Adding a table to capture midway means its stream starts at a different position from its siblings, so any join involving it is inconsistent until it has been snapshotted and caught up (Snapshot and Stream: the Bootstrap Problem).
A source change that alters the primary key — a natural key replaced by a surrogate — changes what "the same entity" means to the partitioner, and every guard keyed on the old value stops guarding anything (Surrogate Keys).

How to re-run this safely

Recovery from an ordering incident is a re-materialisation, not a repair. Replay the change history for the affected keys with the position guard applied correctly, and the table converges to the right state (Replay from the Log).
This is only possible if the change history was retained with positions. A pipeline that collapsed to current state on ingest has thrown away the evidence needed to fix an ordering bug (Keeping Raw History: The Recovery Position and the Liability).
Never repair by copying current state from the source for the affected keys. It works, and it leaves the table at a position that does not correspond to any point in the stream, so the next replay disagrees with it.
After a partition count change, the only correct recovery is a re-materialisation of every affected key from a snapshot plus the stream, because the ordering information across the split no longer exists anywhere (Snapshot and Stream: the Bootstrap Problem).

What can go wrong

Failure modes

Last-write-wins by arrival time, applied to a stream that is only ordered per partition. It works in every test and fails on the first rebalance.
A join between two CDC-derived tables that produces a state the source never had — an order with a total, and no items.
A consumer that parallelises within a partition, discarding the only ordering guarantee it had, for a throughput gain that was not the bottleneck.
A backfill republished onto the live topic, rewinding every key it touches (What Backfills Break).
The mitigation fails too: a position guard implemented with a string comparison on a hex log position, which orders 0/9F after 0/1A2F and silently drops changes.
Transaction buffering that works until a bulk statement produces a transaction larger than the buffer, at which point the consumer either fails or silently splits the transaction it existed to keep together.

Misreads

"Kafka guarantees ordering." It guarantees ordering within a partition. Across partitions there is none, and a key that moved partition has lost order against its own history (Topics and Partitions).
"The events are timestamped, so we can sort them." Commit timestamps from one source are consistent, and timestamps are still coarser than log positions — two changes committing inside the same millisecond are ordered by the log and not by the clock.
"We use the transaction id, so we have atomicity." The transaction id tells you which events belonged together. Applying them together is a consumer implementation, and metadata is not a mechanism.
"Adding partitions is a throughput change." It is a data-consistency change with a throughput benefit, and it deserves the review a schema migration gets (Schema Migrations from the Application Side).
"Our join is only inconsistent for a moment." The moment is unbounded. It is the difference in lag between two independent streams, which during an incident is hours (CDC Failure Modes and the Retention Deadline).

Operating it

How you see it in production

Count of position regressions per table: how often an event arrived whose position was below the one already applied for its key. A non-zero, steady value is normal after retries; a spike is a rebalance or a republished backfill (Pipeline Metrics).
The position gap between paired tables — orders versus order items — as a single chart. It is the only direct measure of how inconsistent a join across them currently is.
Consumer group rebalance events, correlated against downstream anomaly timestamps. Most out-of-order incidents line up exactly (Consumer Groups and the Parallelism Ceiling).
Partition-level lag rather than aggregate lag: one stuck partition inside a healthy total is invisible in the average and is precisely the case that reorders a subset of keys (The Backlog Arithmetic: Four Levers and a Drain Time).

What changes at 10x and 100x

At 10x throughput the pressure is to add partitions, and adding partitions is the operation that breaks per-key ordering. This is the single most common way a scaling change becomes a correctness incident (Topics and Partitions).
At 100x, per-transaction buffering is no longer viable and consumers must be designed to tolerate partial transactions permanently rather than to reassemble them.
Table count multiplies the pairwise consistency problem quadratically: three tables have three pairs to reason about, ten have forty-five, and nobody reasons about forty-five (Model Layering).
Multiple source shards remove global order entirely — there is no cross-shard log position — so any question spanning shards is answerable only after materialisation (Partitioning and Sharding).

What drives cost here

Preserving ordering costs parallelism. A single partition per source table is perfectly ordered and caps throughput at one consumer, which is the trade in its purest form.
Transaction buffering costs memory proportional to the largest transaction and latency proportional to its duration — both of which are set by the source's workload, not by your consumer.
The position guard costs one column and one comparison per write, and is the cheapest correctness mechanism in the whole domain (What Actually Drives Data Platform Cost).
Re-materialising after a partition change costs a full snapshot plus catch-up, which is why partition counts should be chosen with more care than they usually are (Compute Waste).

What this approach costs

Per-key ordering with a position guard buys correctness under reordering and replay for the price of one column, and buys nothing at all for cross-table consistency.
Transaction buffering buys downstream atomicity and costs latency, memory and a hard failure mode on large transactions. It is the right choice for a small number of consumers that genuinely need it and the wrong default.
A single partition per table buys perfect order and caps throughput. That is a legitimate design for a low-volume table with a strict consumer and an unacceptable one for a hot fact source.
Materialising both tables to a common position before joining buys consistency and costs freshness — the join is now as fresh as the slower stream, which is the honest answer rather than a regression (Cost vs Freshness).

CDC ordering lab

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

CDC ordering lab

Brokers order records within a partition. Nothing orders them across partitions — so where a change lands decides whether the final state is right.

Partitions3Lag on the last lane2 ticks

Rows

Rows in the wrong final state

1sim

Partitioning

round-robin

Guarantee

per-partition only

partition 0

lsn 1 · A → placed

lsn 4 · C → placed

lsn 7 · C → shipped

partition 1

lsn 2 · B → placed

lsn 5 · A → delivered

lsn 8 · A → returned

partition 2

lsn 3 · A → shipped

lsn 6 · B → cancelled

Row	Final state after applying	Truth at the source
A	shipped	returned (lsn 8)
B	cancelled	cancelled (lsn 6)
C	shipped	shipped (lsn 7)

1 row settled on a state the source never ended in. Nothing was lost and nothing errored: the changes were simply applied in an order the source did not perform them in. Partitioning by the primary key puts every change to a row in one partition, which is the only thing that makes "last write wins" mean the same thing at both ends.

SIMULATEDEight changes to three rows, drained by one consumer per partition with one lane running behind. The lag is a declared parameter, not a measurement; what it demonstrates is that any lag at all is enough.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThat a total order at a single log is reduced to a per-partition order by any parallel transport is a property of parallelism itself, not of a product. Any system that fans one ordered stream into many independent ones has this behaviour.
BROKER-SPECIFICKafka orders per partition, so a key that rehashes after a partition count increase loses order against its own history; Kinesis orders per shard, and a shard split has the same effect; Google Pub/Sub gives no ordering at all unless an ordering key is set, and enabling it changes the delivery behaviour of the subscription. A design that assumes one of these on another is wrong in a different way each time.
SOURCE-SPECIFICA single-node source has one totally ordered log; a sharded or multi-primary source has one per shard with no relation between them, so cross-shard ordering does not exist to be preserved. Some sources also expose commit order only at transaction granularity, so intra-transaction ordering between two rows may not be recoverable at all.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasewrite-ahead-logging transactions-internally crash-recovery replication-internals partitioning-and-sharding

Backendeventual-consistency-in-backends webhook-retries-and-ordering at-least-once-delivery

Architecturekafka-partitions distributed-transactions saga-pattern

Observabilityqueue-backlog

Concurrencyordering-guarantees happens-before deterministic-replay

Distributed Systemsordering-models total-order-broadcast clock-skew

Domains that do not exist yet

— Distributed Systems owns the real depth here: total versus partial order, logical and vector clocks, why a global order across independent logs requires consensus, and what that consensus costs in latency and availability. This lesson deliberately stops at the practical substitute — a monotonic position guard per key — and that domain should be linked from every guarantee above once it exists.
— DevOps / Production Engineering owns the change-management side: a partition count increase is a production change with data-consistency consequences and belongs in the same review process as a schema migration, not in a capacity ticket.