Reconciliation Is a Component, Not a Cleanup Script

Every mechanism that keeps a derived store in step with its source can fail without raising an error, and there are at least six such mechanisms in a typical system.

A missed event: the consumer was down past the stream’s retention, or a message went to a dead-letter queue and was never replayed. A bug in the projection: it computed the wrong value for a class of inputs, consistently and successfully. A partial failure: the source committed and the propagation did not, or the propagation succeeded and the source rolled back. A manual write: an operator fixed something in the derived store during an incident. A duplicate: a non-idempotent update applied twice. An out-of-order delivery: two updates to one key applied in the wrong sequence, leaving the older value.

None of these produces an error, and there is no rate at which they stop happening. Which gives the design conclusion: since drift is inevitable, the repair mechanism is part of the system, not a response to a bug. A system with derived state and no reconciliation is not a system that has avoided drift; it is a system that has not measured it.

Compare. Read both stores and identify the differences. For small datasets, a full comparison. For large ones, compare cheap summaries first — a checksum or count per key range, refining only into ranges that disagree, which is the same idea as [[merkle-trees]] and [[anti-entropy]] applied to a service boundary rather than to replicas.

Classify. Not every difference is drift. A record written two seconds ago and not yet projected is expected lag, not an error. Reconciliation must know the acceptable staleness window and exclude anything inside it, or it produces constant false positives and gets muted — which is the most common way a reconciliation job dies.

Repair. Rewrite the derived value from the source. Repair should be idempotent and rate-limited: a job that discovers two million discrepancies and fixes them all at once will saturate the store it is repairing, and a repair job that causes an incident does not get to run again.

Alert on the delta. This is the step teams skip and it is the most valuable. The size of the delta is a health signal for the whole derivation pipeline. A steady small number is normal. A step change means something broke. A slow rise means a systematic bug is accumulating. Repairing silently hides the failure it is repairing, so a reconciliation job that fixes without reporting is a system that has learned to conceal its own defects.

Treating every discrepancy identically is a common and expensive mistake, because the three shapes have different causes and different correct responses.

Missing in the derived store. The source has it, the derivation does not. Usually a dropped or unprocessed event. The repair is to write it, and the diagnostic question is whether the event was lost or merely never consumed — the answer determines whether this recurs.

Divergent values. Both have the record, the values differ. This is the interesting case: it points at a projection bug, an out-of-order application, or a second writer. Repairing it without asking which is how a projection bug survives for months, quietly repaired every night.

Orphan in the derived store. The derived store has a record the source does not. Either a delete was not propagated, or — much more importantly — the derived store holds data that exists nowhere else, in which case it is not derived at all and deleting the orphan destroys the only copy. An orphan is the signal that your ownership map is wrong, and it should be surfaced rather than automatically cleaned up.

The objection is predictable: "if the pipeline were correct we would not need this." The answer is that the pipeline is a distributed system, and every distributed system loses messages, duplicates them and reorders them under conditions you do not control. Reconciliation is not compensation for a poor implementation; it is the mechanism that makes an at-least-once, eventually-consistent derivation *operable* — the same relationship that a filesystem check has to a journalling filesystem, or that an inventory count has to a stock system that is otherwise perfectly good.

Design decisions worth making explicitly. Scope: full sweep or incremental over a recent window? Incremental is cheap enough to run every few minutes and catches the common cases; full sweeps are expensive and catch old drift, so run both at different cadences. Load: reconciliation reads both stores, which is real traffic — run it off replicas and rate-limit it. Authority: it must be unambiguous which side wins, or the job has no defined behaviour; if you cannot say, the problem is [[source-of-truth]], not reconciliation. Ownership: a job nobody owns gets muted the first time it is noisy and is never re-enabled.

Two implementation notes that matter more than they look. Comparing a moving system produces false positives from records that changed *during* the comparison; snapshot both sides at a consistent point, or compare with a window and tolerate churn. And the job must be safe to run twice at once, because it will be.

This is where the module closes back on [[detect-contain-recover]]. Reconciliation is the fourth step of that spine, and having it as a standing component is what makes the step executable rather than aspirational. During an incident you do not want to be writing a comparison script; you want to run the existing one bounded to the incident window.

That is the practical argument for building it before you need it, and it is stronger than the correctness argument. A reconciliation job written under pressure, against stores that are still recovering, by an engineer who has been awake for six hours, is exactly the tool most likely to make the incident worse — a repair with no rate limit, no staleness window and no dry run.

The mature version has a bounded mode: reconcile only entities modified between these two timestamps, report first, repair on confirmation. That single capability converts the vaguest step of incident response into a command someone can run.