SIMULATED

Backfill Lab

The revenue model has excluded refunds incorrectly for six months. Fixing the code fixes tomorrow — six months of published history is still wrong, and repairing it is a write into production data that a great many teams get wrong in the same way.

SIMULATEDEverything numeric on this page

Ranges, partition counts and validation outcomes below are produced by the pipeline model in this repository. They exist to make the sequence concrete, not to tell you how long a real backfill takes — which depends on your data, your engine and how much of it you can afford to recompute at once.

The scenario

The revenue model has excluded refunds incorrectly for six months.

What it forces
Fixing the code fixes tomorrow. Six months of published history is still wrong.
What you have to decide before touching anything
Which range, run where, validated against what, and published how — without disturbing the periods the bug did not touch?

The lab

Choose the range, recompute, validate and publish. Then check what happened to the partition that was already correct.

Backfill lab — re-running a range
Backfill lab — re-running a range
A backfill is a re-run, and a re-run is only a repair if the pipeline is idempotent. Most are not, and nothing tells you.
Write strategy
Copies of the period
2sim
Rows in the serving table
8,000sim
Reported total
1,868,997.80 minor unitssim
True total
934,498.90 minor unitssim
The range was re-run 1 time and appended each time, so the period is present 2 times over. Every additive measure is multiplied by 2. No job failed, no alert fired, and the number went up — which is the direction nobody questions.
PropertyAppendMerge on keyReplace partition
Idempotentnoyesyes
Needs a unique keyNoYes — and a key that is nearly unique is worse than none, because it merges rows that were different.No, but it needs the partition boundary to match the backfill range exactly.
Readers during the runSee a growing, double-counted table.See consistent rows if the merge is atomic; a table format decides whether it is.See an empty partition between the delete and the write, unless the format supports an atomic swap.
CostCheapest write, most expensive mistake.Rewrites matched files; more work than an append.Rewrites the whole partition even for one changed row.
FAILUniqueness4000 order ids appear more than once.
FAILReconciliationServing table reports 1,868,997.80 against a source total of 934,498.90 (minor units).
The question to ask before every backfill is not "will it work" but "if I run it twice, is the table the same". If the answer is no, the operation is not a repair — it is a second copy of a period that was already correct.
SIMULATEDRow counts and totals come from the pipeline model's unsafe-backfill path, which is asserted fault-by-fault in the domain's test suite. Amounts are minor units in a model with no currency.

The sequence

Eight steps, and the order is the argument. Each one exists to prevent a specific way the previous one goes wrong.

  1. Why this step, in this position
    It stops the bug producing new wrong history while you deal with the old, and it gives you a period computed by the new logic to validate the backfill against.

The way this goes wrong

A real backfill, run without that sequence. Read the symptom and decide what you would do before opening the trap.

The six-month revenue backfill runs against production during business hours.

What you see
Dashboards are slow, and the current day's partition briefly contains data from March.
What is really happening
The backfill shared compute with live queries and wrote into the live partition rather than replacing a bounded range.