Reprocessing vs Retrying
The same button means two different things: finishing work that never completed, and redoing work that completed and is now wrong.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A task in yesterday's DAG is red and a task in March's DAG is green but wrong. Both are cleared and re-run with the same command. Why is only one of them safe?
The on-call engineer at 3 a.m., who has one action available and needs to know which of the two operations they are about to perform. Downstream of them, every consumer of the affected dataset, who experiences a retry as nothing at all and a reprocess as their numbers moving.
A retry operates on a task attempt — one execution of one unit of work that did not reach a successful end state. A reprocess operates on a published partition — a unit of work that did complete, whose output exists and is being read. Same code, same command, different object.
Treat "clear and re-run" as one operation with one risk profile, because the orchestrator presents it as one button and the runbook says "re-run the failed task". For a pipeline that writes idempotently this is even correct — the two operations genuinely are the same when re-running is a no-op.
A task failed after writing half its output and before recording success. The retry runs the whole unit again, and the half that already landed is now there twice (Partial Failure).
- A task failed after writing half its output and before recording success. The retry runs the whole unit again, and the half that already landed is now there twice (Partial Failure).
- The engineer clears a range of dates to "re-run the failed one" and clears twelve green ones alongside it, silently converting a retry into an unplanned, unvalidated, unannounced backfill of the twelve (Backfills).
- A retry of a March task reads the source as it is today rather than as it was in March, so the retry does not restore the missing output — it fabricates a different one (Snapshot Tables).
- The retry succeeds, the metric does not recover, and hours go into debugging the transformation. The input was never re-fetched: the failure was upstream and the retry re-ran a task whose source was already empty (Missing Rows).
- A reprocess is performed as a retry, so it inherits none of the backfill discipline — no staged output, no validation, no announcement — and consumers discover a restatement by noticing it.
- Automatic retries with a short delay fire against a source that is already struggling, and the pipeline becomes part of the outage rather than a victim of it (Retry Storms: The Load You Generated Yourself).
What is actually happening
- A retry re-executes work whose effect on the world is unknown or incomplete. Its safety is entirely a property of the write: if the operation is idempotent, retrying is free; if it appends, retrying duplicates. Nothing about the retry mechanism decides this (Idempotent Data Pipelines).
- A reprocess re-executes work whose effect on the world is known, complete and being used. Its risk is not duplication but restatement: the numbers people already acted on will change, correctly or otherwise (Planning a Backfill).
- Orchestrators conflate the two because from their perspective there is only one object — a task instance for a logical date — and only one verb. "Clear" means "make this eligible to run again", and the tool cannot know whether the previous run left an output behind (When a Task Fails Mid-DAG).
- The distinguishing question is not "did it fail" but "is there a correct published output for this unit right now". If no: retry, and the only concern is partial effects. If yes: reprocess, and every concern in What Backfills Break applies.
- Retries have their own second axis: whether the input can still be re-fetched. A retry that re-reads from a retained raw layer is deterministic; one that re-queries a live source is a different computation wearing the same name (Keeping Raw History: The Recovery Position and the Liability).
- The transformation code is identical in both cases, which is precisely why the distinction has to live somewhere else — in the plan, the runbook and the guard on the write path.
Two operations wearing the same button
The orchestrator offers one action: make this task instance eligible to run again. What that action means depends entirely on something the orchestrator cannot see — whether a correct output already exists for that unit.
The comparison below is not about being careful. It is about asking one question before acting, and letting the answer select which of two different disciplines applies. The question takes ten seconds and separates a routine repair from an unannounced restatement of published history.
The uncomfortable part is that the dangerous version is the one that feels safer. Re-running a task that succeeded feels lower risk than re-running one that failed, because nothing is broken. It is the opposite: the failed task has no output to damage.
Something is wrong with a dataset. Find the tasks that produced it, clear them, let them run. If the range is uncertain, clear generously — extra days will just be recomputed. Watch for green.
Ask: is there a correct published output for these units right now? If no — the task failed, or produced nothing — this is a **retry**: run it, then assert uniqueness on the partition in case the failed attempt wrote something first. If yes — the units are green and their output is wrong or stale — this is a **reprocess**: it goes through the backfill path, with a stated range, a staged output, validation and an announcement.
A retry's risk is a partial previous effect; a reprocess's risk is restating numbers people have already used. Those need different mitigations, and the orchestrator presents them identically because from its point of view a task instance is a task instance. "Clear generously" is the specific advice that turns a five-minute repair into an unplanned six-month backfill.
Which failure you actually have
Before deciding between the two operations, it is worth being precise about what failed, because several very different situations present as "the task is red" or "the numbers are wrong" and only some of them are fixed by running anything again.
The response column is the useful one. Two of these rows say do not retry — retrying a permanent error and retrying a task whose upstream produced nothing are both work that will fail again in exactly the same way, and the second one fails *successfully*, which is worse.
The last row is the one that turns into a reprocess. It is included here because it is the row people arrive at by way of the others: something looked wrong, a retry did not fix it, and the actual answer is a planned correction of history rather than another execution (Planning a Backfill).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The task raised on a transient fault — a connection reset, a throttled API, a pre-empted worker. | One red task, no output written, everything else green. | A transient dependency failure. The work is incomplete rather than wrong. | Retry, with backoff. If the write is not idempotent, check the partition for a partial previous effect first (Retries in Pipelines). |
| The task raised after writing part of its output. | A red task and a partition that already contains rows, usually fewer than expected. | The write was not atomic, so the failure landed between "some data written" and "run recorded as successful" (Atomic Publish). | Do not simply retry. Replace the partition rather than appending to it, or delete the partial output first — then run. |
| The task raised on a schema mismatch or a permission error. | Red, and red again on every attempt, with an identical message. | A permanent error. Nothing about running it again changes the condition (An Error Taxonomy Clients Can Branch On). | Stop retrying and fix the cause. Automatic retries here spend the attempt budget producing the same message and delay the human by exactly that long. |
| Everything is green and the metric is flat at zero for one period. | No failures anywhere. The partition exists and contains no rows, or rows with no values. | The upstream produced nothing, or a cast nulled the measure. The task processed what it was given, which was nothing (The Pipeline Succeeded. The Data Is Wrong.). | Retrying reproduces the same empty result. Walk upstream to the first hop with the expected data, fix there, and then reprocess the affected periods deliberately. |
| Everything is green and a range of history is wrong because the logic was wrong. | A metric that is confidently incorrect across months, with no operational signal anywhere. | The code did what it was told over the whole range. There is no failure to retry (Two Dashboards, Two Numbers). | This is a reprocess. It goes through the backfill path — range, staging, validation, atomic publish, announcement — and not through the retry button. |
Where the two paths diverge
Drawn as a flow, the decision is a single guard early on, and everything after it differs. The retry path is short: run, verify the partition, done. The reprocess path is the whole of Planning a Backfill, because reprocessing *is* a backfill — the only difference is that it was reached from an incident rather than from a plan.
The guard is worth building into tooling rather than leaving to judgement. A wrapper that refuses to run a historical unit when the target partition already contains rows, unless an explicit reprocess flag is passed, converts a discipline into a property of the system.
Note where the two paths rejoin: both end at a check on the affected partition. Whatever operation you performed, the last step is to look at what is now in the target, because that is the only thing either path was ever trying to change.
How to build it
Most important first.
- Make the write idempotent, and the distinction stops being dangerous — a retry becomes a no-op and a reprocess becomes a deliberate, validated operation rather than an accident (Upserts and Merges).
- Before clearing anything, ask what is currently published for those units. One query against the target, and it converts an ambiguous action into a known one.
- Bound what a clear can touch. An operator clearing one task should not be able to clear a quarter by dragging a selection, and most orchestrators will let you constrain that (Airflow Concepts).
- Separate the two in the runbook and in the vocabulary. "Retry the failed load" and "reprocess January to June" are different sentences that should trigger different checklists, and a team that uses one word for both will eventually do the wrong one.
- Bound automatic retries with capped attempts and exponential backoff with jitter, and stop retrying on errors that are not transient — a schema mismatch will fail identically on the fifth attempt (Retries in Pipelines).
- Make retries deterministic by pinning inputs to the logical period, so a retry of March reads March. Without that, a retry is only safe on the day the work originally ran.
- Record which one happened. A reprocess is a change to published history and belongs in the same record as a backfill; a retry usually does not (Data Incidents).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A retry guarantees another attempt. It does not guarantee the attempt has the same inputs, the same effect, or any effect at all, and it does not guarantee the previous attempt left nothing behind.
- A retry against an idempotent write guarantees convergence: however many attempts run, the target ends in the same state — under the same key-uniqueness assumption every merge rests on (Surrogate Keys).
- A reprocess guarantees that the output is recomputed from whatever inputs are current at the time it runs. Whether that reproduces history depends on those inputs being immutable, which is a design choice made much earlier.
- Neither guarantees the downstream datasets are consistent with the result. Propagation is separate in both cases and is more often forgotten after a retry, because a retry does not feel like a change (Impact Analysis).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that separates the two before acting: for the units about to be re-run, does a published output already exist, and how many rows does it have? A count against the target partition is one query and decides which discipline applies.
- After a retry, assert uniqueness on the business key for the affected partition. It is the direct detector for a retry that duplicated a partially-written output (Deduplication).
- It misses the case where the retry produced *different* correct-looking data because the inputs moved — the counts are fine, the values changed, and only a comparison against what was published before would show it (Validating a Backfill Before You Publish).
- A successful retry restores the dataset's normal freshness and nothing else changes. That is what makes it the cheap operation and why conflating the two is tempting.
- A reprocess has no freshness benefit at all — its output is old — while consuming the capacity that freshness depends on. A reprocess run during business hours degrades exactly the thing a retry protects (The Freshness SLO).
- Repeated automatic retries against a persistent failure keep a pipeline in a state that is neither succeeded nor failed for a long time, which delays the human decision and is worse for freshness than failing fast (Freshness Checks).
- A retry of an old task runs today's code against an old period. If the transformation changed since, the retry is a reprocess in disguise, and the output will not match its neighbours (Semantic Changes).
- This is the most common accidental restatement in the field: a task retried months later, with the current model, quietly producing a partition computed under different logic from the partitions either side of it.
- Pinning the code version to the logical date makes retries faithful and makes fixing an old bug impossible without an explicit reprocess — which is the correct division and one very few platforms implement.
- Recovering from a duplicating retry is the same operation as recovering from an unsafe backfill: restore the partition from a snapshot or rebuild it from raw (Rolling Back Data).
- Recovering from an accidental reprocess of a wide range is worse, because the previous version of every affected partition must be restored, and nobody recorded which ones were cleared. Orchestrator history is the only evidence, and it expires (Data Lineage).
- The general recovery position for both is the same: retained raw data, deterministic transformations and a snapshot-capable target. With all three, both operations are re-runnable; with none, neither is.
What can go wrong
- A retry that duplicates because the previous attempt wrote before failing (Duplicate Rows).
- A clear that selected far more units than intended and became an unplanned backfill.
- A retry that re-reads a mutable source and produces a different answer that looks like the original.
- Automatic retries hammering a struggling upstream and extending the outage (Retry Storms: The Load You Generated Yourself).
- A reprocess executed with retry discipline: no staging, no validation, no announcement.
- Retries capped and backed off correctly, still retrying a permanent error until the attempt budget is exhausted — the mitigation working exactly as designed and buying nothing (An Error Taxonomy Clients Can Branch On).
- "Re-running is always safe." It is safe when the write is idempotent, which is a property somebody had to build. In an append-mode pipeline, re-running is the mechanism by which data gets duplicated (What Backfills Break).
- "The task is green now, so the data is fixed." Green means this attempt completed. It says nothing about what previous attempts left behind, and nothing about whether the input was there this time (The Pipeline Succeeded. The Data Is Wrong.).
- "Clearing a date range is just a bigger retry." Clearing a range of successful runs is a backfill. It republishes history, and it should be planned as one (Planning a Backfill).
- "Automatic retries make the pipeline reliable." They make transient faults invisible, which is valuable, and they hide a growing failure rate until the day the attempt budget is not enough (Pipeline Reliability).
Operating it
- Attempt count per task instance, and the distribution of it over time. A pipeline that quietly succeeds on the third attempt every night is failing in a way nobody is looking at (Pipeline Observability).
- Row count for the affected partition before and after the action, which distinguishes a retry that repaired from a retry that duplicated in one number.
- An explicit log line naming the operation and its scope — retry of one unit, or reprocess of a range — because the orchestrator's own record does not distinguish them (Pipeline Metrics).
- At 10x task count, manual clearing stops being viable and retry policy has to be declarative — attempts, backoff and which error classes are retryable, per task rather than per platform.
- At 100x, the distinction has to be enforced by the system rather than by the operator, usually by making every write idempotent so that the dangerous version of the operation stops existing (Idempotent Data Pipelines).
- More consumers change the reprocess side only: a retry is invisible to them at any scale, and a reprocess is a communication event whose cost grows with the audience.
- A retry costs one more execution of one unit. It is the cheapest corrective action available and its cost is almost never the reason not to do it.
- A reprocess costs the range, and running it accidentally costs the range plus the restoration plus the investigation (Compute Waste).
- Automatic retries against a permanent failure cost attempts times the unit, spent producing the same error, and they consume concurrency that working pipelines need.
- Idempotent writes collapse the two operations into one safe one, and cost a more expensive write plus a key you must guarantee. It is the best trade in this module and it is not free.
- Pinning code to the logical date makes retries faithful and makes bug fixes non-retroactive, so every correction becomes an explicit reprocess. More ceremony, fewer accidental restatements.
- Fast automatic retries recover from transient faults without waking anyone and amplify load during a real outage. Backoff with jitter is the compromise, and it makes recovery slower on purpose (Without Jitter, Every Client That Failed Together Retries Together).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe distinction between finishing incomplete work and redoing completed work exists in every scheduling system. What varies is whether the tool exposes them as different verbs — almost none do, which is why the discipline has to live in the runbook.
- TOOL-SPECIFICAirflow exposes "clear" for both, so re-running a failed task and re-running six months of successful ones are the same gesture with a different selection. Dagster distinguishes a run retry from a backfill as separate concepts with separate UIs, which removes the accident but not the underlying idempotency requirement.
- SIMPLIFIEDTreating a unit of work as either wholly published or wholly absent is a teaching simplification. Real tasks fail midway and leave partial output, which is the case that makes retries dangerous in the first place and is covered in Partial Failure.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a retry is unavoidable in the first place: a client that does not receive a response cannot distinguish a lost request from a lost reply, so every at-least-once system is built on retries whose effects must be idempotent.