When a Task Fails Mid-DAG
A succeeded, B succeeded, C failed. Re-run everything, only C, or C and everything downstream? The answer is decided entirely by idempotency.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A task fails in the middle of a graph that has already written data. What is safe to re-run, and how do you know the parts that succeeded are still correct?
Consumers who are, at this moment, reading tables that a half-finished run has already updated. They cannot see that the run is incomplete; they see a table with rows in it and assume the day is done (Atomic Publish).
The unit of the decision is the (task, interval) pair, and the unit of the blast radius is the set of pairs reachable downstream from the failure. Reasoning at the level of "the DAG" is what turns a one-partition repair into a full recompute.
Click retry on the failed task. If it goes green, the incident is over. This is right far more often than it is wrong, which is exactly why the cases where it is wrong are so damaging.
C appends rather than overwrites. Its first attempt wrote half the rows before failing; the retry writes all of them; the interval now holds one and a half copies and every measure is inflated (Duplicate Rows).
- C appends rather than overwrites. Its first attempt wrote half the rows before failing; the retry writes all of them; the interval now holds one and a half copies and every measure is inflated (Duplicate Rows).
- B succeeded and published, and C's failure means B's output was built from an input C was supposed to correct. B is green, its table is wrong, and nothing in the graph will revisit it.
- C succeeded on retry but read a source that has changed since A ran. The interval now mixes data captured at two different times, and no column records that (Reprocessing vs Retrying).
- The retry succeeded and the downstream tasks that were already marked as upstream-failed stay that way. Everything looks recovered; forty downstream tables were never rebuilt (Partial Failure).
- Someone re-runs the whole DAG "to be safe". A, which calls an external system with side effects, sends the notification a second time — and the expensive extract runs again against a source with a connection budget (Retry Storms: The Load You Generated Yourself).
What is actually happening
- A DAG is not a transaction. Each task commits its own effects independently, so a failure part-way leaves the graph in a partially applied state — some outputs at the new interval, some at the old, and no record connecting the two (Transactions and ACID).
- The orchestrator marks downstream tasks as blocked rather than failed, which is correct and misleading: those tasks have not failed, they have not run, and the two look different in a dashboard but identical in a table that was never refreshed.
- Whether re-running is safe is a property of each task, not of the platform. An overwrite of a partition keyed by the interval is safe to repeat any number of times; an append, an increment or an external call is not (Idempotent Data Pipelines).
- The blast radius is the downstream closure of the failed task in the dependency graph. Computing it is a graph traversal, which is why lineage is an incident-response tool and not documentation (Impact Analysis, DAG (Directed Acyclic Graph)).
- Tasks with side effects outside the data platform — emails, webhooks, payments, cache invalidations — are the hard part, because the graph can re-run them and the world cannot un-receive them (Idempotency Keys: The Mechanism).
- Retries are the same mechanism as manual re-runs, applied automatically and invisibly. Anything unsafe to re-run by hand is already happening automatically on every transient failure (Retries).
The three answers, and when each is right
A succeeded. B succeeded. C failed. Three responses are available and all three are defended in real incident channels: re-run only C, re-run C and everything downstream of it, or re-run the whole graph. There is no universally correct answer, and the criterion is not taste — it is whether each task replaces its interval or adds to it.
The reason this is worth treating as a formal decision is that the wrong choice is not merely inefficient. Re-running a graph containing a non-idempotent task duplicates data or repeats an external effect; re-running too little leaves published tables derived from inputs that have since changed. Both produce a second incident, and the second one is always harder to see than the first.
Work through the options in order. The first question is never "what do I re-run" — it is "which tasks in this graph are safe to re-run at all", and if the answer is not "all of them", the real fix is upstream of this incident (Idempotent Data Pipelines).
Which tasks already published, which already consumed C's output, and which of them replace their interval rather than adding to it?
when C is idempotent for its interval, nothing downstream of C has run yet, and C's inputs have not changed since A and B ran.
cost Cheapest and correct in the common case. Wrong whenever a downstream task already consumed a partial or stale version of C's output, which the orchestrator will not tell you.
when Downstream tasks have already run for this interval, or you cannot establish that they have not. Every task in the closure replaces its interval.
cost Costs the compute of the whole closure and the source load that comes with it. This is the default answer for a well-built graph and should feel routine rather than dramatic.
when The dependency structure is not trustworthy — undeclared edges, shared tables, tasks whose inputs you cannot enumerate — and every task is idempotent.
cost Maximum compute and maximum blast radius. Safe only under full idempotency, and it re-triggers any task with an external side effect unless those are excluded explicitly.
when Almost never. Reserved for a task whose effect cannot be re-executed at all and whose output you have verified independently.
cost Writes a permanent claim into the metadata store that the interval was processed. Every future coverage query and every future backfill decision will believe it.
when The source data for the interval is itself wrong, so no amount of re-running produces a correct result.
cost Consumers see a gap rather than a number, which is honest and unpopular. Requires a way to mark a partition as known-bad that downstream models actually check (Data Incidents).
What the failure has already done downstream
The word "downstream" is doing a lot of work in the decision above, and during an incident it needs to be a concrete list rather than a gesture. The lineage below is the ordinary shape: a failed staging task with a fact table, a mart and a dashboard beneath it, plus a second consumer nobody remembered.
Walk it in the direction of the data and ask one question at each node: what does this hold *right now*, for the affected interval, and could the failure have made it wrong? The answers differ. Some nodes never ran, and hold yesterday's state while presenting as current. Some ran before the failure and hold output built from an input that has since been replaced. Those two conditions look identical from outside and need opposite responses.
The last node is the one that turns a technical incident into an organisational one. A number that has already been read, quoted in a meeting, or exported into a spreadsheet is not fixed by re-running a DAG, and the recovery has a communication step whether or not anyone plans for it (Debugging a Data Incident).
- stg_orders (task C, failed)
holds A partial write of the interval, or nothing, depending on whether the task publishes atomically.
could corrupt Everything beneath it. If it appends, the retry adds a second partial copy on top of the first.
↑ reads from - fct_orders
holds Either yesterday's partition, because it never ran, or today's built from partial staging if it ran before the failure.
could corrupt Every measure for the interval. Presents as a complete, low day rather than as a missing one.
↑ reads from - revenue_daily mart
holds A pre-aggregated country-day row derived from whatever the fact table held when the mart last ran.
could corrupt Aggregates that will not be recomputed unless the mart is explicitly included in the clear.
↑ reads from - Executive dashboard
holds A rendered number with no indication of which interval its inputs actually cover.
could corrupt Trust, and only once — a dashboard that showed a wrong number is questioned for months afterwards (Trusting Data).
↑ reads from - Weekly finance export
holds A file already generated from the mart and already sent, in a different DAG that nobody listed.
could corrupt Something outside the platform entirely. Re-running the pipeline does not recall the file (Who Actually Consumes This Data).
↑ reads from - Feature pipeline for a model
holds Training rows read from the fact table on a schedule of its own.
could corrupt A model trained on an interval that was later corrected, with no record that the training data and the current table now disagree (Feature Pipelines).
The last two nodes are why the closure must come from lineage rather than from the DAG you are looking at. Both consume the affected data through a different graph, and neither will be cleared by any action taken inside this one.
The failures the retry button creates
Retrying is the correct first response often enough that it becomes reflexive, and the reflex is fine on a graph where every task replaces its interval. On any other graph, the retry button is a data-modification tool with no confirmation dialog.
These rows are the recurring ones. Read the cause column as a single theme: in every case the platform did exactly what it promised — another attempt at the same task — and the damage came from what the task does when attempted twice.
The response column is deliberately split between the incident fix and the structural fix, because the two are different work. Deduplicating a partition closes the incident; making the task overwrite its interval is what stops the incident recurring, and only the second one is progress (Deduplication).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Retry of a task that appends to its output | Row counts and every summed measure are inflated for that interval, by an amount nobody can predict | The first attempt wrote rows before failing; the second wrote the full set; nothing removed the first | Deduplicate the interval on the business key now; change the task to replace its interval rather than add to it (Upserts and Merges) |
| Retry after the source has changed | The repaired interval disagrees with its neighbours in ways no code change explains | The task reads a mutable source, so a re-run at a later time sees different rows for the same window | Reconcile against the source for the interval; extract from an immutable raw landing so re-runs are reproducible (Keeping Raw History: The Recovery Position and the Liability) |
| Full-DAG re-run to be safe | A notification is sent twice, or an export lands twice in a partner's system | A task with effects outside the data platform was included in the re-run | Compensate externally; move side-effecting tasks to the end of the graph behind idempotency keys (Idempotency Keys: The Mechanism) |
| Task marked successful to unblock the graph | Six weeks later a coverage query says the interval was processed and the partition is empty | The metadata store now holds a claim about the data that was never true | Clear and re-run the interval properly; treat mark-success as an action requiring the same review as a production data edit |
| Downstream cleared without the failed task | Downstream tables rebuild from the same bad input and look freshly wrong | The closure was cleared in the wrong order, or the upstream fix had not landed yet | Fix upstream first, then clear in dependency order; let the graph decide execution order rather than clearing tasks individually |
| Automatic retries on a non-idempotent task | Small, unexplained inflation on random days with no incident attached | Transient failures are being retried silently, and each one duplicates part of an interval | Audit retry counts against duplicate rates; either make the task idempotent or set its retry count to zero and page instead (Job Idempotency) |
How to build it
Most important first.
- Make the safe answer the default by making every task idempotent for its interval. Then "re-run C and everything downstream" is always correct and the decision below stops being a decision.
- Publish at the end of a task, atomically, so a failure leaves no partial output. A task that writes progressively is a task whose failure state has to be cleaned up by hand (Atomic Publish).
- Isolate external side effects into their own terminal tasks with their own idempotency keys, so re-running the data portion of a graph never re-sends anything (Webhook Idempotency).
- Record, on every output row or partition, the interval and the code version that produced it. During an incident that is what distinguishes "rebuilt" from "looks rebuilt" (Reprocessing vs Retrying).
- Write down the downstream closure before clearing anything. The instinct to fix quickly is what leaves forty tables built from data that no longer exists.
- Prefer clearing a bounded set of (task, interval) pairs over re-running a DAG. "Re-run the DAG" is how a fix for one day becomes a rewrite of a year (Planning a Backfill).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The orchestrator guarantees that downstream tasks did not start. It guarantees nothing about the outputs of tasks that did complete, which remain published and readable.
- A retry guarantees another attempt at the same interval. Whether that attempt is safe is a property of the code, and the orchestrator has no way to know.
- Marking a task as success after a manual fix guarantees only that the graph will proceed. It is a statement about the scheduler's state, not about the data, and it is the most dangerous button in any orchestrator.
- Nothing guarantees that a re-run reads the same source data as the original attempt. Sources move; an interval-bounded read against a mutable source is reproducible only if the source retains history (Keeping Raw History: The Recovery Position and the Liability).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that makes this survivable is a post-publish reconciliation per interval: for the affected window, compare row counts and a summed measure between the source of truth and each published table, and flag any table whose interval is present but divergent.
- It misses tables that were never written at all — absence reconciles to nothing — so it must be paired with an existence check over expected intervals (Freshness Checks).
- It also misses the case where a partial run wrote exactly the right count of wrong rows, which happens when a retry re-processes a different slice of a mutable source.
- A mid-graph failure produces a dataset that is stale while appearing current: the table exists, it has yesterday's rows, and nothing marks it as not-yet-updated for today.
- The correct signal is per-dataset freshness measured against the data, not the run. A task in a blocked state produces no alert of its own on most platforms (Freshness Monitoring).
- Recovery has its own latency. A downstream closure of forty tasks is forty tasks of runtime, so an incident found at 08:00 may not be resolved by the time the numbers are read (The Freshness SLO).
- If the failure was caused by a schema change upstream, re-running with unchanged code reproduces the failure and re-running with fixed code produces an interval whose rows differ structurally from its neighbours (Schema Evolution).
- A fix deployed mid-incident means the graph now contains two code versions across one interval range. Without a produced-by column that fact is unrecoverable later (Semantic Changes).
- Recovery procedures written for one graph shape stop being correct when the graph changes. The closure of a task is a property of today's edges, not of the runbook's (Data Lineage).
- Establish the closure first: which (task, interval) pairs are downstream of the failure, and which of those already ran and published before the failure occurred. That set, not the DAG, is the scope of the repair.
- For each task in the closure, answer one question: does re-running it replace its interval or add to it? Replace is safe to repeat; add must be made safe before anything is cleared (Upserts and Merges).
- Clear the failed task and its closure together, with a concurrency limit, and let the graph re-derive in dependency order. Clearing the failure alone leaves downstream tables built from replaced inputs (Partial Failure).
- Never mark a task successful to unblock a graph unless you have separately confirmed its output is correct. That action tells the platform a lie it will remember forever.
- If a task had external side effects, exclude it from the re-run and replay it deliberately with its idempotency key, or accept the duplicate and compensate (Idempotency in Backends).
- After the repair, reconcile the affected interval against the source before declaring the incident closed. A re-run that succeeded is not evidence, for exactly the reasons the rest of this domain keeps repeating (Reconciliation).
What can go wrong
- A retry duplicating rows because the task appends.
- A downstream table left stale because the closure was not cleared, so its numbers silently disagree with its inputs.
- A manual "mark success" hiding a task that never produced output, permanently.
- A full-DAG re-run re-triggering an external side effect such as a notification or an export (A Dead-Letter Queue Is a Workflow, Not a Bin).
- A re-run reading a source that has moved on, producing an interval that is internally inconsistent with its neighbours.
- The mitigation failing: an automated downstream clear that includes a task with external effects, so the safety feature is what sends the duplicate.
- "The retry worked, so we are fine." The retry proves the code can exit zero. Whether it left one copy of the data or two is a different question with a different query (Duplicate Rows).
- "Only C failed, so only C is affected." C's failure may mean B consumed an input that was never corrected, and everything downstream of C never ran at all.
- "Re-running everything is the safe option." It is the safe option only when everything is idempotent. When it is not, it is the option that maximises the number of duplicated effects.
- "The orchestrator will re-trigger downstream when I fix C." It will, for tasks it still considers pending in that run. Tasks already terminal — including ones marked upstream-failed on some platforms — may need explicit clearing (Partial Failure).
Operating it
- Counts of task instances in blocked or upstream-failed states, per interval. This is the population of work that will never run unless someone clears it (Pipeline Observability).
- A lineage query for the downstream closure of the failed task, produced during the incident rather than after it (Lineage Debugging).
- Per-dataset freshness, so a table stranded by a mid-graph failure raises its own alarm independently of the graph (Freshness Monitoring).
- A record of manual interventions — clears, marks, one-off runs — because the next incident's investigation begins with what was done during this one (Structured Logging: Fields a Program Can Read).
- At 10x graph size, the closure stops being something a human can enumerate and becomes a query. Teams without lineage tooling start under-scoping repairs at exactly this point (Impact Analysis).
- At 100x, cross-team closures dominate: the affected tables belong to people who do not know your DAG exists, and notification becomes part of recovery (Data Incidents).
- More intervals per day means more, smaller repairs. Hourly pipelines fail more often and recover more cheaply, which is usually the better trade (Incremental Processing).
- Re-running the closure costs the compute of every task in it. That is the price of correctness, and it is usually far smaller than the cost of a wrong number that survives a quarter.
- The avoidable cost is over-scoping: re-running a DAG when one interval of one branch was affected multiplies scan and write volume by the size of history (Compute Waste).
- The hidden cost is source load. Backfill and closure re-runs hit operational systems with a concurrency they never see during normal operation (Scan Cost).
- Making every task idempotent costs design effort on every task and buys a recovery procedure that is always the same. It is the cheapest insurance in the domain and it is paid up front, which is why it gets skipped.
- Clearing the full closure is correct and expensive; clearing only the failure is cheap and sometimes wrong. Choosing cheap requires knowing which downstream tasks already consumed the bad data, which is knowledge most teams do not have at 03:00.
- Isolating external side effects into terminal tasks makes graphs longer and slightly harder to read, in exchange for making full re-runs safe.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat a graph is not a transaction, and that partial application leaves inconsistent published state, is true of every orchestrator. What differs is how much help you get computing the downstream closure — some platforms clear it for you, others leave it as a manual traversal.
- TOOL-SPECIFICWhether clearing a task automatically clears its downstream is a per-tool behaviour and sometimes a per-action flag. Getting this wrong in either direction matters: clearing too little strands downstream tables, clearing too much re-triggers side effects, and the default is not the same everywhere.
- SCALE-SPECIFICIn a graph of ten tasks the closure is obvious by inspection and a runbook is unnecessary. Above roughly a hundred tasks with cross-team edges, the closure is only knowable from lineage metadata, and teams that under-scope repairs at that size do so because enumerating by hand stopped working.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the general form of this problem — partial failure across independent commit domains, and why no amount of retrying makes a multi-system update atomic. A DAG is that problem with a scheduler on top.
- — DevOps / Production Engineering owns the incident process around it: who declares, who communicates to consumers whose reports have already been read, and how a manual intervention during an incident is recorded so the next investigation can see it.