Partial Failure
Ninety-eight partitions succeeded and two failed. Re-running only the two is right — but only if the unit is idempotent and independently publishable, and most people check neither before doing it.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A run processes a hundred partitions and two of them fail. What is the correct next action, and what has to be true for it to be safe?
Any reader of a dataset that is built in pieces. They see a table that is present, queryable and quietly missing two days — which reads as two unusually quiet days rather than as a fault, because nothing about a partitioned table announces which partitions are absent (Missing Rows).
The unit of failure and the unit of repair must be the same thing, and that thing is the independently publishable unit: a partition-day, a source table, a tenant, a file. If the run's unit of work is finer than its unit of publish — a hundred tasks writing into one table that is swapped once at the end — then there is no such thing as partial success and the whole run is the unit (Atomic Publish).
Treat the run as the unit. If any task failed, the run failed; re-run the run. It is simple, it is what a task-level orchestrator naturally expresses, and for a job with five tasks and a ten-minute runtime it is completely correct — the cost of redoing the successful work is smaller than the cost of reasoning about which work that was.
The run takes six hours and the failure was in hour five. Re-running from the start burns another five hours redoing work that was already correct, and the freshness SLO is gone regardless of whether the second attempt succeeds (Pipeline SLOs).
- The run takes six hours and the failure was in hour five. Re-running from the start burns another five hours redoing work that was already correct, and the freshness SLO is gone regardless of whether the second attempt succeeds (Pipeline SLOs).
- The ninety-eight successful partitions are re-processed against a source that has changed since the first attempt, so partitions that were correct yesterday are now different — and nobody asked for that (What Backfills Break).
- One of the re-processed tasks appends rather than replaces, so the ninety-eight correct partitions now hold duplicates and the incident is larger than the failure that caused it (Duplicate Rows).
- The operator instead re-runs only the two failed partitions, and one of them depended on a shared intermediate table that the successful run overwrote in the meantime. The repair produces a partition computed against different inputs from its neighbours (Reprocessing vs Retrying).
- The two failures are marked as skipped so the DAG can go green, and the missing partitions are never mentioned again. Six weeks later a quarterly report is short by two days and the run history is long gone (Data Incidents).
- A downstream job triggered on the parent's success runs against the ninety-eight, materialises a mart with two days missing, and is not re-triggered when the repair lands (The Transformation DAG).
What is actually happening
- Distributed work fails partially by default. With enough units, the probability that all of them succeed in one attempt goes to zero, so "the run failed" stops being a useful statement and "these units are not published" becomes the only one that carries information (Straggler Tasks).
- The reason a targeted repair is usually correct is that the units are independent: partition-day D is computed from inputs scoped to D, and nothing about D depends on D-1 having been computed in the same run. Where that independence holds, re-running two units is exactly equivalent to having succeeded on the first attempt.
- Where it does not hold, the targeted repair is a trap. A cumulative model — a running balance, a slowly changing dimension, a sessionisation that carries state across days — makes unit N depend on unit N-1, so repairing a gap in the middle leaves everything after it computed from a version of history that no longer exists (SCD Type 2 in Practice).
- The second precondition is idempotency over the unit. A repair is a second execution; if the task appends, or increments, or inserts without replacing, the second execution is not a repair but an addition (Idempotent Data Pipelines).
- The third is that the unit is independently publishable. If all hundred partitions are staged and swapped together, then partial success is not observable, the repair is a re-run of the whole build, and that is the correct behaviour rather than a limitation — it is what atomicity across the set means.
- What makes this hard operationally is that the orchestrator usually models tasks, not units. A task that loops over a hundred partitions internally is one task, so it is one status, and the information about which ninety-eight succeeded exists only in the logs. Getting that information into a durable, queryable place is most of the work (When a Task Fails Mid-DAG).
Ninety-eight succeeded. What does the repair actually touch?
The practical question is not philosophical: when the operator runs the repair, which paths get read and written, and which are left alone? The layout below answers it for a partitioned fact table where two partition-days failed.
The contrast worth holding is with the default action. A full re-run reads every partition of the source for the period, recomputes a hundred outputs, and rewrites a hundred partitions — including ninety-eight that already held the right answer and will now hold a *newly computed* right answer, which is not the same thing if the source has moved.
A targeted repair reads two source partitions and writes two output partitions. Everything else is untouched, which means everything else is also unchanged — the property that makes the repair trustworthy rather than merely fast.
- raw/orders/dt=2026-08-15/one day of change records · 12 files · skipped
- raw/orders/dt=2026-08-16/one day of change records · 11 files · skipped
- raw/orders/dt=2026-08-17/one day of change records · 14 files · read
- raw/orders/dt=2026-08-18/one day of change records · 13 files · skipped
- raw/orders/dt=2026-08-23/one day of change records · 15 files · read
- raw/orders/dt=2026-08-24/one day of change records · 12 files · skipped
- dim/customers/current/current dimension snapshot · 3 files · read
The last row is the one to argue about. A repair that joins against a *current* dimension produces two units built on a newer world than the ninety-eight around them. Where that matters, the dimension needs history and the join needs to be as-of the unit's date rather than as-of now.
Choosing the response, with the preconditions stated
There are four honest responses to a partial failure and the wrong one is chosen most often because it is the button the orchestrator puts in front of you. The criteria below are what separate them, and none of them is about how urgent the incident feels.
The first question is always whether the units are independent, because it removes options rather than ranking them. For a cumulative model, the targeted repair is simply not available, and pretending otherwise produces a table that reconciles per unit and is wrong in aggregate.
The second is whether the source is still what it was. If the raw inputs are immutable and retained, a repair reproduces what the first attempt would have produced. If the pipeline reads a mutable operational table directly, the repair produces something new, and whether that is acceptable is a question for the consumer rather than for the engineer (Source of Truth).
Are the units independent, are the tasks idempotent, and is the source still in the state the successful units were built from?
when Units are independent, the task replaces rather than appends, and the inputs for those units are immutable and available.
cost Requires per-unit status and a parameterised repair path. Produces a table whose units were computed at different times, which is fine when inputs are immutable and subtly wrong when they are not.
when Units are sequentially dependent — running balances, slowly changing dimensions, sessionisation carrying state across days.
cost Recomputes everything after the gap, so the cost grows with how old the failure is. It is the only correct option for cumulative models and it is why detecting these failures quickly matters more there (Backfills).
when Units are cheap, the run is short, or there is no per-unit status to target with — and the source is immutable so recomputing correct units changes nothing.
cost Redundant compute proportional to the successful fraction, plus the risk of recomputing correct units against a source that has since moved. Perfectly defensible for a ten-minute job; indefensible for a six-hour one.
when Consumers cannot tolerate a partially complete set — a financial close, a regulatory extract, a dataset whose whole point is a period total.
cost One failed unit blocks the entire dataset, so freshness is hostage to the least reliable unit. Buys a set that is never partially visible (Atomic Publish).
when Almost never. Defensible only when the units are genuinely optional — a non-critical enrichment source — and the omission is recorded and visible to consumers.
cost Silently incomplete data with a green pipeline. If chosen, it must write the omission into the publish ledger and surface it, or it is indistinguishable from a bug.
One task loops over a hundred partitions. It fails on the seventeenth. The orchestrator retries the task, which starts again at the first partition and reprocesses sixteen that already succeeded before reaching the one that did not.
The task expands into a hundred unit instances at runtime, each with its own status and its own retry policy. Two fail; two are retried; ninety-eight are never touched again.
A retry can only be as precise as the status it is retrying. When a hundred units share one status, the orchestrator has no way to express "these two" and the operator has no way to ask for it — so the granularity of the failure record, not the size of the data, is what decides whether targeted repair is possible at all.
The ledger that makes the repair a query
Everything above depends on being able to answer one question quickly and durably: which units are published, and how many times. Without a durable answer, the repair begins with an archaeology exercise across orchestrator logs that may already have rotated.
The ledger is deliberately boring — a small table written by the publish step itself, in the same transaction as the publish where the engine allows it. Writing it separately reintroduces a smaller version of the same dual-write problem, which for a bookkeeping table is usually an acceptable trade as long as the ledger is periodically reconciled against what is actually in the data.
The three queries below are the operational value. The first finds the gap. The second finds units published more than once, which is how a non-idempotent repair announces itself. The third finds units that are present in the ledger and empty in the data, which is the failure that neither presence nor uniqueness catches.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Repair run against a cumulative model. | The repaired unit is correct; every unit after it disagrees with it. | Unit N was computed from unit N-1, and N-1 changed after they were built. | Recompute forward from the earliest affected unit. Detect the dependency by asking whether the transformation reads its own output (Snapshot Tables). |
| Repair task appends instead of replacing. | The repaired unit now holds two of everything, and reconciliation fails on a unit that was previously merely absent. | Idempotency was never verified for the repair path, which is exercised far less often than the normal one. | Make replace-the-unit the only write mode the job supports, so the repair path and the normal path are the same code (Idempotent Data Pipelines). |
| Repair joins against a dimension that has since changed. | Two days of history attribute orders to a customer segment that did not exist on those days. | The join was as-of-now rather than as-of-the-unit. | Use a versioned dimension and join on the unit's effective date (SCD Type 2 in Practice). |
| Failed units marked skipped to get a green run. | Nothing, for weeks. Then a period total that is short and nobody can explain. | The only record of the omission was a task status that has since rotated out of retention. | Make missing units a first-class alert derived from the ledger, independent of run status (Quality Alerting). |
| Downstream mart built before the repair landed. | Upstream table complete, derived mart still missing two days. | Downstream triggering is on parent run completion rather than on unit publication. | Trigger on publish events and record which downstream artifacts consumed which unit versions (Data Lineage). |
| Repair executed by hand from an operator's terminal. | The unit exists but the ledger does not know about it, so completeness checks keep firing. | The manual path bypasses the publish step that writes the ledger. | Make the repair a parameterised invocation of the same job, never a separate script (Debugging a Data Incident). |
1-- Written by the publish step, ideally in the same transaction as the swap.2-- unit_key is whatever "independently publishable unit" means for this dataset.3CREATE TABLE IF NOT EXISTS meta.publish_ledger (4 dataset VARCHAR,5 unit_key VARCHAR, -- e.g. '2026-08-17' or 'tenant=42/2026-08-17'6 published_at TIMESTAMP,7 run_id VARCHAR,8 input_version VARCHAR, -- which raw snapshot this unit was built from9 row_count BIGINT10);11 12-- 1. Which units are missing? This is the completeness definition.13SELECT d.expected_unit14FROM meta.expected_units d15LEFT JOIN meta.publish_ledger p16 ON p.dataset = 'fct_orders' AND p.unit_key = d.expected_unit17WHERE d.dataset = 'fct_orders'18 AND d.expected_unit BETWEEN '2026-08-01' AND '2026-08-25'19 AND p.unit_key IS NULL;20 21-- 2. Which units were published more than once? A non-idempotent repair22-- shows up here before it shows up in anyone's revenue number.23SELECT unit_key, COUNT(*) AS publishes, MIN(run_id), MAX(run_id)24FROM meta.publish_ledger25WHERE dataset = 'fct_orders'26GROUP BY unit_key27HAVING COUNT(*) > 1;28 29-- 3. Which units are present but empty, or built from a different input30-- version than their neighbours? Presence is not completeness.31SELECT unit_key, row_count, input_version32FROM meta.publish_ledger33WHERE dataset = 'fct_orders'34 AND (row_count = 035 OR input_version <> (SELECT MAX(input_version)36 FROM meta.publish_ledger37 WHERE dataset = 'fct_orders'));Query 3 is the one people leave out and then need. A repaired unit legitimately carries a different input_version from the units around it — the ledger does not make that wrong, it makes it visible, so someone can decide whether it matters for this dataset.
How to build it
Most important first.
- Make the unit of work equal to the unit of publish, and make both visible to the orchestrator — mapped tasks, dynamic task expansion, one task per unit. A hundred statuses is not clutter; it is the data the repair decision needs (Task Dependencies).
- Record a publish ledger: for each unit, whether it is published, when, by which run, and with what input version. This is the artifact that turns "which ones failed" from a log-reading exercise into a query.
- Make each unit idempotent over its key before enabling any targeted repair, using replace-the-partition or merge-on-key semantics (Upserts and Merges).
- Decide and document whether units are independent. Cumulative and stateful models are not, and for them the correct repair is "recompute from the earliest affected unit forward", which is a range, not a set (Planning a Backfill).
- Fail the run visibly when units are missing, rather than allowing a skip that turns a data gap into a green dashboard. A run that publishes ninety-eight of a hundred units is a partial success and must be reported as one.
- Trigger downstream work on unit publication rather than on parent-task success, so a repaired unit propagates the same way an original one did (Data Lineage).
- Give the repair a first-class path — a parameterised run over an explicit unit list — so that the on-call action is a documented operation rather than an improvisation at three in the morning.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A per-unit publish model guarantees that each published unit is complete and consistent within itself. It guarantees nothing about the set: a table can be internally consistent per partition and missing two of them.
- A targeted repair guarantees the repaired units are recomputed under current logic. It does not guarantee they were computed against the same inputs as their neighbours, and for a mutable source they were not (Keeping Raw History: The Recovery Position and the Liability).
- Task-level success guarantees the process exited zero. Where a task loops over units internally, it guarantees nothing about how many units it completed before doing so.
- Nothing guarantees that a downstream consumer noticed the difference between "published" and "published and complete". That is what a completeness check on the set exists to provide (Freshness Checks).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The completeness check that matters here operates on the set, not on the rows: for a given period, assert that every expected unit is present in the publish ledger and published exactly once. It catches missing partitions, skipped tasks and units published twice.
- Pair it with a per-unit row-count comparison against the source, so a unit that is present but empty is distinguishable from one that is present and correct (Reconciliation).
- What both miss: units that are all present, all published once, and all computed from a source that was in a different state for two of them. Presence and uniqueness say nothing about input consistency across the set.
- A targeted repair restores freshness for the failed units in the time it takes to compute two of them, rather than all hundred. That is the entire operational argument for the pattern and it is a strong one.
- It also creates a period in which the dataset is *unevenly fresh* — ninety-eight units current, two hours behind. A single freshness number for the table averages that away and reports health; a per-unit view shows it (The Freshness SLO).
- For cumulative models the repair does not restore freshness at all until everything after the repaired unit is recomputed, which is why treating them as independent produces both a wrong result and a false sense of speed.
- Changing the unit — daily to hourly, per-table to per-tenant — invalidates the publish ledger's history and every repair runbook written against it. Plan the migration as a data migration, because that is what it is.
- Adding a dependency between units, usually by introducing a cumulative measure, silently converts an independent model into a sequential one. Nothing in the code announces it, and the repair procedure that was correct last month is now wrong (Snapshot Tables).
- A schema change applied between the original run and the repair means the repaired unit has a different shape from its neighbours, which readers that select all columns will notice and readers that name columns will not (Schema Evolution).
- The repair is the recovery, and its correctness rests entirely on the two preconditions: idempotent over the unit, and units independent. Verify both before running it, not after.
- For sequential models, recover forward from the earliest affected unit rather than repairing a hole. That costs more compute and is the only correct option (Backfills).
- When the source has changed since the original run, a repair produces a unit that differs from what a successful first attempt would have produced. Where that matters — financial periods, regulatory reporting — the source snapshot must be retained and the repair run against it rather than against current state (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- Re-running the whole job to fix two units, and thereby recomputing ninety-eight against a changed source.
- Re-running only the failed units of a cumulative model, leaving everything after the gap inconsistent with it.
- A repair against a non-idempotent task, converting a gap into a duplicate.
- Marking failed units as skipped so the run reports success, which is the single most damaging response available and the most tempting one at the end of a long night.
- Downstream jobs triggered on the original run and never re-triggered by the repair, so the correction stops one hop from the consumer (Impact Analysis).
- A repair path that exists only as an operator's shell history, so the second person to do it does something slightly different.
- Per-unit statuses that live only in logs with a short retention, so "which ones failed" becomes unanswerable a week later.
- "Two tasks failed, so re-run the DAG." That recomputes ninety-eight correct units against whatever the source looks like now. It is the default action in most orchestrators and it is usually the wrong one.
- "Re-running only the failures is always safer." It is safer when units are independent and idempotent. For a cumulative model it produces a table where a middle unit is correct and everything after it is not.
- "The run succeeded, so all partitions are there." A task that loops internally can complete after handling ninety-eight of a hundred if its error handling swallows exceptions, and it will exit zero doing so.
- "We can backfill it later." Later the source has changed, the retained raw data may have aged out, and the person who knew which two units were affected has moved teams.
- "Skipping the failed units unblocks the pipeline." It unblocks the *pipeline*. It leaves the *dataset* incomplete, and it removes the only signal that would have said so (The Pipeline Succeeded. The Data Is Wrong.).
Operating it
- Units expected versus units published, per period. One chart, and it is the definition of completeness for a partitioned dataset (Pipeline Metrics).
- Publish count per unit, so a unit published twice is as visible as a unit published zero times.
- Age distribution across units of the same table, rather than a single freshness number — the two-hour-old outlier is the whole signal (Freshness Monitoring).
- Repair runs as a distinct, labelled category of run, so that "how often do we repair" is answerable and the trend is visible (Pipeline Observability).
- At 10x units, partial failure moves from an exception to the expected condition of every run, and a manual repair procedure stops being viable.
- At 100x, the orchestrator itself becomes the constraint — scheduling and tracking hundreds of thousands of unit statuses is a workload — and the answer is usually to make the unit coarser again, accepting more redundant recompute per failure in exchange for a manageable control plane.
- More consumers make an incomplete set more expensive, because the repair has to be propagated to everything derived from it rather than simply landing in one table (Impact Analysis).
- The dominant saving is the compute not spent recomputing the successful units, which scales with the fraction that succeeded — and at a hundred units that fraction is nearly all of them (Compute Waste).
- The dominant added cost is bookkeeping: per-unit tasks, a publish ledger, and orchestrator overhead that scales with unit count rather than with data volume.
- Very fine units invert the trade. A thousand tiny tasks cost more in scheduling and produce more small files than the work they contain justifies (File Size and the Small-Files Problem).
- Repairs against retained source snapshots cost the retention. That is the price of being able to reproduce what a first attempt would have produced.
- Fine units buy targeted repair and cost orchestration overhead, more files and more metadata. Coarse units buy simplicity and cost a full recompute per failure.
- A publish ledger is a second source of truth about your data that can itself drift from reality. It is worth it, and it needs its own reconciliation against what is actually in the table.
- Making units independent sometimes means giving up a genuinely useful cumulative model, or recomputing it separately. That is a real modelling cost, not a free win (Event vs Snapshot Modeling).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe dependency of a targeted repair on idempotency and unit independence is arithmetic rather than tooling. It applies to a Spark job over partitions, a dbt run over models, and a Python script looping over tenants.
- TOOL-SPECIFICOrchestrators differ in whether they can expand a task into per-unit instances at runtime, retry an individual instance, and expose per-instance status durably. Where they cannot, the unit-level model has to be built in the job itself with its own ledger, which is more work and exactly the same idea.
- SCALE-SPECIFICBelow roughly a dozen units per run, re-running everything is cheaper than the bookkeeping needed to re-run one, and per-unit tasks are pure overhead. The inversion happens when the full recompute stops fitting inside the freshness window — that is the signal to split, not a unit count.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why partial failure is the normal condition of any multi-node computation rather than an anomaly, and what compensating actions are available when a set of operations cannot be made atomic.
- — DevOps / Production Engineering owns the incident-response side: the runbook that turns "two units failed" into a documented, parameterised, auditable repair rather than an improvisation, and the postmortem that asks why the unit was not independently repairable in the first place.