ReliabilityGENERALTOOL-SPECIFICWAREHOUSE-SPECIFIC

Retries in Pipelines

Retrying a task that already published is not a retry — it is a second publish. Retry is safe exactly when the task is idempotent, and a uniform retry policy applied to tasks that are not uniformly idempotent is the honest failure here.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

A task times out after its write committed. The orchestrator retries it. What just happened to the data?

Who needs this

Every reader of the table the retried task writes into. They cannot see attempt numbers, and the artifact of a bad retry is not an error — it is a number that is too high, which is the direction people question least (Duplicate Rows).

What one row is

A retry operates on whatever the task publishes: a partition, a table, a set of rows matched by a key. Idempotency is a property of the task relative to that unit, never a property of the task alone. The same SQL is idempotent when it replaces a partition and not idempotent when it appends to a table, and nothing about the code distinguishes the two (Partial Failure).

The obvious build

Set retries: 3 with exponential backoff on every task in the project, as a default in the orchestrator configuration. It is one line, it removes an entire category of overnight pages, and it is the single highest-value reliability change most teams ever make. The advice to do it is good advice.

Why it breaks

The task wrote its rows, committed, and then the connection dropped before the acknowledgement arrived. The task raises, the orchestrator retries, and the same rows are written again — the retry is doing exactly what it was told and the data is now doubled (At-Least-Once Delivery).

How it breaks with real data

The task wrote its rows, committed, and then the connection dropped before the acknowledgement arrived. The task raises, the orchestrator retries, and the same rows are written again — the retry is doing exactly what it was told and the data is now doubled (At-Least-Once Delivery).
A task calls a partner API to enrich rows, times out at ninety seconds, and is retried. The API had already accepted and charged for the request; the pipeline now has two enrichment runs and the partner has two records of one (Idempotency Keys: The Mechanism).
The retry runs an hour later against a source that has moved on, so the retried unit is computed from different inputs than its siblings and nothing records that fact (Reprocessing vs Retrying).
Three retries on every task in a two-hundred-task DAG hits a warehouse that is already struggling, and the retries are the reason it stays struggling — a self-inflicted load spike at exactly the moment capacity is scarce (Retry Storms: The Load You Generated Yourself).
A task that sends a notification, refreshes a materialised view in another system, or writes to a queue is retried, and the side effect happens twice while the data write happens once (The Transactional Outbox).
The retry succeeds, the run goes green, and the incident that produced the first failure is never investigated — so the underlying degradation continues until it is bad enough to exhaust the retries too.

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

The failure a retry responds to is almost never "the work did not happen". It is "the report of the work did not arrive". Those are different, and no protocol distinguishes them from the caller's side: a timeout is silence, and silence is compatible with success (Timeouts: The Latency Contract Nobody Writes Down).
So a retry is not a second *attempt*; it is a second *execution*. Whether that is harmless depends entirely on what the first execution left behind, and the orchestrator has no way to find out.
Idempotency over the unit is what makes the second execution harmless. Concretely: delete-then-insert the unit inside one transaction, or merge on the business key, or write to a deterministic location that the second execution overwrites. All three make "ran once" and "ran twice" produce the same table (Upserts and Merges).
The property that makes idempotency work is determinism of the key. A key derived from the input — order id, partition date, a hash of the source record — is stable across executions. A key derived from the execution — a generated id, a load timestamp, an auto-increment — is different every time, so the merge inserts instead of updating and the deduplication does nothing (Deduplication).
Retries also change *when* work runs, not just how often. A retried task reads its inputs later, which for a mutable source means it reads different inputs. Idempotency makes the second execution safe with respect to duplication and says nothing about it being safe with respect to consistency (Keeping Raw History: The Recovery Position and the Liability).
The honest failure mode of this whole area is uniformity. Retry policy is set per project because that is how the configuration is shaped, while idempotency is a per-task property that varies wildly across the same project. The mismatch is structural, not careless (Orchestration).

The run log that looks fine

The log below is what a duplicating retry looks like in an orchestrator. It is worth reading line by line, because there is nothing in it that would attract attention during a morning review — a task failed with a timeout, it was retried, the retry succeeded, the run completed. That is the normal, healthy shape of a retry working as intended.

The information that would have changed the reading is not in the log at all: between 02:04:11 and 02:04:41 the warehouse committed eight hundred thousand rows. The task never learned that, because the connection carrying the acknowledgement is exactly what broke. From the orchestrator's side the attempt is indistinguishable from one that did nothing.

The last two lines are the part that matters operationally. The publish ledger records the unit twice, and the uniqueness assertion fires — but only because both of those exist. Without them the run is green, the table has 1.6 million rows where 800 thousand belong, and the first person to notice is whoever compares the month to last month.

The first attempt succeeded at the database and failed at the client. No retry configuration can tell those apart, and no amount of care in the task code changes it.
The retry is not a bug. It is the mechanism working. The bug is that the task appends rather than replacing its unit.
The assertion is the only thing standing between this and a wrong quarterly number, and it works only because it runs before the unit becomes visible (Atomic Publish).
The fix is four words: replace the unit, transactionally. Not "reduce the timeout", not "lower max_attempts".

run_id=2026-08-25T02:00Z  dag=orders_daily

02:00:03  transform_orders[dt=2026-08-25]  RUNNING   attempt=1
02:04:11  ...  warehouse COMMIT succeeded          <- not visible to the task
02:04:41  transform_orders[dt=2026-08-25]  FAILED    attempt=1
          error: connection reset by peer while awaiting result
02:04:41  retry policy: max_attempts=3, backoff=exponential, jitter=full
02:05:58  transform_orders[dt=2026-08-25]  RUNNING   attempt=2
02:09:30  transform_orders[dt=2026-08-25]  SUCCESS   attempt=2
02:09:31  publish_ledger: fct_orders / 2026-08-25 -> published (run 02:00Z)
          publish_ledger: fct_orders / 2026-08-25 -> published (run 02:00Z)   [2nd row]
02:09:34  assert_unique(fct_orders.order_id, dt=2026-08-25)  FAILED
          800,412 keys with COUNT(*) = 2

          Without the ledger row and the assertion, this run is GREEN.

Three classes of task, three policies

WAREHOUSE-SPECIFICMulti-statement transactions and MERGE are not universally available: some engines support one, some the other, some neither for large tables. Where neither exists the retry-safe form is writing a new partition or snapshot and swapping it, which is the same idea implemented at the storage layer rather than in SQL.

Retry policy is a per-task decision because idempotency is a per-task property. Applying one number across a project is convenient exactly to the degree that the project is uniform, and no real project is. The classification below is coarse on purpose — three classes is few enough that people actually apply it.

Notice that the recommendation for the middle class is not "do not retry". It is "convert it, then retry freely". An append-only task is not a permanent constraint; it is a task that has not yet been given a unit to replace. Most conversions are a WHERE clause and a transaction.

The third class is the one that leaves the data domain. Once a task calls something outside your platform, safety is a property of the receiving system's idempotency handling, and the correct instrument is a caller-supplied key that the receiver stores, not a retry policy on your side (Webhook Idempotency).

What should this task's retry policy be?

What does a second execution of this task, with the same parameters, do to the world?

Replaces a unit — retry freely

when The task deletes-and-inserts a partition in one transaction, merges on a deterministic business key, or writes a versioned snapshot that a later commit supersedes.

cost The idempotent write costs more per run than an append: a scan of what is being replaced, or a read-before-write for the merge. Buys a task where retry count is purely a tuning decision (Upserts and Merges).

Appends — convert before retrying

when The task inserts rows without deleting or matching. This is most tasks that were written before anyone thought about retries.

cost Conversion work: define the unit, scope the delete, wrap in a transaction. Until then, set attempts to one and accept the pages — a page is cheaper than a silent duplicate.

Has an external side effect — key it, do not count it

when The task calls a partner API, sends notifications, triggers a job in another system, or writes to a queue other people consume.

cost Requires the receiver to honour an idempotency key you supply and derive from the input. Where the receiver does not, the only safe policy is zero retries and a human (The Idempotency Key Flow).

Reads a mutable source — bound the inputs first

when The task computes from an operational table whose contents change between attempts, so a retry an hour later is a different computation.

cost Requires pinning inputs: an as-of timestamp, an immutable raw snapshot, a source version recorded with the output. Without it the task is idempotent about duplication and non-deterministic about content (Keeping Raw History: The Recovery Position and the Liability).

Terminal failure — do not retry at all

when The error is a failed assertion, a schema mismatch, a parse error, a permission denial. Deterministic failures do not become successes on a second execution.

cost Requires classifying exceptions rather than catching everything, which is real code. Buys back the response window that three doomed attempts would have spent (An Error Taxonomy Clients Can Branch On).

Converting an append task into a retry-safe one

1-- CLASS: appends. A second execution doubles the unit.
2INSERT INTO fct_orders
3SELECT order_id, order_date, net_amount_minor, customer_key
4FROM   stg_orders
5WHERE  order_date = DATE '2026-08-25';
6 
7-- CLASS: replaces a unit. A second execution is a no-op in effect.
8BEGIN;
9  DELETE FROM fct_orders WHERE order_date = DATE '2026-08-25';
10  INSERT INTO fct_orders
11  SELECT order_id, order_date, net_amount_minor, customer_key
12  FROM   stg_orders
13  WHERE  order_date = DATE '2026-08-25';
14COMMIT;
15 
16-- CLASS: replaces by key. Use where the unit is not a contiguous partition,
17-- for example late-arriving corrections that touch several dates.
18MERGE INTO fct_orders AS t
19USING (
20  SELECT order_id, order_date, net_amount_minor, customer_key
21  FROM   stg_orders
22  WHERE  ingested_batch_id = :batch_id     -- deterministic, from the input
23) AS s
24ON t.order_id = s.order_id
25WHEN MATCHED THEN UPDATE SET
26  order_date = s.order_date,
27  net_amount_minor = s.net_amount_minor,
28  customer_key = s.customer_key
29WHEN NOT MATCHED THEN INSERT VALUES
30  (s.order_id, s.order_date, s.net_amount_minor, s.customer_key);
31 
32-- THE TRAP: a merge key that the execution generates rather than the input.
33-- Every attempt produces new keys, so nothing ever matches and the merge
34-- degenerates into an insert -- an idempotent-looking statement that is not.
35--   ON t.load_id = s.load_id      -- load_id := uuid() at run time.  WRONG.
36--   ON t.loaded_at = s.loaded_at  -- loaded_at := current_timestamp. WRONG.

The three working variants differ in cost, not in safety: the partition replace scans the partition, the merge reads before it writes. The trap at the bottom is the one that survives code review, because a MERGE statement reads as idempotent regardless of what its ON clause actually compares.

What a retry actually costs, and where it lands

Retries are usually discussed as a correctness question, which is what the rest of this lesson has done. They are also a load question, and the load is unusual in shape: it arrives on a dependency that is by definition already failing, from many tasks at once, at the exact moment capacity is scarce.

The relative weights below establish an ordering rather than a magnitude. The point is that the largest driver is not the retried compute itself — it is the concurrent pressure a wide DAG applies to one struggling dependency when all its tasks back off on the same schedule. Jitter is not a refinement; it is what stops a retry policy from being a synchronised load test (Without Jitter, Every Client That Failed Together Retries Together).

The smallest driver is the one teams optimise first: shaving an attempt off the policy. It changes the total least and it removes the tolerance that made the policy worth having.

What drives the cost of a retry policy, relative to each other

Synchronised load on a failing dependency

Attempts multiplied by concurrent tasks, all arriving together when backoff is unjittered. This is what turns a degradation into an outage, and it is the driver with the largest spread between good and bad configuration.

Recomputing work that already succeeded

Scales with task runtime and with how late in the task the failure occurs. A task that fails at the end is the expensive case and also the most likely one, because most of the elapsed time is at the end.

Absorbing and later removing duplicate output

Zero when tasks replace their unit; large and delayed when they append, because the work is discovering it, scoping it, and reprocessing a range to remove it.

The idempotent-write premium on every run

A merge reads before writing and a partition replace scans what it replaces. Paid on every run rather than only on retried ones, which is why it feels expensive and is still the cheapest line here.

Extended run duration from backoff waits

Pure waiting rather than work, so it consumes freshness budget rather than compute. Matters where the SLO is tight and is otherwise the least significant driver.

Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.

Relative weights for a wide batch DAG against a shared warehouse, shown to establish an ordering — not measurements. The ordering is the teaching: configuration of concurrency and jitter dominates, and reducing the attempt count, which is what people reach for, is at the bottom.

Retry failures, including the failure of retrying itself

Trigger	Symptom	Cause	Response
Timeout after a successful commit.	Duplicate rows for exactly one unit, with no error anywhere.	Success and the report of success are separate events, and only the second one failed.	Make the task replace its unit. This is the only response that addresses the cause rather than the odds (Idempotent Data Pipelines).
Fixed-interval retries across a two-hundred-task DAG.	The dependency recovers, is immediately overwhelmed by synchronised attempts, and fails again.	Every task backed off by the same amount from roughly the same moment.	Full jitter on the backoff and a cap on concurrent attempts against any one dependency (Thundering Herd).
Retrying a deterministic failure.	Three identical failures separated by backoff, and a page forty minutes later than necessary.	The task catches all exceptions and reports them uniformly as retryable.	Classify errors: retry transport and throttling, fail fast on assertions, schema and permissions (An Error Taxonomy Clients Can Branch On).
Retry runs an hour later against a mutable source.	The unit reconciles, is unique, and disagrees with the units around it.	Idempotency was implemented for duplication and not for determinism.	Pin the inputs — read from an immutable raw snapshot and record which version the unit was built from (The Raw Landing Zone).
Retries succeed every night for a month.	A platform that appears healthy, then a hard failure with no warning.	Retries were absorbing a worsening degradation and nothing trended the attempt count.	Alert on first-attempt success rate, not only on final failure (Quality Alerting).
External API call retried without an idempotency key.	The partner has two records of one request; your table has one.	The receiver deduplicates on a key it was never given.	Derive a key from the input, send it, and confirm the receiver honours it. Where it does not, do not retry (Webhook Idempotency).

How to build it

Most important first.

Classify every task by what a second execution does, and write the classification down next to the task. Three classes cover nearly everything: replaces a unit, appends to a unit, or has an external side effect. Only the first is unconditionally retry-safe.
Make replacement the default write mode. A pipeline in which every task replaces its unit needs no special reasoning about retries at all, and that is worth more than any policy tuning (Idempotent Data Pipelines).
Set retry policy per class, not per project: generous on replace-tasks, zero or one on append-tasks until they are converted, and for external side effects use an idempotency key supplied by the caller rather than a retry count (Idempotency in Backends).
Use backoff with jitter, and cap total attempts. Fixed-interval retries across a wide DAG synchronise into a load spike on the dependency that is already failing (Without Jitter, Every Client That Failed Together Retries Together).
Distinguish retryable from terminal failures. A timeout, a connection reset and a throttle are retryable; a schema mismatch, a division by zero and a failed assertion are not, and retrying them wastes the window in which a human could have acted (An Error Taxonomy Clients Can Branch On).
Never retry across the publish boundary. Retry the build; do not retry the swap unless the swap itself is idempotent, which for a versioned commit it usually is and for an append it never is (Atomic Publish).
Record attempt counts as data, not just as log lines, so that a task quietly retrying every night is visible as a trend rather than as an anecdote (Pipeline Metrics).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

Retries guarantee that a transient failure does not require a human. They guarantee that the number of executions is one *or more*, never exactly one, and no configuration changes that.
An idempotent task guarantees that any number of executions over the same unit with the same inputs leaves the same result. It does not guarantee the same result across executions with *different* inputs, which is what a delayed retry against a mutable source produces.
A merge on a deterministic business key guarantees no duplicate rows for that key. It guarantees nothing about rows whose key differs — a redelivery with a fresh event id is a new row by every definition the merge has access to.
Retries guarantee nothing about external side effects. Those are made safe by an idempotency key the receiver honours, and by nothing on the sender's side (The Idempotency Key Flow).
Nothing here promises that the second execution is *equivalent* to the first, only that it does not duplicate. Equivalence requires immutable inputs.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check that catches a bad retry directly: uniqueness on the business key, asserted before publish. A second execution that duplicated rows fails it; a second execution that replaced them passes.
The complementary one: publish count per unit from the ledger. Duplicated *publishes* are visible there even when the rows themselves happen to overwrite cleanly (Partial Failure).
What both miss: a retry that produced *different* rather than duplicate data, because it ran later against a changed source. Uniqueness is satisfied, counts are satisfied, and the unit disagrees with its neighbours. Only an input-version record catches that (Reconciliation).

Freshness

Retries extend the tail of a run, not its middle. Publishing a freshness commitment derived from typical runtimes and then discovering the retry path on the day it matters is a common and avoidable form of self-deception (Pipeline SLOs).
Backoff trades recovery speed for dependency protection: longer waits are gentler on a struggling system and push the publish further out. Where the SLO is tight, that trade has to be made explicitly rather than inherited from a default.
A capped retry policy makes the worst case computable — attempts multiplied by task runtime plus total backoff — which is the number a freshness SLO should actually be built from (The Freshness SLO).

When the schema or meaning changes

Adding a column that is populated from current_timestamp or a generated id silently breaks idempotency, because the merge key or the row content now differs between executions. This is the most common way a previously safe task becomes unsafe, and no review catches it because the change looks like metadata (Nullability & Defaults).
Changing a task from replace to append — usually to make it faster — changes its retry safety class without changing its retry configuration. The configuration is the thing that should have been reviewed.
Adding an external call to an existing pure transformation moves it into the side-effect class. Anything that talks to a system you do not control needs the caller-supplied idempotency key, not a retry count (External Calls Inside a Transaction).

How to re-run this safely

A retry *is* the recovery for transient failure, and it is the cheapest one available. The point of this lesson is not to discourage it but to make it safe enough to use freely.
When a retry has already duplicated data, recovery is a replace of the affected unit from immutable inputs — which is the same operation the task should have been doing in the first place (Backfills).
Where the duplication reached an external system, recovery is a reconciliation with that system rather than a data operation, and it usually cannot be automated (Data Incidents).

What can go wrong

Failure modes

Retry after a committed write, producing duplicates. The canonical case.
Retry of a non-deterministic task, producing a unit that differs from a first attempt in ways nothing records.
Retries synchronised across a wide DAG, converting a dependency's degradation into its outage (Retry Storms: The Load You Generated Yourself).
Retrying terminal errors, burning the response window on failures that will never succeed.
Idempotency keyed on something the execution generates, so the merge inserts every time and the mitigation is inert.
A retry that succeeds and masks a degradation whose next appearance exhausts the attempts.
Retry counts configured once at project level and never revisited as tasks changed class underneath them — the failure of the mitigation itself.

Misreads

"Retries are safe because the task failed." The task reported failure. Whether it had already written is a separate question, and the report is the part that failed most of the time.
"We use at-least-once delivery, so we are fine." At-least-once is the *problem statement*, not the solution. It is fine only because something downstream deduplicates, and that something is what deserves the attention (At-Least-Once Delivery).
"Exponential backoff makes retries safe." Backoff protects the dependency from load. It does nothing about duplicate effects, and a slower duplicate is still a duplicate (Without Jitter, Every Client That Failed Together Retries Together).
"The task is idempotent, we wrote it that way." Idempotent over which unit, keyed on what? A merge on a key that includes a load timestamp is a merge that never matches (Deduplication).
"More retries means more reliable." Beyond a small number, additional attempts mostly convert a fast failure into a slow one, and against a struggling dependency they make recovery less likely (Retry Storms: The Load You Generated Yourself).

Operating it

How you see it in production

Attempts per task per run, retained as data. The absolute number matters less than the trend: a task that started retrying last Tuesday is reporting an upstream problem (Pipeline Metrics).
The gap between first-attempt success rate and final success rate. A high gap means the platform looks healthy and is being held together by retries (Pipeline Observability).
Publish count per unit, which distinguishes a harmless retry from a duplicating one without needing to read any code (The Data Quality Dashboard).
Retry concurrency against shared dependencies — how many tasks are backing off against the same warehouse at once — because that is the signal that precedes a self-inflicted outage (The Backlog Arithmetic: Four Levers and a Drain Time).

What changes at 10x and 100x

At 10x tasks the per-task retry configuration stops being reviewable by hand, and the classification has to be enforced by convention or by a lint rather than by attention.
At 100x, retries against a shared dependency become a capacity planning input in their own right: worst-case load is attempts multiplied by concurrency, and that is the number the dependency must survive (Capacity Planning: Traffic to Machines).
More consumers do not change retry mechanics but raise the cost of a duplication incident, because the duplicated unit has already been read and materialised downstream before anyone notices (Impact Analysis).

What drives cost here

A retry costs a full re-execution of work that was expensive enough to have already been running. The driver is task runtime multiplied by attempts, and it lands on the same shared compute the first attempt used (Compute Waste).
It also costs the source: a retried extract reads the source again, and for an operational database that read competes with the application at the moment the pipeline is already misbehaving (Workload Isolation).
Idempotent writes cost more than appends — a merge reads before it writes, a partition replace scans what it replaces — and that cost is paid on every run, not only on retried ones. It is the standing premium for retry safety.
The largest cost is the one that only appears sometimes: absorbing duplicate output, discovering it late, and reprocessing a range to remove it.

What this approach costs

Idempotent writes cost throughput on every run to buy safety on the rare retried one. For a large append-only fact table that premium is real and it is still the right trade, because the alternative is a table you cannot repair.
Backoff protects dependencies and costs freshness. Aggressive retries restore freshness and can prevent a struggling dependency from recovering.
Per-class retry policy is more correct and more work than a project-wide default, and it has to be maintained as tasks change. A project-wide default plus a convention that every task replaces its unit gets most of the benefit with less bookkeeping.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThat a timeout cannot be distinguished from a lost acknowledgement is a property of networks rather than of any tool, so the requirement that retries be paired with idempotency holds identically for orchestrated batch jobs, streaming sinks and hand-written scripts.
TOOL-SPECIFICOrchestrators differ in whether retry policy can be expressed per task, per task class or only per project, and in whether a retry re-evaluates the task's parameters — a retry that re-reads "yesterday" from the wall clock can run against a different date than the original attempt, which is a subtle non-determinism worth checking for in your specific tool.
WAREHOUSE-SPECIFICWhether a delete-then-insert can be made atomic depends on the engine supporting multi-statement transactions, and whether a merge is efficient depends on clustering and on the engine's merge implementation. On engines lacking either, the idempotent pattern must be partition replacement instead, which changes the cost profile completely.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Backendretries backoff-and-jitter idempotency-in-backends idempotency-keys idempotency-key-flow webhook-idempotency error-taxonomy

Architectureidempotency circuit-breaker

Observabilityretry-storms timeouts-and-latency tail-latency capacity-planning

Concurrencythundering-herd timeouts singleflight

Domains that do not exist yet

— Distributed Systems owns the underlying result — that a caller cannot distinguish a lost request from a lost response — and the taxonomy of delivery semantics built on top of it. Every argument in this lesson is that result applied to a scheduled job.
— DevOps / Production Engineering owns retry budgets and the operational side of protecting a dependency from its own clients, including circuit breaking and load shedding as deployment-time concerns rather than per-task configuration.