Pipeline Reliability
Retries, checkpoints, atomic publish, partial failure and rollback — plus the SLOs that make freshness a commitment instead of a hope.
Reliability is not a low failure rate. It is seven mechanisms — retries, idempotency, checkpoints, atomic publish, validation, rollback, reprocessing — that are only safe as a set.
A consumer must never read a half-written dataset. Build somewhere they are not looking, validate it there, then make it visible in one operation — and know which of the available operations is genuinely one.
A restarted job has to resume from somewhere. A checkpoint is correct only if it records the input position and the computed state together, in one atomic action — otherwise it is a dual write wearing a reliability hat.
Ninety-eight partitions succeeded and two failed. Re-running only the two is right — but only if the unit is idempotent and independently publishable, and most people check neither before doing it.
Retrying a task that already published is not a retry — it is a second publish. Retry is safe exactly when the task is idempotent, and a uniform retry policy applied to tasks that are not uniformly idempotent is the honest failure here.
A published, measured promise about a dataset — when it arrives, how fresh it is, how often it is right — agreed with the people who depend on it rather than declared by the team that runs it.
Now minus the event time of the newest **complete** data. The word "complete" does all the work: a table holding a few hours of today is extremely fresh and completely wrong to aggregate.
Reverting the transformation code does not revert the tables it wrote. A data rollback is either a restore from a retained snapshot or a re-run of the previous logic over the affected range — and both of them are forward operations.