Partial Failure
The payment succeeded and the backend crashed before the order was written. Two things that were supposed to be one thing happened separately, and the system is now in a state nobody designed. The move is to find every such gap and decide what fills it.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Checkout does several things across several systems. What happens if it stops half-way, and how do you find the gaps before a customer does?
A deploy restarted the backend during a checkout. The provider shows the charge; our database has no order. The customer has a receipt email from the provider and nothing from us. I always thought of checkout as one operation, and now I can see it was five, and the crash could have landed between any two of them.
Wrap the whole checkout in a database transaction. Transactions make several steps into one, which is exactly the problem, and the word "atomic" is reassuring.
The provider is not in the transaction. A database transaction covers the rows in one database; the charge lives in another company's system, and it does not roll back when ours does. The transaction produced atomicity over the part that was already fine and none over the part that was not.
- The provider is not in the transaction. A database transaction covers the rows in one database; the charge lives in another company's system, and it does not roll back when ours does. The transaction produced atomicity over the part that was already fine and none over the part that was not.
- The crash is not an exception, so nothing runs when it happens. Rollback code, compensation code, cleanup code — all of it assumes the process is still alive to run it. A partial failure is precisely the case where it is not.
- The gap was invisible because the steps were written as one function with no boundary marked. Reading the code shows a sequence; it does not show which line, if it were the last one to run, leaves money in one system and nothing in the other.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Write the operation as a sequence of steps and, between every pair of steps, ask: if the process stops here, what exists on each side? Every gap where the two sides disagree is a partial-failure state, and it needs either a design that makes it recoverable or a decision that it is acceptable (Failure Modeling).
- Order the steps so that the durable, local record comes before the external effect. Write a pending order first, then charge; a crash after the charge leaves an order that says pending with a payment reference attached, which is recoverable. The other order — charge first, then write — leaves a charge with no record on our side, which is only recoverable from the provider's data (The Order Lifecycle, Built).
- For each remaining gap, choose a recovery mechanism and name who runs it: a job that finds pending orders and asks the provider what happened; a notification handler that completes the order when the provider says paid; a reconciliation that compares both sides on a schedule. The mechanisms are engineered in Distributed and Backend; the move is to know which gap each one closes (Treating External Systems as What They Are).
- Accept that some gaps close by compensation rather than rollback — a refund is not an undo, and the customer sees both the charge and the refund. Decide in advance which gaps you would compensate and which you would complete forward, and write that down beside the step list.
The gaps, made visible
The pipeline is checkout as it was written, with the system each step touches and, in the failure column, what a crash immediately after the step leaves behind. Reading the failure column top to bottom is the whole analysis: the gaps where the two sides disagree are the ones that need a closer.
- 1Validate cart
Local read: items exist, prices current, stock available.
fails by Crash here leaves nothing on either side. Harmless.
- 2Reserve stock
Local write: decrement available stock.
fails by Crash after: stock held, no order, no charge. Needs a release.
- 3Create charge
External: provider charges the card.
fails by Crash after: charge exists, no order on our side, no reference stored. Recoverable only from the provider's data.
- 4Write order (paid)
Local write: order row with the charge attached.
fails by Crash after: consistent — paid on both sides, email missing.
- 5Send confirmation
External: email service.
fails by Crash before: paid order, no email. Complete forward with a retry.
The third gap is the incident. It is also the one gap that the reorder removes entirely, by making sure a reference exists locally before the provider is ever called.
Before and after the reorder
The comparison is the same five steps with the local record moved ahead of the external effect. The count of steps is unchanged; the set of states a crash can produce is smaller, and every remaining state carries enough information to be finished by a job.
reserve stock → charge card → write order(paid) → email. A crash after the charge leaves a charge our database has never heard of; recovery means searching the provider's records for charges with no matching order.
write order(pending, reference) + reserve stock in one transaction → charge card with reference → mark paid → email via retrying job. A crash after the charge leaves a pending order whose reference the provider recognises; a job or the provider's notification completes it.
The durable record now exists before the irreversible step, so every gap leaves a row that says what was attempted and carries the key needed to ask the provider what happened. Recovery becomes "finish pending orders" instead of "find orphan charges".
What the incident left unknown
The fix above closes the gap; the board records what still has to be found out before the closers can be built. Each unknown started as a worry and is written here as the question and the experiment that would settle it.
- ✓Checkout is five steps across three systems, and the crash landed between charge and write.
- ✓Order rows can carry a payment reference before the charge is made.
? Can we ask the provider about a charge?
becomes Given our reference, does the provider's API return the state of a charge we attempted, including one whose response we never received?
experiment Create a test charge with a reference, discard the response, query by the reference, and confirm the state comes back.
? How long can an order stay pending?
becomes After what interval should the recovery job treat a pending order with no provider answer as abandoned, and what does it do with the stock?
experiment Stub the provider to answer "no such charge" and watch the job cancel the order and release the reservation; then stub "paid" and watch it complete.
? What if the job and the notification both arrive?
becomes Is the transition pending → paid safe to perform twice, once from each closer?
experiment Fire the notification and run the job for the same order concurrently; check for one paid order, one stock decrement, one email.
How to do it
Most important first.
- List the steps with their system: local database, external provider, email service, stock table. A step is only "one thing" with the previous step if both are in the same transaction on the same database.
- For each gap, write the two-sided state: "charge: yes; order: no". If both sides say the same thing, the gap is harmless.
- Reorder so that a local durable marker precedes each external effect. The marker carries the reference you will need to ask the external system later.
- Assign each harmful gap a closer: forward completion (a job that finishes the work), a notification that does it for you, or compensation. Write which and why.
- Test each gap by killing the process at that point — a fake provider and a kill switch make this a test rather than a hope (Failure Injection).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Checkout as steps and systems: validate cart (local, read); reserve stock (local DB); create charge (provider); write order paid (local DB); send email (email service). Gaps with harm: between charge and write-order — charge yes, order no; between write-order and email — paid, no email; between reserve and charge — stock held, nothing charged.
- Reordered: write order as pending with a fresh payment reference (local); reserve stock (local, same transaction); charge with that reference (provider); mark paid (local); email (external, retried by a job). Now the worst gap — after the charge, before mark paid — leaves an order in pending whose reference the provider recognises.
- Closers: pending orders older than the chosen window are picked up by a job that queries the provider by reference — paid becomes paid, absent becomes cancelled and stock released. The provider's notification does the same thing sooner when it arrives. Email is completed forward by a retrying job; nobody compensates an email.
- The one compensation in the design: an order paid at the provider whose stock reservation was lost — impossible after the reorder, because the reservation and the pending order are one transaction — is refunded, and the refund is an explicit admin action with a reason, not automatic.
How you know it worked
What now exists that did not before, and what question you can now ask.
- The step list has the system beside every step and a two-sided state in every gap, and each harmful gap names its closer.
- Every external effect is preceded by a local record carrying the reference needed to ask about it later.
- A killed process at any point leaves the system in a state that a job, a notification or an admin can recognise and finish.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?If the process stops between these two steps, what exists on each side, and do they agree?
- ?Which step has the lasting external effect, and is there a durable local record with a reference before it?
- ?For each gap that matters: who closes it — a job, a notification, a person — and how would I know it closed?
- ?Is this gap closed by finishing forward or by compensating, and what does the customer see in each case?
What can go wrong
- Every operation in the system gets the full treatment and a saga, and updating a product description now has a compensation step. The gap analysis is for sequences that cross a boundary with a lasting effect; a single-database write has no gaps.
- The recovery job is designed and never run, or runs and is never watched. A closer that nobody has seen close a gap is a hypothesis; cause the gap and watch it.
- Reconciliation is used as the only closer, so the customer waits for a nightly job to see their order. Reconciliation catches what the faster closers missed; it is the backstop, not the path.
- Pending-first ordering creates orders that may never be paid, and they have to be cleaned up, counted correctly in reports and hidden from the customer's history until they resolve.
- A recovery job and a reconciliation are two more things to run, monitor and explain, on top of the checkout that works when nothing fails.
- "So I need a saga." You need to know your gaps and their closers; a saga is one shape those closers can take, and for a two-step checkout a pending row and a job are the same idea with less machinery.
- "The transaction was useless." It was the wrong tool for the boundary and the right one for the local pair — order row and stock reservation should be one transaction, precisely so that gap does not exist.
- "Compensation undoes the step." It performs a new step that approximately cancels the first, and it is visible: the customer sees a charge and a refund. Design for that visibility rather than pretending the first step never happened.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny sequence that crosses a boundary with a lasting effect has gaps: upload a file then write its record, send a message then mark it delivered, create a short link then update the counter. The closers differ; the gap analysis is identical.
- SCALE-SPECIFICAt small scale a nightly reconciliation against the provider's export closes every gap and is easy to reason about; as order volume grows the window between failure and repair becomes customer-visible and the faster closers — notification, job — stop being optional.
- ILLUSTRATIVEThe deploy, the crash and the five steps are invented; the reordering and the closers show the shape of the design and not a recommendation for any particular provider.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's layer view at /manifesto/layers is useful here: a "transaction" at the application layer means something different from one at the database layer, and the gap lives between the two.