Failure Modeling
The basic path works. Now ask, step by step, what happens if the payment fails, the database is down, the request repeats, or the user closes the browser — and turn each answer into a state the system can be in and a test that puts it there.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Checkout works when everything goes right. How do you find out, systematically rather than by accident, what happens when something goes wrong?
Checkout is done: cart in, order out, test-mode payment succeeds, confirmation page renders. I demoed it and it looked finished. Then someone asked "what if the card is declined?" and I realised I do not know — not because I skipped it, but because I never had a list of the things that could go wrong, so I never noticed it was missing.
Wrap everything in try/catch and return a generic error. It is quick, it makes the demo stop crashing, and "handles errors" can now be ticked. The error page exists, and an error page looks like error handling.
The catch block treats every failure the same, so every failure produces the same wrong state. A declined card and a timed-out provider both show "something went wrong", but one order should be marked unpaid and the other should not be touched until the provider says what happened. The code cannot tell them apart because nobody listed them apart.
- The catch block treats every failure the same, so every failure produces the same wrong state. A declined card and a timed-out provider both show "something went wrong", but one order should be marked unpaid and the other should not be touched until the provider says what happened. The code cannot tell them apart because nobody listed them apart.
- The failures that are not exceptions are not caught. A request that arrives twice does not throw. A backend that crashes after the payment but before the order is written does not throw either — the process is gone. The try/catch produced a feeling of coverage over the failures that raise, and silence over the ones that matter.
- Nothing was written down, so the next person — or you, next month — has no way to know which failures were considered and which were never thought of. The error handling exists; the model of what can fail does not.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Take the happy path as a sequence of steps and, for each step, ask the same short set of questions: what if this step fails? What if it is slow? What if it happens twice? What if it succeeds but we never hear the answer? What if the thing it talks to is unavailable? The questions are boring on purpose — their value is that they are the same for every step, so nothing depends on inspiration (Failure-First Questions).
- For every answer, name the state the system is in afterwards and whether anyone can tell. "Payment succeeded, order not written" is a state. If nothing records it and nobody is notified, that is a silent failure, and silent failures are the ones that cost money (States That Must Be Unrepresentable, The Order Lifecycle, Built).
- Sort the list by what is at stake, not by how likely it looks. A duplicate charge on a slow network is rare and unacceptable; a broken image on a product page is common and fine. The ranking decides which failures get a designed response and which get a generic error, and the decision is written down either way.
- Turn each designed response into a test that produces the failure and observes the state. A failure you cannot cause is one you have not understood; the next lesson in this module that makes failures happen on purpose is Failure Injection.
The same questions, asked of every step
The table is the deliverable. Each row is one step of checkout under one of the questions, with the state the system lands in and the response that was chosen. Rows whose response is "generic error, logged" are still rows: they record that the failure was seen and ranked low, which is different from never having been seen.
The response column is where the design happens. Notice that none of the responses is "retry" on its own — a retry is only a response once the step is known to be safe to repeat, and that is a separate question (Duplicate Requests).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Provider returns declined | Customer sees a failure; nothing charged | Card refused — a normal outcome, not an error | Order to payment-failed, customer told the reason the provider gave, cart kept so they can retry with another card. |
| Provider request times out | Our call returns nothing; the card may or may not have been charged | Network or provider slowness after the charge was made or before it | Order to payment-unknown; query the provider by our reference before doing anything else; never re-submit the charge blind. |
| Customer submits checkout twice | Two payments for one cart | Double click, a browser retry, or a refresh on the confirmation request | One idempotency key per checkout attempt, generated before the first submit; the second submit returns the first result. |
| Database unavailable at "write order" | Payment taken, no order exists | The database failed between two steps that should have been one | Reserve the order row before charging, so the charge always has an order to attach to; reconcile from the provider's records if the write still fails. |
| Browser closed after paying | Customer never sees the confirmation; may assume it failed and buy again | The confirmation depended on the browser being there | The order's paid state comes from the provider's server-side notification, not from the browser; the confirmation email is the proof the customer sees. |
| Confirmation email fails to send | Paid order, no email | Email service unavailable | Generic: log, retry in the background, show the order in the account page regardless. |
From failures to testable pieces
The ranked rows decompose into work. The tree below is not the store's feature list; it is the list of failure behaviours checkout needs, each with the observation that would show it works. A leaf without an observation is a worry that has not become a task.
The leaves are small on purpose. "Handle payment failure" is a heading; "a declined test card leaves the order in payment-failed and the cart intact" is a thing you can watch happen.
- ├Payment outcomes— the highest-stakes boundary
- └Declined cardtestable A declined test card leaves the order in payment-failed with the provider's reason, and the cart is unchanged.
- └Timed-out providertestable With the provider stubbed to hang, the order lands in payment-unknown and a status query to the provider is logged before any retry.
- └Repeated submittestable Submitting the same checkout twice produces one payment and one order; the second response equals the first.
- ├Our own half fails— the failures that do not throw
- └Crash between charge and order writetestable Killing the process after the charge returns leaves a record from which the order can be recovered, and the recovery job recovers it.
- └Stock reserved and never releasedtestable An abandoned checkout releases its reservation after the chosen window, and the product is buyable again.
- ├The customer's half fails— the browser is not a reliable witness
- └Browser closed after payingtestable With the confirmation request never sent, the order still becomes paid from the provider's notification and the email still goes out.
Everything here is a behaviour of code that already exists. Failure modelling rarely adds features; it adds states and the tests that prove the states are reached.
An order to work in, and when to break it
The rows do not all need doing today. The sequence below handles the failures in stakes order, and it is one sequence. A store with a single warehouse and generous stock might push the reservation work to the end; a store selling concert tickets would move it to the front, because for tickets "sold twice" is the double charge.
The slogan "handle errors gracefully" becomes checkable here: graceful means every row has a chosen state and the customer can tell which one they are in. A page that says "something went wrong" for a card that was charged is not graceful, however friendly the copy.
- 1Add the missing states to the order lifecycle
because Every later step needs somewhere to land; payment-unknown in particular cannot be retrofitted into a boolean "paid" column.
- 2Idempotency on checkout submit
because The duplicate charge is the highest-stakes row and the cheapest to test — submit twice and count.
- 3Order-before-charge ordering and a recovery job
because The crash between charge and write is the second row and needs the states from step one to record what happened.
- 4Timeout handling with a status query, not a retry
because Depends on knowing the provider's query API, which is a research item — do it once the earlier rows are solid.
- 5Reservation release, email retry
because Lower stakes; a generic response is acceptable while the higher rows are unfinished.
How to do it
Most important first.
- Write the happy path as numbered steps with the boundaries marked — which steps talk to the database, which to the provider, which to the browser (Where Does My System End?). Failures cluster at boundaries.
- Ask the five questions of each step, and write the answer even when it is "nothing bad" — that is a decision too, and it is checkable later.
- For each bad answer, write the resulting state in the language of the order lifecycle: paid, unpaid, pending, unknown. If there is no word for the state, the lifecycle is missing one (Finding the State Machine).
- Rank by stakes. Anything that moves money or loses an order gets a designed response; anything cosmetic gets the generic error and a note saying so.
- For the top of the list, write a test that causes the failure. If you cannot cause it, write down what you would need in order to — a fake provider, a killed process — and go build that next (Failure Injection).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Checkout, as steps: validate the cart, calculate the total, reserve stock, create the payment with the provider, receive the provider's result, write the order, send the confirmation. Boundaries: the database at reserve and write, the provider at create and receive, the browser at confirmation.
- "Create payment" under the five questions: fails → provider returns declined, order stays unpaid, customer told why. Slow → we wait; how long, and what does the customer see meanwhile? Twice → the same customer could be charged twice; this is the highest-stakes row and becomes Duplicate Requests. Succeeds but we never hear → the provider charged the card and our request timed out; we do not know whether to write the order. Unavailable → nothing can be charged; the order can still be created as pending, or checkout can refuse — a decision to make, not a bug.
- The resulting state list for the store's orders: unpaid, payment-pending, paid, payment-failed, payment-unknown. The last one did not exist before this exercise. It is the state for "we asked the provider and did not get an answer", and without it the system has to lie — mark the order either paid or failed while it is neither.
- Ranked: duplicate charge, then payment succeeded but order not written (Partial Failure), then payment-unknown handling, then stock reserved but never released, then confirmation email failing. The first three get designed responses and tests; the last gets a retry and a log line.
How you know it worked
What now exists that did not before, and what question you can now ask.
- A list exists: step, failure question, resulting state, response, and whether it is tested. It is short enough to read and specific enough to argue with.
- The order lifecycle gained at least one state you had not thought of. If it gained none, either the lifecycle was already thorough or the questions were asked too gently.
- You can now say which failures the system handles by design, which it handles generically, and which it does not handle — and the third category is a decision, not an oversight.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?For each step on the path: what if it fails, is slow, repeats, succeeds silently, or cannot be reached?
- ?What state is the system in after each of those, and can anyone tell from the outside?
- ?Which of these failures moves money or loses data, and therefore needs a designed response rather than a generic one?
- ?Which of these failures can I cause on purpose today, and which need a fake or a kill switch first?
What can go wrong
- Every step gets every question and every answer gets a designed response, and the store has a state machine with thirty states before a second customer has used it. The ranking step exists so that most rows end in "generic error, logged"; a failure model that treats a broken thumbnail like a double charge has not modelled anything.
- The questions are asked once, at design time, and never again. Each new step in the path — a coupon, a second warehouse — adds new rows, and the list only stays honest if adding a step means asking the questions of it.
- Failures are modelled in prose and never caused. "Payment timeout: mark order pending, retry status check" reads well and may not work; the model is a hypothesis until an injected timeout confirms it (Failure Injection).
- The exercise takes an afternoon for a path that worked this morning, and produces states and tests rather than features. On a prototype that will be thrown away, the afternoon is wasted.
- Every new state is code, UI and a support question. "Payment-unknown" is honest, and it also means someone has to decide what the customer sees and what the admin does about it.
- "So I should handle every failure." No — you should know every failure on the path and decide which to handle. The decision to show a generic error is fine when it is a decision.
- "The failures are in the code, so I should read the code to find them." The failures are in the steps and the boundaries, which is why the model is built from the happy path as a sequence rather than from the implementation. Code tells you what is caught; the sequence tells you what can happen.
- "Failure modelling is a testing activity." It changes the state model and often the data model — the payment-unknown state is a column — which is why it comes before the tests, not after.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe five questions apply to any sequence of steps across any boundary — a message send, a file upload, a pipeline stage. What changes is the stakes ranking: in an analytics dashboard "twice" is usually harmless and "slow" is the failure that matters.
- CONTESTEDSome practitioners hold that failures should be designed with the happy path, not after it — that "happy path first" produces a data model with no room for payment-unknown and the retrofit is the expensive part. The strongest form: for payments and anything else that moves money, the failure states are the requirements, and a happy path built without them is a prototype. This lesson answers that the happy path is where the steps come from, and that the failure model should follow it within the same day, not the same quarter.
- ILLUSTRATIVEThe checkout steps, the five states and the ranking are invented for the running example; a real provider's failure modes and a real store's priorities will differ.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — Testing & Reliability, when it exists, owns the mechanics of fault-injection frameworks; this module owns the question of which faults are worth injecting.