External Systems Fail
Everything outside the line can fail, can be slow, can change and can rate-limit — and will, on a schedule you do not control. For each crossing, deciding what the store does in each of those four cases is design work, not error handling.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
For each external system the store calls, what happens when it fails, when it is slow, when it changes and when it says "too many requests" — and who decided that?
Checkout worked for months. This afternoon the payment provider had "elevated latency" and our whole site went unresponsive — not just checkout, everything — and some customers were charged for orders we never created. I have a try/catch around the payment call. I thought that was the failure handling.
Wrap the call. A try/catch around the provider call, log the error, show "something went wrong". It catches exceptions, and exceptions are what failure looks like in code; the rest of the system is unaffected because the exception did not escape.
The try/catch handles the case that never happens — a clean, fast error — and ignores the one that did: no error at all, just a call that has not returned yet. The request thread waits, the connection pool fills with waiting threads, and every other page that needs a database connection queues behind checkout. The catch block was never entered.
- The try/catch handles the case that never happens — a clean, fast error — and ignores the one that did: no error at all, just a call that has not returned yet. The request thread waits, the connection pool fills with waiting threads, and every other page that needs a database connection queues behind checkout. The catch block was never entered.
- Some calls did succeed on the provider's side after our timeout. The provider charged the card and sent a webhook; our code had already told the customer it failed and never created the order. "Failed" was decided by us before the outside system had finished deciding.
- Nobody can say what the store should have done, because the behaviour under a slow provider was never designed — it was whatever the language runtime did with an unanswered socket. The incident review produces "add a timeout", which is one of four decisions and the easiest.
- The same crossing will be hit by a payload change and a rate limit within the year, and each will be handled after the fact, by a different engineer, as a bug rather than as the remaining rows of a table that could have been filled in on day one.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- For every crossing found in Inside and Outside the System, write four rows: it *fails* (returns an error or no connection), it is *slow* (does not return within the time the user will wait), it *changes* (the payload, the semantics, the version), it *rate-limits* (says no because you asked too often). Each row gets a decision, and "let the exception propagate" is a decision only if you write it down and can defend it.
- For the slow row, decide the timeout from the *user's* budget, not the provider's documentation, and decide what the user is told when it fires. Then notice that a timeout is not a failure: the outside system may still complete, so the store needs a state — pending — that admits it does not know yet, and a path to find out later.
- For the fail row, separate "we know it did not happen" from "we do not know". A connection refused is the first; a timeout is the second. The two need different user messages and different follow-ups, and treating both as "failed" is how the customer gets charged for an order that does not exist.
- For the change and rate-limit rows, decide where the store learns about it: a versioned adapter that fails loudly on an unknown shape, and a queue with a rate that matches the limit rather than a retry loop that makes it worse. Then write the decisions next to the crossing, where the next engineer will find them.
Four rows per crossing
The failure table is the device this lesson exists for. Each row is one of the four ways an outside system misbehaves, applied to the payment-authorise crossing; the response column is a design decision, and the one in the "slow" row is the one that was missing on the afternoon of the outage.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Fails: connection refused or a clean error response | checkout cannot proceed | the provider is down or rejected the request, and we know it | order stays unpaid; customer told to retry; nothing pending |
| Slow: no response within the user's budget | without a timeout, every request thread waits and the pool empties | the call holds a shared resource while an outside system decides | timeout from the user budget; order enters pending-confirmation; worker re-checks by idempotency key; webhook may resolve first |
| Changes: a new field, a renamed status, a new version | orders silently mis-classified | the adapter assumed the old shape | adapter validates the shape and fails loudly to us; version pinned; one contract test in CI |
| Rate-limits: "too many requests" on a sale day | checkouts fail in bursts; retries make it worse | our traffic exceeds the contract, and each retry adds to it | no retry loop on checkout; the customer is told to wait; background work drains at the allowed rate |
What the timeout does not tell you
The pipeline is the slow row in detail, because it is the row that produces a state the reflex never designed. Follow one checkout through a slow provider: the store gives up waiting, the provider finishes anyway, and the two records now disagree until something reconciles them. Each step names what would go wrong if it were skipped.
- 1Send authorise with an idempotency key
the provider can recognise a repeat of this exact request
fails by without the key, any later re-check or retry is a second charge
- 2Wait up to the user budget
bounds how long the request thread and its connection are held
fails by no bound: the pool drains and every page waits behind checkout
- 3Timeout fires: mark order pending-confirmation
records that we asked and do not know
fails by marking it failed: the provider may still charge; the customer is told the opposite of what happens
- 4Tell the customer honestly
"we are confirming your payment; you will receive an email"
fails by telling them it failed invites a second checkout and a second charge
- 5Reconcile: webhook arrives, or a worker re-checks by key
moves the order to paid or unpaid from the provider's answer
fails by no reconciliation: pending forever, and support finds out from the customer
The pending state and its exit are the design. The timeout is the trigger for them, not the fix on its own.
Which policy, for which crossing
Not every crossing deserves the full table's worth of machinery, and the decision below is the one to make per crossing before reaching for a breaker or a queue. The criterion is what the crossing holds while it waits and what it means if it never happens; the options are the ones a store actually uses.
How much failure machinery does this crossing need?
when a wait-for call the user can simply retry and nothing was committed — a shipping quote, a stock check on the product page
cost a slow provider makes the feature slow; there is no memory of the attempt
when a wait-for call where the outside system may complete after you stop waiting and money or stock moved — payment authorise
cost a new state, a worker, a support path and product copy for "we are confirming"
when fire-and-forget work that must eventually happen and is safe to repeat — emails, tracking updates, exports
cost the work is now asynchronous and the user is told "will", not "did"
when a wait-for call on a hot path where a failing provider would otherwise be hit by every request and every retry
cost a period where the store refuses the feature on purpose, and a decision about what the user sees during it
How to do it
Most important first.
- Take the boundary diagram and, for each crossing arrow, write the four rows. Do the wait-for crossings inside user-facing requests first; they are the ones that take the whole site down.
- Set every timeout from how long the user will wait, and make it visible where the call is made. A crossing without a timeout is an unbounded wait sitting in your request path (Which Dependency Must Answer Before the User Can Be Told Anything?).
- For every wait-for crossing, add a state that means "we asked and do not know the answer yet" to whatever record the call is about. Design the reconciliation for that state before it exists.
- For inbound messages, plan for duplicate, late and out-of-order delivery — the provider's retries are their failure handling, and they become your duplicates (Duplicate Requests).
- Inject each row on purpose against a working system: a sandbox that sleeps, a stub that returns an unknown field, a stub that answers "too many requests". Each should produce the designed behaviour, not a stack trace (Failure Injection).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Payment authorise, the four rows. Fails with a clean error: order stays unpaid, customer sees "payment declined or unavailable, try again", nothing charged. Slow past the user budget: the request returns "we are confirming your payment" and the order enters *pending-confirmation*; a worker re-checks the provider by our idempotency key, and the webhook can also resolve it; if the provider later says it went through, the order is created from the pending record, not from a second checkout. Changes: the adapter validates the response shape and fails loudly to us, not silently to the customer. Rate-limits: checkout is never retried in a loop; the customer is told to wait, and the worker backs off.
- Confirmation email, the four rows. Fails: the worker retries with backoff; the order is already placed, so the customer is not blocked and the page says "a confirmation is on its way". Slow: irrelevant, the worker is not on any user's path. Changes: the template API version is pinned and a test renders one email in CI. Rate-limits: the worker drains a queue at the provider's allowed rate; a sale day fills the queue rather than dropping emails.
- The site-wide outage, explained by the table. The payment call had no timeout and ran on the request thread; the slow row was undesigned; the connection pool was the shared resource that turned one slow crossing into every page. The fix is the slow row plus moving the shared resource out of the crossing's path — the timeout alone would have saved the site and still lost the charged-but-not-created orders.
How you know it worked
What now exists that did not before, and what question you can now ask.
- Every crossing has four decisions written next to it, and you can point at the code that implements each.
- A slow provider makes checkout slow and nothing else, and a customer who waited sees a page that admits uncertainty rather than one that lies in either direction.
- Injecting each failure against the sandbox produces the designed behaviour, and the test that does so is in the build.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?For this crossing, what happens when it fails, is slow, changes, and rate-limits — and can I point at the decision for each?
- ?When this call times out, does the outside system know something I do not, and how will I find out?
- ?What shared resource — thread, connection, lock — does this crossing hold while it waits, and who else needs it?
- ?Where does my system learn that the contract changed: from a loud adapter, or from a customer?
What can go wrong
- Every crossing gets a circuit breaker, a retry policy and a fallback on principle, including the fire-and-forget email and the image URL. The four rows are questions; for some crossings the honest answer to three of them is "nothing special", and adding machinery there hides the crossings where it matters.
- Retries are added to the fail row without noticing they belong to the rate-limit row too: a provider that is struggling receives three times the traffic from every client that wrapped its call in a retry, and the store becomes part of the outage.
- The pending state is added but never reconciled. Orders sit in pending-confirmation forever because the re-check worker was "phase two"; the state was designed and the path out of it was not.
- The rows are filled in from the provider's documentation rather than from an experiment. Documentation says the API returns within a bound; the sandbox, when told to sleep, says what your code actually does.
- Designing the four rows per crossing is a table's worth of decisions for every external system, and most of those decisions will be "nothing special" — which is only knowable after asking.
- A pending state is honest and it is more product: a screen, an email and a support process for "we are confirming your payment" that a store without the state does not need until the day it does.
- Timeouts from the user's budget can be shorter than the provider's typical latency at its worst, which means deliberately giving up on calls that would have succeeded; the trade is a bounded wait against a lower success rate under load.
- "So wrap every external call in retries with backoff." Retry only what is safe to repeat and only where you are not making a struggling provider worse; a checkout retried three times is three charges unless the idempotency key is in place first (One Retry per Tier Is Not One Retry — It Multiplies in the distributed domain shows the storm).
- "A timeout means the operation failed." A timeout means you stopped waiting. The other side may have finished; treating "unknown" as "failed" is how the charged-but-uncreated order happens.
- "This is defensive programming." Defensive programming guards against your own bugs. This is designing for the documented behaviour of a system you do not control, which is not a bug and will happen on schedule.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe four rows apply to every crossing in every system: a model provider called by an AI assistant, a package registry called by a build, a carrier API called by a pipeline. Which rows matter most changes per crossing; the questions do not.
- STAGE-SPECIFICIn a prototype against a sandbox, filling in the four rows for every crossing is premature; fill them in for the one crossing that touches money and note the rest. In production the table is the definition of done for adding any new external dependency.
- CONTESTEDThe strongest opposing view: most external failures should be handled by a generic infrastructure layer — a service mesh, a shared HTTP client with sane timeouts, retries and breakers configured once — so that application code stays free of failure logic and the policy is consistent instead of hand-written per crossing. That view is right for the fail and rate-limit rows of most crossings, and it is where large organisations end up; it cannot decide the slow row's user-facing outcome or the pending state, because those are product decisions per crossing and no shared client can make them.
- ILLUSTRATIVEThe outage, the elevated-latency afternoon and the charged-but-uncreated orders are invented; the four-row table for the payment crossing is one reasonable design and a real provider's idempotency and webhook semantics decide the details.
Where the depth lives
This domain asks the question and hands the answer off by name.