When the Provider Fails
Every bought capability will fail, be slow, change under you, or rate-limit you — not as a possibility, as a schedule. The build-vs-buy decision owes the design a paragraph for each, and the payment provider and the email provider need very different paragraphs.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
You have decided to buy payment and email. What does the store do while each of them is down, slow, changed or throttled — and which of those four does the design actually have to survive?
The provider integrations work. Then, on a Friday evening, the payment provider's status page turns orange, checkout starts timing out, and the order page shows a spinner. Nobody decided what the store should do in this case, so it is doing whatever the code happens to do — which is to leave orders pending and customers refreshing.
Treat the provider as reliable and handle its failure when it happens. Providers are, after all, more reliable than anything you could build, and designing for their failure feels like distrust of a decision you just made — and like work with no feature attached.
The provider is more reliable than your code and it still fails, on a schedule you do not control, at a time that is by definition inconvenient. "Handle it when it happens" means designing the failure path during the outage, in production, with customers waiting.
- The provider is more reliable than your code and it still fails, on a schedule you do not control, at a time that is by definition inconvenient. "Handle it when it happens" means designing the failure path during the outage, in production, with customers waiting.
- "Fails" is one word for four different things — down, slow, changed, throttled — and the design that survives one may not survive the others. A timeout is not an outage; a rate limit is not a timeout; a silent change to a webhook payload is none of them and breaks everything.
- The payment provider and the email provider fail in the same four ways and the store owes them completely different responses, because one is on the critical path and one is not. A design that has not decided which is which will either block orders on email or shrug at payment.
- The failure paragraph was owed by the build-vs-buy decision — it was the fifth question — and it was skipped, so the decision was made without knowing what it committed the store to.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Name the four failures separately and ask each of the provider: *down* (the call fails or never returns), *slow* (it returns, but after the customer gave up), *changed* (the API, the payload, the terms, the pricing), *rate-limited* (it refuses because you asked too often). Each is a different event with a different signature and a different response (External Systems Fail).
- Decide, per provider, whether it is on the critical path — whether the user's action can complete without it. Payment is: an order cannot become paid without the provider. Email is not: an order is an order whether or not the confirmation was sent. That one distinction decides most of the design (Which Dependency Must Answer Before the User Can Be Told Anything?).
- For a critical-path provider, design the *honest degraded state*: what the system says and stores when it cannot get an answer, so that nothing is lost and nothing is claimed. For payment that is a pending order, a truthful page, and a reconciliation path that asks the provider later (What If Payment Fails?).
- For an off-path provider, design so that its failure never reaches the user: the action completes, the side effect is queued and retried, and its absence is visible to operators rather than customers (Partial Failure).
- Write the four paragraphs down as part of the decision. The paragraph is the design; the build-vs-buy decision is not complete without it.
Four failures, two providers
The reflex has one word — "fails" — and one plan — "handle it". The table has four failures per provider, and the point is in the comparison of the columns: payment's responses are all about the order's state, and email's are all about a queue the customer never sees. The critical-path question made the columns different before any row was filled in.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Provider down | Payment: redirect cannot be created. Email: send call fails. | Outage on their side; nothing the store did. | Payment: keep the cart, tell the customer, create no order. Email: job retried later; order already committed. |
| Provider slow | Payment: confirmation late; customer back before it. Email: send takes seconds. | Their queue, their latency; your timeout decides what "slow" means. | Payment: pending state, truthful page, reconciliation asks later. Email: off the request, so nobody waited. |
| Provider changed | Payment: confirmation payload unparseable. Email: template rejected. | A versioned or unversioned change on their side; or a message that is not from them. | Payment: verify, reject, keep pending, alert on rejection count. Email: failed job to the operator queue, order untouched. |
| Provider rate-limits | Payment: reconciliation refused after an outage. Email: bulk resend refused. | Your retries, usually — a storm during their recovery. | Payment: poll with a budget and backoff. Email: queue drains at the allowed rate; nothing is lost. |
The critical path, drawn
The diagram is the whole design decision in one picture: the provider on the left of the commit can block the order; the provider on the right cannot. Where a provider sits relative to the commit is the answer to "what does the store owe its failure".
What the design owes each failure
The decision below is the one the build-vs-buy question deferred: not whether to buy, but what buying commits the store to. The options are the postures a design can take toward a provider, and the when is the critical-path question again.
What does the store do when this provider cannot answer?
when The provider is on the critical path and its answer is the truth about the order — payment.
cost A state, a page, a reconciliation job and a customer who sees "confirming" for a while. Nothing lost, nothing guessed.
when The provider is off the critical path — email, analytics, a recommendation service.
cost A queue, retries with a budget, and a side effect that can be late. The customer never waits; the operator sometimes does.
when The provider powers something optional — search ranking, image resizing — and a plainer version exists.
cost Two code paths, and a plainer experience that must be acceptable rather than broken (Graceful Degradation: Which Dependency Is Actually Critical).
when The action cannot proceed at all and pretending would be worse — the provider is down and no pending state makes sense, as with creating the redirect.
cost A lost conversion and a truthful message. The alternative is an order that claims something the store cannot know.
1every N minutes:2 for order in orders where status = pending and age > threshold:3 if budget.exhausted(): backoff(); break4 result = provider.lookup(order.reference) // timeout: short5 match result:6 paid -> markPaid(order) // idempotent, same path as webhook7 failed -> markFailed(order); releaseStock(order)8 unknown -> leave pending; count it9 throttled -> budget.spend(); backoff()The same idempotent path as the webhook, so an order confirmed both ways is paid once. The budget is what stops the store from being the reason the provider's recovery takes longer.
How to do it
Most important first.
- Put a timeout on every provider call and decide what the code does when it fires. A call with no timeout has decided that "slow" is the same as "down" and that both last forever.
- For the critical-path provider, make "I do not know yet" a first-class state — pending — with its own page text and its own resolution path. Never map an unknown result to success or failure.
- For the off-path provider, move the call off the request: after the order is committed, queue the email, retry it, and alert when retries run out. The customer never waits on it (Job Idempotency).
- Read the provider's changelog and versioning policy before integrating, pin what can be pinned, and verify every inbound message is genuinely theirs — "changed" includes "someone else is sending you payloads" (Webhook Signature Verification).
- Find the rate limits in the documentation and compute when the store would hit them — usually during a retry storm you caused, which is why retries need a budget (Cap Retries as a Fraction of Traffic, Not as a Count per Request).
- Inject each failure on purpose before it happens on its own: block the provider's host, add latency, send a malformed payload, return a rate-limit response (Failure Injection).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Payment, down: the redirect cannot be created. The store keeps the cart, shows "we cannot take payment right now, your cart is saved", and does not create a pending order for a redirect that never happened. Payment, slow: the confirmation is late. The return page says "confirming your payment", the order stays pending, and a scheduled reconciliation asks the provider about every pending order older than a threshold — some resolve to paid, some to failed, none are guessed.
- Payment, changed: the confirmation payload gains a field and renames another. Verification still passes; parsing fails; the handler rejects the message and the order stays pending — safe, because the degraded state was designed — and an alert fires because rejected confirmations are counted. Payment, rate-limited: reconciliation polls too often after an outage and gets refused. The poll has a budget and backs off; without one it would have turned the provider's recovery into a second outage.
- Email, all four: the order is committed first, the email is queued after, retried with backoff, and dropped into an operator queue when retries run out. Down, slow and throttled look identical to the customer — they see their order — and are visible to the team as a growing queue. Changed: a template rejection is a failed job, not a failed order. The same four failures, and nothing reached the critical path.
- The URL shortener, same move: the analytics provider for click counts is off-path — a redirect must never wait on it — so clicks are queued and counted later; a link-safety provider that checks destinations is on the critical path for *creation* and not for redirects, so creation degrades to "pending review" while existing links keep working. The four questions found two different paths in one small system.
How you know it worked
What now exists that did not before, and what question you can now ask.
- For each provider there are four written answers — down, slow, changed, throttled — and the payment ones and the email ones are different.
- The store has a pending state with its own page text and a reconciliation path, and no code maps "unknown" to "paid".
- Every provider call has a timeout and a decision attached to it, and the off-path calls happen after the commit, not during.
- Each failure has been injected on purpose at least once, and the store did what the paragraph said.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Is this provider on the critical path — can the user's action complete without it?
- ?What does the system say and store when the provider is down, when it is slow, when it has changed, and when it refuses us — four answers, not one?
- ?What is the honest state for "I do not know yet", and how does it get resolved later?
- ?Which calls happen before the commit and could be moved after it?
- ?Where is the retry budget, and what does the store do to the provider during the provider's own recovery?
- ?Has each of the four failures been injected on purpose, and did the store do what the paragraph said?
What can go wrong
- Every provider gets the full critical-path treatment — pending states, reconciliation, circuit breakers — including the email provider, whose failure the customer should never have been able to see. The move starts with the critical-path question so that most providers get the simpler answer.
- The degraded state is designed and never exercised, so when the outage comes the pending page has a typo and the reconciliation job was never scheduled. A failure path that has not been injected is a hypothesis.
- Retries are added without a budget, and the store's response to the provider's outage is to make it worse — for the provider and for itself. "Retry" is a decision that needs a limit and a backoff or it is a retry storm (Retry Storms: The Load You Generated Yourself).
- "Changed" is dismissed as unlikely because the provider is large. Providers version APIs, deprecate endpoints and alter payloads; the question is not whether but what the handler does with a message it does not recognise.
- A pending state and a reconciliation path are real work with no visible feature, and they are built for an outage that may be months away. The alternative is building them during the outage.
- Moving email off the request means the confirmation can be late, and a customer who did not get one will write in. That is the chosen cost of never blocking an order on email.
- Designing for "changed" means rejecting messages you do not recognise, which will occasionally reject a legitimate message after a benign change. Safe and annoying, by design.
- "Providers are reliable, so this is over-engineering." Providers are more reliable than your code and they still fail; the question is whether the store has an opinion when they do. "Don't over-engineer" is falsifiable here: made precise it says "do not build failure handling for a failure that cannot reach a user", and email is exactly that — which is why email gets the simple answer and payment does not.
- "Handle failure by retrying." Retrying handles "slow" and some of "down"; it does nothing for "changed", makes "rate-limited" worse, and turns an outage into a storm without a budget. Retry is one tool for one of the four failures.
- "This is the provider's responsibility." The provider owns its uptime. The store owns what its order says while the provider is unavailable, and no contract transfers that.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALDown, slow, changed and rate-limited are the four ways any external dependency fails — a provider, a partner API, another team's service — and the critical-path question sorts them for any system, not only a store.
- DOMAIN-SPECIFICIn a store the critical-path provider is payment and the degraded state is a pending order; in the chat app it is the realtime transport and the degraded state is "sent, not yet delivered"; in the AI assistant it is the model and the degraded state is an honest "I cannot answer right now" rather than a fabricated answer. The four failures are the same; what "pending" means is not.
- ILLUSTRATIVEThe Friday outage, the orange status page, the reconciliation threshold and the URL shortener's two providers are invented to show the four failures separating; no real provider incident is described.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — Testing & Reliability has no domain yet; the failure-injection practice here is its problem-solving half — inject each failure once, before it injects itself.