ReliabilityGENERALSCALE-SPECIFICDOMAIN-SPECIFIC

Partial Failure

A local operation is all-or-nothing because the language and the transaction say so. A distributed one is not, and an interface that returns one boolean for five sub-operations is lying about what happened.

Constraints & invariants Who owns what What the next change costs

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.

The question

This operation does four things across three systems. What should it return when the second one fails?

The requirement

completeOrder() charges the card, decrements stock, sends a confirmation email and writes an analytics event. The email provider is down. It returns false, and nobody can tell from that whether the customer was charged.

The obvious build

Do the four things in sequence inside one function. If any throws, return false and let the caller retry the whole thing. It reads well and mirrors how the business describes the flow.

Why it breaks

The failure of step three does not undo steps one and two, so false means "somewhere between zero and three of these happened" — the return value has lost the information the caller needs most.

How it breaks as requirements change

The failure of step three does not undo steps one and two, so false means "somewhere between zero and three of these happened" — the return value has lost the information the caller needs most.
Retrying the whole operation repeats the steps that already succeeded. Without idempotency the customer is charged again, so the obvious recovery makes it worse.
The steps are ordered by how the business narrates the flow, not by consequence, so the least reversible thing often happens first and the recoverable things fail after it (Designing the Happy Path Last).
It conflates importance. An analytics failure and a charge failure produce the same false, so either the caller treats a metrics blip as an order failure, or it treats a charge failure as ignorable.

Requirement→Constraints→Invariants→Responsibilities→Boundaries→Interfaces→State→Dependencies→Failure→Implementation→Tests→Feedback→Evolution

What limits the solution, and what must never stop being true

This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.

Constraints

The four systems cannot participate in one transaction, and adding a coordinator is not on the table (Two-Phase Commit: Buying Atomicity With a Promise in Distributed Systems).
The four steps have genuinely different importance: the charge is critical, the analytics event is not.
The caller is an HTTP handler with a customer waiting, so it cannot block until everything eventually succeeds.

Invariants

If the charge succeeded, the order exists. The reverse must never happen — an order that was never paid for.
The result of the operation must let the caller determine the state of each consequential step. A single boolean cannot do that (Result Types).
No step may be silently retried in a way that produces a second effect (Idempotency by Design).

Who owns what, and where the seams fall

Responsibilities decide boundaries; boundaries decide what an interface has to say.

Responsibilities

The operation owns reporting per-step outcomes, because it is the only place that knows them.
The design owns classifying each step: must-succeed, should-succeed-eventually, best-effort. That classification is a product decision that has to be made by someone (Requirements Before Design).
Something owns driving the eventual steps to completion — an outbox, a queue, a job. "The caller will retry" is not an owner (The Transactional Outbox in Backend).
The domain owns the invariant that survives every ordering: paid implies ordered.

Boundaries

The real boundary is the atomicity boundary: what can be committed together. Everything inside it is all-or-nothing; everything outside is a separate outcome that must be represented separately (Consistency Boundaries).
Draw the synchronous part around exactly what the customer must know now — the charge and the order — and push the rest behind an asynchronous boundary (Effect Boundaries).
The seam between critical and best-effort steps is a design line, and the failure of a best-effort step must not be able to cross it.

What "it failed" is hiding

The boolean is the bug. Four steps produce sixteen combinations of outcome, of which the caller genuinely needs to distinguish maybe four — and one bit cannot carry four states, so the information is destroyed at the return statement.

The table below is the set of situations a single false collapses together. Notice that the correct customer-facing response differs in every row.

Six ways `completeOrder()` returns false

Trigger	Symptom	Cause	Response
The card was declined	`false`	Nothing happened at all — the cleanest possible failure	Tell the customer to try another card. Safe to retry the whole operation
The charge timed out	`false`	The customer may or may not have been charged (Designing for Failure)	Do not tell them it failed. Record `UNKNOWN`, reconcile, and never retry without an idempotency key
The charge succeeded, stock decrement failed	`false`	Money taken, nothing reserved. The worst state in the table	Compensate — refund or reserve manually — and alert. This is the case that must never be silent
Charge and stock succeeded, email failed	`false`	The order is genuinely complete; a notification is missing	Return success and queue the email. Failing the order here is a self-inflicted outage
Everything succeeded, analytics write failed	`false`	A metrics dependency is degraded	Ignore it entirely, and say so in the code so the next reader does not "fix" it (Swallowed Errors)
The process crashed after the charge	No return value at all	No caller is left to interpret anything	Recovery must be driven by persisted state, not by the caller — which is why intent is written before acting

Classify the steps, then choose the mechanism

The refactor is not primarily structural. It is a classification exercise that someone from the business has to participate in, and once the three classes are named the mechanisms follow almost mechanically.

The key move is that the synchronous path shrinks to what the customer must know now. Everything else moves behind a durable queue written in the same transaction, so it cannot be lost and cannot fail the order.

One sequence, versus three classes

Four steps, one boolean

async function completeOrder(o: Order): Promise<boolean> {
  try {
    await payments.charge(o.total, o.card)
    await stock.decrement(o.lines)
    await email.sendConfirmation(o)
    await analytics.track('order_completed', o)
    return true
  } catch {
    return false          // which of the four? nobody can tell
  }
}

Classified, with the outbox in the same commit

async function completeOrder(o: Order): Promise<OrderOutcome> {
  // MUST: atomic with each other
  const charge = await payments.charge(o.commandId, o.total, o.card)
  if (charge.kind !== 'Charged') return { charge }

  return db.tx(async (t) => {
    await t.orders.create(o, charge.id)
    await t.stock.decrement(o.lines)
    // EVENTUAL: same commit, delivered later
    await t.outbox.add('SendConfirmation', { orderId: o.id })
    return { charge, order: 'Created', email: 'Queued' }
  })
  // BEST-EFFORT: not here at all. It is a consumer of the order event.
}

The availability of the operation is now the availability of the database and the payment provider, instead of the product of four dependencies. The email cannot fail the order because it is a row in the same commit rather than a call; the analytics write cannot fail anything because it is no longer in this code path at all. And the return value distinguishes the six rows in the table above, so the handler can say something true to the customer.

The classes, and what each one costs

SIMPLIFIEDThree classes is a teaching model. Real systems have gradations — a step that must happen within an hour, a step that must happen before month end, a step that must happen unless the customer cancels first — and each of those wants a different mechanism with a different deadline. The three classes are the minimum vocabulary that makes the conversation possible, not a complete taxonomy of durability requirements.

Each class buys something and gives something up, and the mistake is applying one mechanism uniformly — either everything synchronous, which makes availability collapse, or everything queued, which makes the customer's screen lie.

Class	Mechanism	Fails the operation?	What it costs
Must	One transaction, or an idempotent remote call whose intent is committed first	Yes — the operation is meaningless without it	Availability is the product of these dependencies, so keep the set as small as the domain allows
Eventual	A row in the outbox, written in the same commit, drained by a worker	No	The customer is told it succeeded before it has happened, and a backlog is now an operational concern (The Backlog Arithmetic: Four Levers and a Drain Time in Backend)
Best-effort	Fire and forget, failure explicitly ignored and commented	No	You will silently lose some of these, and must be genuinely willing to — otherwise it was never best-effort
Compensating	A forward action that undoes a prior one — refund, release, cancel	It runs because the operation failed	Visible to the customer, can itself fail, and needs its own idempotency (A Refund Is Not a Rollback in Distributed Systems)

How to build it

Most important first.

Classify every step before writing any of them: must (the operation is meaningless without it), eventual (must happen, not now), best-effort (nice to have). Three classes, three different mechanisms.
Make the must-steps atomic. Charge and order creation belong in one commit, which usually means recording the charge intent locally and letting the outbox drive the remote effect (Where the Transaction Boundary Goes in Backend).
Move eventual steps to a durable queue, written in the same transaction as the must-steps. The email is then guaranteed to be attempted without being able to fail the order (The Transactional Outbox in Backend).
Let best-effort steps fail silently, and say so in the code. An analytics write in a try/catch with a comment naming it best-effort is a design statement, not sloppiness (Swallowed Errors).
Return per-step outcomes, not a boolean. { charge: Charged, order: Created, email: Queued, analytics: Skipped } tells the caller what to say to the customer.
Order the steps so the least reversible thing happens last where you have the choice. Reserving stock before charging means a failed charge releases a reservation; charging before reserving means a refund (A Refund Is Not a Rollback in Distributed Systems).

What the next change costs

The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.

Cost of the next change

Under the naive design, "add a loyalty-points step" costs one more line in the sequence and one more way the whole operation can fail — availability degrades with every feature, and the tenth step makes the operation fail ten ways.
Under the classified design, the same change costs one more message on the outbox and no change to the synchronous path. Availability is unchanged, which is the property that matters as features accumulate.
The next change to *which* steps are critical — "we no longer need to charge before confirming, we invoice monthly" — costs moving one step between classes, because the classes are explicit in the code rather than implicit in the ordering (Explicit State).
What stays expensive: adding a step that must be atomic with the charge. That requires it to be in the same transaction, which means in the same database, which is a much bigger constraint than adding an eventual step (Consistency Boundaries).

What the recommended approach costs

Asynchronous steps mean the customer is told the order succeeded before the email is sent, so a failure is now visible to them later rather than immediately. Some products genuinely cannot accept that.
Per-step results are more verbose at every call site and force callers to think about combinations they would rather ignore.
An outbox is real infrastructure: a table, a poller, monitoring, and a new class of incident when it falls behind (Designing for Cost).

What can go wrong

Failure modes

Compensation is treated as rollback. A refund is a new transaction with its own failure modes, not an undo — it can itself fail, and it leaves a visible trace on the customer's statement.
The outbox write and the business write end up in different transactions, which reintroduces exactly the dual-write problem the outbox exists to remove.
Every step is classified as "must", so the operation fails whenever anything anywhere is degraded, and availability becomes the product of every dependency's availability (Cascading Failure: When the Response to Failure Causes More Failure in Distributed Systems).
Per-step results are returned and the caller ignores them, checking only whether an exception was thrown. The information is there and nobody reads it.

Dependencies, and their direction

The synchronous path depends only on the database and the payment provider. Everything else moves behind a queue, which converts a runtime dependency into a delivery dependency (Volatile Dependencies).
The queue becomes a dependency of correctness rather than of convenience, so its durability guarantees are now part of your design (Where You Put the Acknowledgement Decides Everything in Distributed Systems).
Compensating actions depend on the forward actions being identifiable, which means each needs an id you can refer to later (Stable Identifiers).

Misreads

"Use a distributed transaction." Two-phase commit trades availability for atomicity and does not work with third-party HTTP APIs at all. It is a real technique with a narrow home, and this is not it (Two-Phase Commit: Buying Atomicity With a Promise in Distributed Systems).
"Just retry the whole operation." Only if every step is idempotent. Otherwise the retry is what charges the customer twice (Idempotency by Design).
"Compensation is rollback." A refund is visible, delayed, and can fail. Designing as if it were an undo produces flows that assume compensation always succeeds (A Refund Is Not a Rollback in Distributed Systems).
"Return a boolean and log the details." The caller has to decide what to tell the customer, and it cannot read your logs (An Error Taxonomy That Survives Contact).

Smells this explains

boolean-flag-explosion

Testing it, and how it ages

What to test, and at which boundary

One test per failure position: fail step two, assert step one's effect is either committed or compensated and never dangling (Failure-Aware Feature Design).
A test that the best-effort step's failure does not fail the operation — with an assertion, because a try/catch that accidentally rethrows is invisible in review.
Crash between the commit and the queue consumer running, restart, assert the email is still sent exactly once (Where a Test Must Be Real).
A property that for any prefix of steps completing, the paid-implies-ordered invariant holds (Property-Based Testing).

How this design ages

The eventual set grows and the synchronous set stays small, which is the healthy direction: latency and availability both improve as features are added rather than degrading.
The compensation logic accumulates and eventually deserves to be modelled explicitly as a saga rather than as scattered catch blocks (Sagas: Trading Isolation for Availability in Distributed Systems).
Pressure eventually arrives to move a step from eventual back to synchronous — "the customer must see the confirmation immediately". That is a real requirement change and should be argued as one, because it is buying visibility with availability.

Where this applies

This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.

GENERALThat effects in separate systems cannot be committed together without a coordinator is a property of the systems rather than of a language, so the classification exercise applies to any multi-system operation.
SCALE-SPECIFICIn a single-process application with one database, all four steps can genuinely be one transaction and none of this applies — the naive design is correct and simpler. Everything here begins the moment one step leaves that transaction, which is why "we will split it later" is a much larger decision than it sounds (What Changes at the Network Boundary).
DOMAIN-SPECIFICThe classification is a business judgement and different domains classify the same step differently: an order confirmation email is best-effort for a retailer and legally required for a regulated financial product, which turns the same step from a fire-and-forget into a must-be-durable one with an audit trail.

Where the depth lives

This domain teaches the codebase-level structure and hands the rest off.

API Designpartial-failure error-model

Backendtransactional-outbox one-transaction-or-two external-calls-in-transactions transaction-boundary

Distributed Systemspartial-failure sagas compensation-is-not-rollback two-phase-commit cross-service-transactions

Domains that do not exist yet

— Testing & Reliability Engineering — fault injection at each step boundary, and verifying that compensation actually runs in production, is that domain's work. Here the subject is only what partial failure does to the operation's return type.
— System Design — whether these four steps should be one service or four is decided before this lesson applies, and it decides how much partial failure you have signed up for.