There Is No Transaction Across APIs
One request that charges payment, reserves inventory and books shipping cannot be atomic — the ACID boundary died at the first network hop. What replaces it is a contract that models the in-between states: workflows, state machines and compensation.
Frame the contract
API design starts with a consumer, a design question and a guarantee — never with a URL.
Where atomicity actually ends
Inside one database, a transaction gives you the full bargain: all-or-nothing, invisible intermediate states, rollback on failure (Transactions and ACID). The bargain is enforceable because one engine owns all the state and the locks. The moment your handler calls the payment provider's API, then your inventory service, then the shipping partner, no such engine exists. Each call commits independently in a system you do not control; between calls, the process can crash, time out, or get a response that was lost in transit (Idempotency).
So POST /checkout returning 500 after step two tells the caller almost nothing: payment charged, inventory reserved, shipping never booked — is that a failure? A success minus one step? The honest answer is that it is a *state*, and the API that pretends the operation is atomic has no name for it. The caller retries (duplicating charges), or gives up (stranding the reservation), or calls support (stranding an engineer). Two-phase commit — the textbook fix — is unavailable in practice: your payment provider and shipping partner do not expose prepare/commit hooks to you, and even where all participants are yours, 2PC couples the availability of everything to everything (Distributed Transactions covers why it is rarely deployed across service boundaries).
The replacement: a process the caller can see
What replaces the transaction is not a cleverer request — it is a different contract shape. Model the multi-system operation as a resource with a state machine: POST /checkouts creates it and returns 202 with an id; the checkout then moves through payment_pending → paid → inventory_reserved → shipped, observable via GET /checkouts/{id} or events (The Async Job Pattern is this shape in general form; Resources Have State Machines governs the transitions). The partial states stop being undefined middles and become documented positions a caller can poll, display, and reason about.
Failure handling becomes compensation: shipping failed after payment and reservation, so the workflow releases the reservation and refunds the charge — forward steps undone by explicit reverse steps, since rollback does not exist. This is the saga pattern (Saga Pattern), and its contract-level consequence is that *compensation states are states too*: refund_pending, refunded, failed_after_payment. A consumer building a checkout UI needs to render them; hiding them produces support tickets shaped like "it says failed but I was charged".
Every step of the workflow needs Idempotency Keys: The Mechanism on its outbound calls — the workflow engine retries steps, and a retried charge without a key is the double-payment bug relocated one layer down. The workflow's own creation endpoint needs a key too: the client that timed out creating the checkout must be able to retry into the same workflow, not mint a second one.
1POST /checkout2→ 200 { "order": "ord_1", "status": "complete" }3 # …when all three calls succeed4→ 500 { "error": "internal error" }5 # …when any step fails — charged? reserved?6 # the contract has no vocabulary for it7 8# Caller's options on 500:9# retry → possible double charge10# abort → stranded reservation, angry customer11# ask support → stranded engineer1POST /checkouts Idempotency-Key: co_7f…2→ 202 { "id": "co_1", "state": "payment_pending" }3 4GET /checkouts/co_15→ { "state": "inventory_reserved",6 "steps": {7 "payment": "succeeded",8 "inventory": "succeeded",9 "shipping": "in_progress" } }10 11# shipping fails → compensation is visible:12→ { "state": "refund_pending", … }13→ { "state": "failed", "resolution": "refunded" }The first contract promises an atomicity nobody can deliver and hands every failure to the caller as an undefined middle. The second promises less — no atomicity — and delivers more: every state the system can actually be in has a name, an owner, and a next step.
Deciding when the machinery is worth it
The workflow shape is heavy: state persistence, a driver that advances and compensates, more endpoints, more documentation. Before reaching for it, try to shrink the problem. Can the boundary move so one system owns the whole invariant — inventory and orders in one service, one database, one real transaction? Can steps become *reservations with expiry* (holds that auto-release) so failure needs no active compensation? Can the operation tolerate deferred consistency — charge now, fulfill by queue, reconcile mismatches nightly? Each of these deletes distributed-transaction machinery instead of managing it.
The machinery earns its cost when the middle states are user-visible and long (minutes to days), when steps cross ownership boundaries you cannot merge (external providers), and when partial failure has money attached. That is checkout, onboarding, provisioning — the flagship flows. For a two-step internal write where the second step is retryable and invisible, an outbox and a queue with at-least-once delivery (Message Queues) plus idempotent consumers is the entire answer, and no caller ever needs to see a state machine.
- Merge the boundary — if one service can own the invariant, a real transaction returns; the best saga is the one you deleted (API Granularity and the Chatty API).
- Reservations with expiry — holds that self-release convert compensation from an action you must run into a timeout you already wrote.
- Outbox + queue — for invisible, retryable second steps: commit locally, relay the event at-least-once, consume idempotently.
- Full workflow resource — for visible, multi-owner, money-bearing processes: named states, compensation states included, idempotency keys throughout.
- Never — a
500whose meaning is "somewhere between charged and shipped".
Key points
- Atomicity ends at the first network hop: each downstream call commits independently, and no rollback spans them.
- A 500 from a multi-system operation without modeled states is an undefined middle — the caller cannot safely retry, abort, or even describe what happened.
- The replacement contract is a process resource with a state machine: partial states, compensation states and terminal resolutions all have names.
- Compensation is forward action, not rollback — and its states (refund_pending, failed_after_payment) are part of the public contract.
- Every workflow step needs idempotency on its outbound calls, and the workflow creation itself needs a key so retries join rather than duplicate.
- Reach for lighter shapes first: merged boundaries, expiring reservations, outbox-and-queue — the workflow machinery is for visible, multi-owner, money-bearing flows.
Follow the failure
How the contract fails or gets misused, hop by hop — and what it costs when it completes.
- 1Team → API: wraps three service calls in one
POST /checkouthandler; the happy path works and ships. - 2Shipping partner → API: times out during a promotion; the handler returns
500after payment and reservation committed. - 3Client → API: retries the 500 — the contract never said not to — and creates a second charge and a second reservation.
- 4Support → databases: engineers reconcile charges, holds and orders across three systems by hand, per incident.
- 5Team → handler: adds ad-hoc cleanup code inside the request path (refund-on-catch), which itself fails on the next timeout, leaving cleanup half-done too.
- Customers are charged for orders that don't exist, or hold reservations that never release — each incident is a manual, multi-system reconciliation.
- Callers cannot build correct retry logic because the contract cannot tell them what a failure left behind.
- The operator's view fragments: no single record says where each in-flight operation stands, so incidents start with archaeology across three systems' logs.
Design, observe, evolve
A contract decision is incomplete until you know how you would notice it failing and how it changes later.
- • Model any multi-system operation with visible middles as a resource with a documented state machine — including compensation and terminal failure states.
- • Accept an idempotency key at workflow creation and use per-step keys on all outbound calls, so retries at every layer converge instead of duplicating.
- • Prefer reservations with expiry over compensating actions where the domain allows — passive cleanup beats active cleanup that can itself fail.
- • Shrink first: merge boundaries or use outbox-and-queue for invisible steps before deploying full workflow machinery.
- • Dashboard workflow states as a population: counts per state, age percentiles per state — a growing `refund_pending` cohort is an incident announcing itself.
- • Alert on stuck workflows (age in a non-terminal state beyond SLO) rather than on step errors alone; retries mask step errors, but stuckness is the real symptom.
- • Reconcile terminal states against downstream truth (charges settled vs `failed`+`refunded` workflows) to catch compensation that silently failed.
- • New steps extend the state machine additively as long as existing states keep their meaning; clients that render unknown states generically absorb them without release-day coupling ([[enum-evolution]]).
- • A synchronous façade can be layered over the workflow later (wait up to N seconds, then return the resource in whatever state it reached) without changing the underlying contract ([[long-running-operations]]).
- • The workflow contract is bigger in every dimension: more endpoints, more states to document, more client code — the price of naming states that previously hid inside a 500.
- • Compensation logic is real engineering with its own failure modes; a saga is not simpler than a transaction, it is merely possible where a transaction is not.
- • Exposing partial states binds you to them: consumers build UI on `refund_pending`, and renaming it later is a breaking change like any other ([[backward-compatibility]]).