The question this answers
If services just react to each other’s events, what happens to the workflow — and to my ability to reason about it?
Each service guarantees that its local state change and its published event are atomic only if it publishes via a transactional outbox; without one, there is no guarantee relating the two at all. Globally there is no guarantee: no component knows the workflow’s state, no component can enforce its completion, and the set of reactions to any event is defined by whoever happens to be subscribed at that moment.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A participant knows the event it just consumed and its own local state. It does not know what step of what workflow it is in, what else consumed the same event, whether an earlier step has since been compensated, or whether anything downstream is listening to what it publishes. Crucially, nothing knows the *set* of consumers of an event — that is held by the broker as configuration, not by any service as knowledge, so nobody can reason about the consequences of publishing.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The shape: events all the way down
Orders publishes OrderCreated. Payments subscribes, charges the card, publishes PaymentCaptured. Inventory subscribes to that, reserves stock, publishes StockReserved. Shipping subscribes to that. No component ever says "next, do X" — each simply reacts. The workflow is real, and it is written nowhere.
The coupling is genuinely different, not merely relocated. In orchestration, participants are coupled to the orchestrator. Here, publishers are coupled to nothing at all — Orders neither knows nor cares that Payments exists — while consumers are coupled to the *event schema*. Adding a new reaction (fraud scoring, analytics, a loyalty service) requires deploying one new subscriber and changing nothing else. For teams that value independent deployment above global legibility, that is a strong argument.
It is also the model that fits when the reactions are genuinely independent and optional. Search indexing, analytics, notifications, cache invalidation and audit logging are all things that should happen when an order is created and none of which the order flow should have to know about. Choreographing those while orchestrating the transactional core is usually the right hybrid.
The outbox is not optional here
In choreography, publishing an event is not a side effect of the work — it *is* the mechanism by which the workflow continues. A lost event does not degrade the system; it silently halts one instance of the workflow forever, with no error anywhere.
So the dual-write problem from Atomicity Stops at the Process Boundary becomes existential. "Commit the row, then publish" loses events when the publish fails after the commit. "Publish, then commit" emits events for work that rolled back, and downstream services act on state that does not exist. Neither is acceptable when the event carries the workflow forward.
The transactional outbox is the standard answer: write the event into a table inside the same local transaction as the state change, and have a relay publish from that table at-least-once. Consumers must therefore be idempotent — which they had to be anyway (Idempotent Is a Property of the Whole Effect, Not the Write) — and now the atomicity that matters is local, which is the only kind you can actually have.
A second, subtler requirement follows: the event must carry everything a consumer needs. A consumer that receives OrderCreated and then calls back to the order service to fetch details has reintroduced a synchronous dependency and a race — the order may have changed, or been compensated, between the event and the fetch. Events that carry state avoid this; events that carry only an id do not.
The dependency graph nobody can see
Ask an orchestrated system "what happens when an order is placed?" and you read one state machine. Ask a choreographed system and there is no correct place to look. The answer is the transitive closure of the subscription graph, which lives in broker configuration, consumer group registrations and deployment manifests across every team.
Two properties make this worse over time. First, the graph changes without any code review touching the publisher. A team deploys a new subscriber to PaymentCaptured; the payments team is not consulted and does not learn. Behaviour of the whole system changed, and the change is invisible in the diff of any service that existed before. Second, cycles form silently. Service A reacts to A’s own downstream effect through two intermediaries; nobody drew the loop because nobody drew anything. The symptom is an event storm with a rising multiplier, and it is very hard to attribute because each service is behaving exactly as designed.
This is the honest cost, and it is not a tooling gap you can close entirely. Event catalogues, schema registries and consumer-driven contracts help you know who *subscribes*; they do not tell you the runtime shape of a workflow, and they cannot answer "where is order 4821 and why has it not shipped?".
Compensation without a compensator
In orchestration, the failure path is code someone wrote: "if step 3 fails, run C2 then C1". In choreography it is another subscription. Inventory publishes StockReservationFailed; Payments must be subscribed to it and must know to refund; Orders must be subscribed and must know to cancel. The unwinding is as emergent as the forward path.
That creates a specific and common bug: the forward path is exercised constantly and the compensating path almost never, so a missing or broken compensation subscription can exist for months without anyone noticing. The forward flow works. The failure flow silently does nothing. You discover it during an incident, which is the worst possible time.
It also makes the ordering discipline from A Refund Is Not a Rollback harder to enforce. There is no single place that knows the pivot, so "do not send the confirmation email until after the pivot" is a rule each subscriber must independently respect — and a new subscriber added by another team has no way to know the rule exists.
- Test the failure path deliberately — inject step failures in a staging environment and assert that the compensating events are consumed, not merely published.
- Assert on subscription existence — a contract test that fails when a compensating event has no consumer catches the most expensive class of bug here.
- Publish failure events with the same rigour as success events — through the outbox, with retries, and with a DLQ.
- Give every event a saga correlation id — without it, reconstructing an instance from logs is impossible rather than merely tedious.
The honest comparison
Choreography is better at *local change* and worse at *global reasoning*. Orchestration is the reverse. Neither is a winner, and the choice should follow from which of those two properties your organisation is currently short of.
Choose choreography when the reactions are genuinely independent, when the number of steps is small, when the teams are autonomous and the workflow is not a business object with an owner. Choose orchestration when someone is accountable for the workflow end to end, when the failure handling is complex, or when support staff must answer questions about individual instances.
And notice the argument that does not work: "choreography is more scalable because it has no central component". The broker is central. Its partitions, consumer groups and retention are shared infrastructure with shared failure modes (A Topic Is Not One Log: Ordering Lives Inside a Partition, Consumer Groups: Queue Semantics Inside, Pub/Sub Semantics Across). What choreography decentralises is the *logic*, not the *infrastructure* — and the logic was rarely the scaling constraint.
| If this matters most… | Choose | Because |
|---|---|---|
| Answering "where is instance X?"protocol | Orchestration | The state is one queryable row |
| Teams deploying independentlytypical | Choreography | A new reaction changes no existing service |
| Complex compensation logictypical | Orchestration | Written once, in one place, testable |
| Many optional side-reactionstypical | Choreography | Publishers should not know about them |
| End-to-end SLA with an ownerassumption | Orchestration | Accountability needs a place to live |
| Incident MTTRtypical | Orchestration | Emergent workflows are slow to diagnose under pressure |
| Avoiding a shared release bottlenecktypical | Choreography | No component that everyone must change |
Key points
- Services react to events; the workflow is emergent and exists in no artefact.
- Adding a reaction requires deploying one subscriber and changing nothing upstream — the central benefit.
- A transactional outbox is mandatory, because a lost event does not degrade the workflow, it halts that instance permanently and silently.
- The subscription graph changes without touching any existing service’s code, so behaviour changes are invisible in diffs.
- Cycles form silently and produce event storms that are hard to attribute.
- Compensating paths are subscriptions too, and are almost never exercised, so they rot undetected.
- The broker is still a central component; choreography decentralises logic, not infrastructure.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A service performs a local transaction and, in the same transaction, writes an event to its outbox table.
- • A relay publishes outbox rows to the broker at-least-once and marks them sent.
- • Zero or more subscribers consume the event. The publisher does not know how many, or which.
- • Each subscriber performs its own local transaction and, in the same transaction, writes its own outbox event.
- • The workflow advances by repetition of this, with no component tracking the whole.
- • Failures publish failure events; whichever services subscribe to them perform compensating local transactions.
- • The instance is "complete" when no further events are produced — a condition no component evaluates.
- • An event is published with no subscriber, and the workflow instance stops there permanently.
- • A consumer is deployed with a subtly different schema expectation and silently drops or mis-parses events.
- • The dual write is done without an outbox, so events are lost after commit or emitted for rolled-back work.
- • Events are redelivered, so a non-idempotent consumer performs the step twice.
- • Events arrive out of order relative to causality, so a consumer acts on a later state before an earlier one (Happens-Before: The Only Ordering You Actually Have).
- • A cycle forms and each traversal multiplies the event count.
- • A compensating event has no consumer, so failures never unwind.
- • Orders stuck at step 3 after a refactor renamed an event: the publisher publishes happily, no consumer exists, and every service reports 100% success. The only symptom is a business metric — a funnel step whose conversion silently dropped to zero.
- • Event storm from an accidental cycle: broker throughput rises by an order of magnitude, consumer lag grows everywhere, and each service’s logs show it doing exactly what it is supposed to. Attribution requires reconstructing the cycle by hand.
- • Compensation never runs: the failure path was published for months but never consumed. Discovered during an incident, when refunds that were assumed automatic turn out never to have been issued.
- • Duplicate side effects after a consumer rebalance: a deploy triggers a rebalance, messages are redelivered from the last committed offset, and a non-idempotent handler emails every affected customer twice (Rebalancing: Everyone Stops So the Partitions Can Move).
- • Unanswerable support question: "why has this order not shipped?" requires joining six services’ logs by correlation id, and takes forty minutes per instance. The operator’s experience of this failure mode is the absence of any place to look.
- • Silent outbox stall: the relay dies and nobody notices, because publishing shows no errors — the rows simply accumulate. The workflow stops globally while every service reports healthy.
- • No synchronous coordination between services at all — the strongest availability property of the model.
- • Coordination is displaced into the broker: ordering guarantees, partition assignment and consumer group membership are all coordination, operated by someone else (Consumer Groups: Queue Semantics Inside, Pub/Sub Semantics Across).
- • Schema agreement is coordination at design time rather than run time: publishers and consumers must agree on event structure without a runtime enforcement point.
- • The absence of a coordination point is exactly why nobody can answer global questions; legibility and coordination are the same resource here.
- • Each service remains internally consistent; local transactions and outbox rows commit together.
- • A workflow instance can halt permanently with no error signal, and nothing detects it except a business-level metric.
- • Events already published remain durable in the log and can be replayed, which is the model’s strongest recovery property.
- • Compensation depends entirely on subscriptions existing; if one does not, the failure path simply does not exist.
- • Detect: monitor the business funnel, not the services. The signature failure here is a step whose count drops while every service reports health.
- • Contain: pause the relevant consumer group rather than the publishers, so events accumulate in the log rather than being lost while you fix the consumer.
- • Recover: replay from the log. Because events are retained, a fixed or newly deployed consumer can reprocess history — the single biggest operational advantage of an event-log-based design.
- • Reconcile: use the saga correlation id to join each service’s records and find instances that stopped mid-flow, then re-drive or compensate them explicitly.
- • Verify: assert, in a test, that every published event type has at least one consumer, and that every failure event has a compensating consumer. This catches the dominant failure mode before deploy.
- • Per-event-type publish and consume rates side by side; a published type whose consumption rate is zero is a broken workflow, and nothing else will tell you.
- • Business funnel counts per workflow step, which is the only end-to-end liveness signal choreography offers.
- • Consumer lag per group, and DLQ depth per consumer (A Dead-Letter Queue Is a Workflow, Not a Bin).
- • Outbox depth and oldest unsent row per service — a stalled relay is silent otherwise.
- • Event amplification factor: events published per business operation. A rising value is how a cycle announces itself.
- • Age of the oldest workflow instance that has not reached a terminal business state, computed by a reconciliation job because no component tracks it.
- • Genuinely independent reactions — analytics, search indexing, notifications, audit — where the publisher should not know the consumer exists.
- • Small workflows of two or three steps, where the emergent graph is still comprehensible.
- • Autonomous teams that need to add behaviour without coordinating a release with another team.
- • Systems built on a retained event log, where replay makes recovery from a broken consumer genuinely easy.
- • As the outer layer of a hybrid, with an orchestrated transactional core publishing events that anyone may react to.
- • Workflows with a business owner, an SLA, or a support process that must answer per-instance questions.
- • Long chains, where the emergent graph exceeds what anyone can hold in their head and cycles become likely.
- • Complex compensation, where the failure path is many subscriptions that are never exercised.
- • Incident response, where "which service is responsible for this stuck order?" has no answer and MTTR suffers accordingly.
- • Teams new to event-driven systems, who will build the dual write rather than the outbox and lose events for months before noticing.
- • Orchestration, when the workflow deserves an owner (Orchestration: One Component Owns the Workflow).
- • A hybrid: orchestrate the transactional core, choreograph the optional reactions. Usually the right answer for a real system.
- • Event sourcing with projections, when the requirement is really "reconstruct the history" rather than "coordinate the steps".
- • A workflow engine that consumes events but maintains explicit per-instance state — choreographed inputs, orchestrated state.
- • Collapse the services, when the chain exists only because the boundary was drawn in the wrong place (Four Questions That Test a Proposed Boundary).
Services react to each other's events, and nobody holds the workflow
| Reacting service | Publishes | Status | |
|---|---|---|---|
| order.createdtypical | Payments | payment.captured | live |
| payment.capturedtypical | Inventory | stock.reserved | live |
| stock.reservedtypical | Shipping | shipment.created | live |
| shipment.createdtypical | Notifications | — terminal | live |
| payment.failedtypical | Orders | order.cancelled | live |
| order.cancelledtypical | Inventory | — terminal | live |
| stock.failedtypical | Payments | refund.issued | live |
| refund.issuedtypical | Orders | order.cancelled | live |
What people believe, and what is true
Choreography has no single point of failure.
The broker is one. You moved the central component from something that understands your workflow to something that does not.
Choreography is more scalable than orchestration.
Both scale by partitioning. What choreography removes is a shared *deployment* bottleneck, not a throughput one — and the orchestrator was rarely the throughput limit.
Services are decoupled because they only exchange events.
Consumers are tightly coupled to event schemas and to the semantics behind them. The coupling is real; it is simply not visible in any call graph.
If the workflow needs changing, we just add a subscriber.
And you have changed system behaviour with no review by anyone who owns the publisher, no test that covers the new path, and no artefact recording that the path exists.
We publish the event after committing, which is fine because publishing rarely fails.
When it does fail, that workflow instance halts forever with no error. Rare and silent is the worst combination; it means the bug accumulates for months before anyone counts.
Event ordering from the broker means the workflow steps are ordered.
Ordering holds within a partition. Two events on different partitions, or the same logical entity keyed differently, can be processed in any order (A Topic Is Not One Log: Ordering Lives Inside a Partition).
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Each service reacts to events and publishes its own. Nobody owns the workflow, which makes it easy to extend and hard to understand.
Practical
Publish through a transactional outbox, always. Put a saga correlation id on every event. Monitor publish rate against consume rate per event type, and alert when a published type has no consumer. Test the failure path explicitly, because it is the path that rots. Watch the business funnel, because no service-level metric shows a halted workflow.
Advanced
The real cost is that the dependency graph is runtime configuration rather than code, so it changes without review and can develop cycles nobody drew. Treat the event catalogue as an architectural artefact with owners, require a consumer to declare its subscriptions in a reviewable form, and compute the transitive closure in CI so a new subscription that closes a loop is caught before deploy. That is the closest choreography gets to the legibility orchestration has by construction.
Apply it
- 🔧 Take a choreographed flow and compute the transitive closure of its subscriptions. Compare the result with what the team believed the workflow was.
- 🔧 Remove a compensating subscription in a staging environment, run the failure path, and see how long it takes anyone to notice. Then build the assertion that catches it.
- 🔧 Introduce a deliberate cycle in a test environment and measure the event amplification factor over time.
- ⚡ Broker throughput rose 12× overnight with no traffic increase. Every consumer is behaving as designed. Find the cause.
- ⚡ Support cannot answer why an order has not shipped. Design the minimum you would add to make that question answerable without abandoning choreography.
- 💬 What guarantees does a choreographed saga give, and what does it give none of?
- 💬 Why is a transactional outbox mandatory in choreography but merely advisable elsewhere?
- 💬 A workflow silently stopped completing after a refactor. Every service reports healthy. What happened and what metric would have caught it?
- 💬 How would you find out, in a choreographed system, what happens when an order is created?
- 💬 When would you deliberately choose choreography for the transactional core rather than just for side reactions?