The question this answers
Both services are running. Why not just call the other one directly?
Once the broker acknowledges a publish, the message is durable to the broker’s configured replication level and will be delivered to each eligible consumer at least once. Nothing is guaranteed about *when*, about ordering beyond the broker’s ordering scope, or about the consumer succeeding.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The producer knows exactly one thing after a successful publish: the broker said it has the message. It does not know whether a consumer exists, whether one is running, whether the work has started, whether it succeeded, or how far behind the consumer is. Everything the producer believes about downstream progress is inference, and the broker is specifically designed so that it need not be true yet.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The dependency you remove, and the one you add
A synchronous call couples the caller’s availability to the callee’s. If orders calls email inline, then email being down makes checkout fail — even though nobody thinks emailing is part of taking money. The availability of the whole path is roughly the product of the availability of every hop, so each inline dependency multiplies your outage budget away.
A broker breaks that multiplication. orders writes to the broker and returns; email consumes when it can. Checkout now depends on the broker rather than on email. That is a real and large win — but note carefully what it is. You did not remove a dependency. You moved it to a component you chose because it is simpler to keep up than an arbitrary business service. A broker does one thing (accept and hand back bytes durably), which is why it can be made more available than the code that consumes them.
The cost is that the system is now *eventually* correct rather than immediately correct. Between publish and consume there is a window in which the order exists and the email does not. Every read path that assumed those happened together is now wrong, and finding those read paths is most of the work of adopting messaging.
Three different reasons to use a broker, often confused
People reach for messaging for at least three distinct reasons, and the right technology differs for each. Conflating them is how teams end up running four brokers, or one broker used four incompatible ways.
Naming which one you are buying tells you what to measure. If you adopted a broker for availability decoupling and then alert on queue depth as if it were a load problem, you will page someone for the exact behaviour you paid for.
- Availability decoupling — the consumer may be down and the producer must not care. Requires durability and retention, not speed.
- Load levelling — the arrival rate is spiky and the consumer’s capacity is flat. The queue is a buffer that converts a spike in rate into a spike in *latency*. Performance owns the arithmetic of that trade.
- Fan-out — several unrelated consumers need the same fact, and the producer should not have to know about them. This is a coupling-of-knowledge problem, not an availability one.
- A fourth, quieter reason: workload shape. A 45-second PDF render does not belong on an HTTP request thread regardless of any of the above.
| Property | Direct synchronous call | Through a durable broker |
|---|---|---|
| Caller availability depends onprotocol | The callee, live | The broker, live |
| Failure is reportedtypical | To the caller, immediately | Nowhere by default — you must build the path |
| Backpressure reaches the callerassumption | Yes: latency then errors | No: the queue absorbs it silently until it cannot |
| Orderingtypical | Caller controls it | Broker’s ordering scope only — often per-partition or none |
| Duplicate deliveryprotocol | Only if the caller retries | Expected, by design |
| Debuggabilitytypical | One trace, one stack | Two processes, a time gap, and a correlation id you must have added |
The publish itself is a distributed call
It is easy to treat publish() as if it were writing to a local list. It is a remote call, and it inherits A Timeout Tells You Nothing About Whether It Happened intact: a publish that times out may or may not have been persisted. Retrying it produces a duplicate; not retrying it may drop the event entirely.
This is why the hardest problem in messaging is not the broker at all — it is the atomicity between your database commit and your publish. There is no transaction spanning both. Commit the order and crash before publishing, and the event is lost forever with no error anywhere. Publish first and fail the commit, and you have announced an order that does not exist.
The standard resolution is to make the publish part of the same commit by writing the message into your own database as a row, and having a separate process read that table and publish it. That converts an impossible cross-system atomic write into a local transaction plus at-least-once delivery — which is exactly the trade this whole domain keeps making. See Atomicity Stops at the Process Boundary for the general shape and Architecture’s event-driven-architecture for the pattern framing.
1// BROKEN — two systems, no atomicity2db.transaction(() => {3 db.insert('orders', order) // committed4})5broker.publish('OrderPlaced', order) // crash here => event lost, silently6 7// FIXED — one transaction, then a separate at-least-once relay8db.transaction(() => {9 db.insert('orders', order)10 db.insert('outbox', { topic: 'OrderPlaced', payload: order, id: uuid() })11})12 13// relay process, runs forever, may publish the same row twice14for (const row of db.select('outbox').where('published_at IS NULL')) {15 broker.publish(row.topic, row.payload, { messageId: row.id })16 db.update('outbox', row.id, { published_at: now() }) // crash here => duplicate17}What you must build before you have finished adopting it
A broker is not deployed when the first message flows. It is deployed when the failure paths exist, because every one of them is silent by default. A synchronous call fails loudly into a caller who is already holding an error path; an asynchronous message fails into a log line nobody is reading.
The minimum viable async pipeline is: a durable publish, a consumer with explicit acknowledgement, a bounded retry policy, a dead-letter destination with an owner and an alert, and a backlog-age metric. Anything less is not a pipeline, it is a place messages go.
- A correlation id propagated from the originating request through the message into the consumer’s logs — without it, Three Nodes, Three Logs, and You Cannot Sort by Timestamp is impossible and every incident becomes archaeology.
- An alert on backlog age, not depth — a queue 10,000 deep and draining is healthy; a queue 3 deep and stuck for an hour is an outage. Performance’s
queue-ageis the reference. - An explicit answer to "what happens on the fifth failure", i.e. A Dead-Letter Queue Is a Workflow, Not a Bin with a human attached.
- Idempotent consumers, because at-least-once means duplicates are normal traffic, not an incident. See Idempotent Is a Property of the Whole Effect, Not the Write.
Key points
- A broker converts an availability dependency on an arbitrary service into a durability dependency on a purpose-built one; it does not remove the dependency.
- The producer’s only knowledge after a publish is that the broker accepted the bytes. Downstream progress is unobservable to it by design.
- Publish and database commit are not atomic. The outbox pattern replaces that impossible atomic write with a local transaction plus at-least-once relay.
- Async failures are silent by default; the DLQ, the retry policy and the backlog-age alert are part of the feature, not follow-up work.
- Decide which of availability-decoupling, load-levelling or fan-out you are buying — the answer determines what you measure and what you can safely lose.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The producer serialises an event and issues a publish, which is a remote call with its own timeout and its own ambiguity.
- • The broker persists the message — typically to disk and to a replica set — before acknowledging.
- • The ack returns. The producer’s obligation is now discharged; it may return to its own caller.
- • The broker holds the message until an eligible consumer takes it, subject to the broker’s retention or claim model.
- • A consumer receives, processes, and signals completion. Failure at any point returns the message for another attempt.
- • After a bounded number of attempts the message is routed to a dead-letter destination rather than retried forever.
- • The publish times out with the message already persisted — retrying duplicates it, not retrying may lose it.
- • The process commits its database transaction and dies before publishing at all.
- • The broker accepts the write but has not yet replicated it, and the accepting node dies.
- • No consumer is running and nobody notices, because the producer sees only successes.
- • The broker’s storage fills, and it begins rejecting publishes — turning a decoupled path back into a synchronous failure at the worst moment.
- • Silent event loss from the dual-write gap: the operator sees orders in the database with no corresponding downstream effect, and zero errors in either service. Discovered by a customer, or by reconciliation, never by a dashboard.
- • Unbounded backlog: the consumer deployment has been crash-looping for six hours. Producer error rate is 0%, customer-facing latency is normal, and the only symptom is queue age climbing linearly until retention expires and messages are dropped.
- • Broker saturation converts async back to sync: publishes start timing out under disk pressure, and the "decoupled" checkout path now fails because the broker is full of an unrelated topic’s backlog.
- • Duplicate side effects after a producer retry: the operator sees two identical events with different broker message ids, and two emails in the customer’s inbox.
- • Ordering surprise: two events published in order arrive out of order because they landed on different partitions or different consumers, and a downstream state machine ends up in an impossible state.
- • None between producer and consumer — that is precisely the point, and the source of every property above.
- • The broker itself coordinates internally: a durable publish waits for replication to a quorum of broker replicas before acknowledging, which is why publish latency has a floor set by Quorums: What R + W > N Does and Does Not Buy rather than by network round trip alone.
- • Cross-system atomicity between your store and the broker would require Two-Phase Commit: Buying Atomicity With a Promise, which is why nobody does it; the outbox is the coordination-avoiding alternative.
- • Messages acknowledged by the broker survive consumer outages for the full retention or until acked, whichever the broker’s model dictates.
- • Messages in flight at the moment of a producer crash, but not yet acked, have unknown status — exactly the ambiguity of A Timeout Tells You Nothing About Whether It Happened.
- • The system remains available to producers while consumers are entirely absent; correctness degrades to "will be true later", with no bound on later.
- • Detect: alert on backlog age and on consumer liveness separately. A dead consumer and a slow consumer need different responses and look identical on a depth graph.
- • Contain: scale consumers or shed lower-priority topics before the broker’s retention window becomes the limiting factor.
- • Recover: drain the backlog. Expect duplicate processing during the catch-up if consumers were killed mid-flight.
- • Reconcile: for the dual-write gap, run a periodic job comparing source-of-truth rows against emitted events and re-emit the missing ones. This is Reconciliation Is a Component, Not a Cleanup Script and it is not optional for money-adjacent flows.
- • Verify: confirm the backlog-age metric returned to baseline, and that the reconciliation delta is zero — not merely that the queue is empty.
- • Backlog age (oldest unprocessed message timestamp), per topic and per consumer group.
- • Publish latency and publish error rate at the producer — the early sign that the broker is becoming a synchronous dependency again.
- • Consumer processing rate against arrival rate; the derivative tells you drain time, the level does not.
- • Count of outbox rows with
published_at IS NULLolder than a threshold — the direct measure of the dual-write gap. - • End-to-end lag: event timestamp at production versus wall clock at completion, which is the number a product owner actually cares about.
- • The consumer’s work is not needed for the caller’s answer: emails, indexing, thumbnails, analytics, webhooks.
- • Arrival rate is spiky and the consumer’s capacity is expensive to scale quickly.
- • Several teams need the same fact and you do not want to redeploy the producer each time a new one appears.
- • The work is long enough that holding a request open is itself a scaling problem.
- • The caller needs the result to answer its own caller. An async call plus polling for the result is a synchronous call with more moving parts and worse latency.
- • The operation must be strictly ordered with respect to other operations and the broker cannot give you that ordering scope.
- • The team has no capacity to operate the failure paths. A broker with no DLQ workflow is a mechanism for losing data quietly, which is worse than failing loudly.
- • Volume is low and latency requirements are tight — you have added a hop and a poll interval to save nothing.
- • A direct synchronous call with a bounded timeout and a retry budget. Simpler, loudly failing, and correct when the callee is at least as available as the caller needs to be.
- • The transactional outbox with a direct HTTP delivery loop and no broker at all — durable and retryable, using only your existing database. Right at low volume with one consumer.
- • A scheduled batch job reading the source table. Higher latency, dramatically fewer failure modes, and often entirely acceptable for analytics or nightly reporting.
- • An in-process background task, when losing the work on a deploy is genuinely acceptable and you are willing to say so in writing.
Both services are up. Why not just call the other one?
| Caller latency | Caller sees | Work happens | Failure surfaces | Who notices | |
|---|---|---|---|---|---|
| Direct calltypical | 30 s (timeout) | 503 — connection refused | never | at the caller, immediately | the user, and your error rate |
| Via a brokertypical | 4 ms | 200 OK | when the consumer comes back | as queue age, hours later | nobody, until retention expires |
What people believe, and what is true
A broker decouples the services.
It decouples them in *time and availability*. They remain tightly coupled in *schema* — a breaking payload change fails just as hard, and now fails asynchronously in a consumer nobody is watching.
Async is faster.
Async is *lower latency for the caller* and usually higher end-to-end latency for the work. You moved the wait, you did not remove it.
The message is safe once publish() returns.
Only if the broker was configured to persist and replicate before acking, and only if the call actually returned — a timed-out publish has unknown status.
We can add the DLQ later.
Then between now and later, every permanently failing message is either retried forever (blocking the queue) or dropped (silent loss). "Later" is a decision to lose data.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A broker lets the producer finish without the consumer being alive. You trade immediate correctness for eventual correctness, and you inherit duplicates, delay, and silent failure paths you must build yourself.
Practical
Adopt messaging as a package: outbox for the publish, explicit acks on the consumer, a bounded retry policy, a dead-letter queue with an owner and an alert, correlation ids end to end, and a backlog-age SLO. Skipping any one of these produces a specific, predictable outage later.
Advanced
The deep reason a broker helps is that it changes what must be simultaneously available. A synchronous chain requires the conjunction of every hop; a brokered chain requires only the producer and the broker at publish time, and only the broker and the consumer at consume time. You have replaced a conjunction over N unreliable services with two conjunctions over two, at the cost of admitting an unbounded window in which your invariants do not hold. Whether that window is acceptable is a business question that engineers routinely answer silently by adopting a queue.
Apply it
- 🔧 Instrument an existing synchronous call path, then replace it with a broker and enumerate every read path that silently became eventually consistent.
- 🔧 Implement the outbox relay and deliberately kill it between publish and mark-published. Show that the consumer handles the duplicate without a visible effect.
- ⚡ A consumer deployment has been crash-looping for six hours over a weekend. Producer error rate is zero. Design the alert that would have caught it in five minutes.
- ⚡ The broker is at 95% disk. Two topics share it: one carries checkout events, one carries clickstream. What do you do first, and what does that tell you about topic isolation?
- 💬 Checkout calls the email service inline. Walk me through what a broker changes, and what new failure you have just signed up for.
- 💬 You commit an order and then publish an event. The process dies between the two. What does the customer see, and what does your dashboard see?
- 💬 Your queue has 200,000 messages in it. Is that an incident? What do you need to know first?