Synchronous vs Asynchronous Communication
HTTP couples availability and returns an answer; messaging decouples availability and returns a promise. Neither is inherently more scalable.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Should this component call that one and wait, or publish a message and move on?
Placing an order must reserve stock, charge a card, send a confirmation email, update the search index and notify the warehouse. Some of those the customer is waiting for; most of them they are not.
Call each dependency over HTTP in sequence inside the request handler. The code reads top to bottom, errors surface immediately, and the response tells the client exactly what happened.
The request now takes as long as the slowest dependency, every time, and its availability is the product of all of them. Five dependencies at 99.9% is roughly 99.5% for the endpoint, which is a different SLO than anyone agreed to.
- The request now takes as long as the slowest dependency, every time, and its availability is the product of all of them. Five dependencies at 99.9% is roughly 99.5% for the endpoint, which is a different SLO than anyone agreed to.
- A dependency being slow rather than down is worse: handlers hold connections and workers while they wait, and the pool fills up. The email provider being slow takes down checkout (Failure Propagation).
- Halfway through, the card is charged and the email fails. There is no transaction across the five, so the handler must decide what "partially done" means, and usually it does not (The Dual Write Problem).
- A traffic spike is passed straight through to every dependency at full amplitude, including the one that rate-limits you (Rate Limiting).
What is actually happening
- Synchronous (HTTP, gRPC): the caller blocks until the callee answers. You get an immediate result, an error attributable to a specific call, ordinary control flow and a stack trace. You also get temporal coupling — both must be up, at the same time, right now.
- Asynchronous (queue, message bus, event log): the caller hands off durably and continues. The producer and consumer no longer need to be available at the same time, and the queue absorbs bursts. You give up the answer, immediate error attribution, and ordering (Job Queues, Queue Semantics).
- The queue does not make the work faster or smaller. The same rows still get written by the same database. What moves is when the work happens and who waits for it — which is a latency and coupling change, not a throughput one.
- Asynchronous delivery is at-least-once in every practical system, so consumers must be idempotent. "Exactly-once delivery" is a queue-level marketing claim; exactly-once *effect* is something your consumer implements (At-Least-Once Delivery, Job Idempotency).
- Async moves the failure surface rather than removing it: backlog growth, consumer lag, poison messages, dead-letter queues, and out-of-order processing are all new things to operate (Queue Backlog, Dead-Letter Queues).
- The deciding question is almost always: does the caller need the result to complete its own job? If yes, it is synchronous, whatever else you would prefer. If no, making it synchronous is coupling you chose by accident.
One operation, split by who is waiting
The productive move is not choosing a style for the system. It is splitting a single operation by whether the customer's answer depends on each step. In checkout, exactly one step changes what you can say in the response.
Note what the synchronous half keeps: if stock reservation fails, the customer is told immediately and no order exists. And note what the asynchronous half accepts: if the email consumer is down for an hour, the emails go out an hour late and nothing else is affected.
- 1Validate + authorize
In process, immediate.
fails by 400 or 403 to the client, nothing else happens.
- 2Reserve stock (sync)
The customer cannot be told "ordered" without it.
fails by Timeout or rejection becomes a 409 or 503; bounded, retried with an idempotency key, circuit-protected.
- 3Write order + outbox (one transaction)
Commits the state change and the intent to publish atomically.
fails by Rolls back together — no order, no events, no partial state (The Transactional Outbox).
- 4Respond 201
Tells the client what is true now and what is pending.
fails by Client disconnect after commit — the retry must be idempotent (Idempotency Keys).
- 5Publish OrderPlaced (async)
Relays the outbox to the broker.
fails by Relay lag; duplicate publication, which consumers must tolerate.
- 6Charge, email, index, notify (async)
Independent consumers, each retried and dead-lettered on its own.
fails by One consumer down delays only its own effect; billing failure surfaces as an order state change, not a lost response.
What each style actually gives you
Read this as a list of properties to choose between rather than a scoreboard. Most rows favour one side, and the row that decides is always the first: whether the caller can proceed without the answer.
The availability row is the one worth computing before a design review. Synchronous dependencies multiply; asynchronous ones do not, which is the strongest single argument for the async half of a split — and it says nothing about throughput.
| Property | Synchronous (HTTP/RPC) | Asynchronous (queue/event) |
|---|---|---|
| Caller gets a result | Yes, immediately | No — an acknowledgement, and a status to check later |
| Availability coupling | Multiplies down the chain | Producer and consumer are independent |
| Latency of the operation | Response waits for all of it | Response is fast; completion is later, and often overall slower |
| Burst handling | Passed straight through at full amplitude | Absorbed by the queue, up to its bound (Backpressure) |
| Sustained throughput | Limited by dependency capacity | Limited by consumer capacity — identical, and the queue does not change it |
| Error attribution | Immediate, to a specific call | Deferred; needs retries, dead-letters and correlation ids |
| Ordering | Natural, by control flow | Not guaranteed without partitioning by key |
| Duplicates | Only on retry | Normal operating condition — consumers must be idempotent |
| Debugging one operation | One trace, one stack | A trace spanning producer, broker and consumer, if propagated |
| Infrastructure to operate | The callee | The callee, the broker, the dead-letter queue, the replay path |
Choosing, per interaction
The decision is made per call, not per system, and the first option below covers most cases correctly. The rest exist because "the caller needs an answer" and "the caller needs an answer *within this request*" are not the same requirement.
The last option is the one to be suspicious of. Async because a dependency is unreliable is often the right call, and it is sometimes a way of making an unreliable dependency invisible rather than reliable — the failures then surface as silent staleness instead of errors.
Can the caller complete its own job without this result?
when The result changes the response: authorization, stock, pricing, fraud decision.
cost Coupled availability and latency. Bound it hard: timeout, retries, circuit policy, bulkhead.
when Emails, indexing, analytics, downstream notification, anything the client does not see.
cost Eventual consistency, idempotent consumers, a broker and a dead-letter queue to operate.
when Report generation, bulk import, video processing.
cost A job resource and a status contract; the client must handle a pending state (Request or Background?).
when Fast path preferred, but degraded operation is acceptable when the dependency is down.
cost Two code paths and two sets of failure semantics; test the fallback or it does not exist.
when The retry and backlog genuinely give it time to recover.
cost Failures become invisible staleness. Only choose this with queue-age alerting, or you have hidden the problem rather than absorbed it.
How to build it
Most important first.
- Split the operation into what the client is waiting for and what merely has to happen. Reserve stock synchronously because the answer changes the response; send the email asynchronously because nothing about the response depends on it.
- Make the synchronous part as small as possible and give it the strictest budget: a timeout well inside the client's, bounded retries, and a defined behaviour when it fails (Timeouts, Retries).
- Publish the asynchronous part from the same transaction that commits the state change, via an outbox — otherwise you have two writes with no atomicity and you will drop events (The Transactional Outbox).
- Design every consumer to be idempotent and to tolerate out-of-order and duplicate delivery. Assume redelivery is normal, because it is (Duplicate Detection).
- Return something honest to the client for the deferred part:
202 Acceptedwith a status resource, or a state field the client can poll or subscribe to. "Async" is a contract decision, not only an implementation one (The Async Job Pattern in API Design). - Where a synchronous call is required but its dependency is unreliable, add a bulkhead and a circuit policy so its degradation cannot consume your capacity (Bulkheads, Circuit Breakers).
- Where the client needs an answer and the work is genuinely long, the pattern is a synchronous acknowledgement plus asynchronous completion — not a long-held request (Long-Running Operations: 202 and the Job Resource in API Design).
What can go wrong
- A queue used to hide a slow consumer: the backlog grows silently, and users see stale state hours later with no error anywhere (Queue Backlog).
- Events published before the transaction commits, so a consumer reads state that does not exist yet — or that never will, because the transaction rolled back.
- Async chosen for something the user is visibly waiting for, producing a UI that shows success and a system that has not done the work.
- A synchronous chain several hops deep, where the deepest service's latency sets a floor for the whole path and nobody has ever measured it end to end.
- Retries without idempotent consumers, turning every redelivery into a duplicate charge (Job Idempotency).
- Ordering assumed because messages usually arrive in order. Usually is not a guarantee, and the code that assumed it fails rarely and expensively (Webhook Retries and Ordering).
- Dead-letter queue configured and never monitored, so failed work accumulates unnoticed for months.
- An event published before commit can be consumed before the data exists — the classic read-your-own-write failure across services (The Transactional Outbox).
- Two events for the same entity processed concurrently by two consumer instances apply out of order. Order by a version or partition by entity key (Writing Event Consumers, Kafka-Style Logs: Topics, Partitions, Offsets in Software Architecture).
- A redelivery arriving while the first delivery is still in flight means the same message is processed twice simultaneously, which a naive "have I seen this id" check does not catch (Duplicate Detection).
- Messages are untrusted input on the consumer side. A message from your own producer is still deserialized by your consumer, and a compromised or buggy producer reaches straight past your HTTP validation (The Trust Boundary, Deserialization: Bytes to Objects).
- Authorization does not travel with a message by default. The consumer must know on whose behalf it is acting, which means the authorization context is part of the message contract or is re-derived — never assumed (Authorization in Backends).
- A queue is a durable store of business data. It has retention, access control and encryption requirements like any database, and it is routinely exempted from all three.
- Messages are frequently logged wholesale during debugging, which is one of the standard routes for credentials and personal data into log storage (Secrets in Logs).
- "Async is more scalable." It absorbs bursts and decouples availability. Sustained throughput is set by the consumer and its database, and a queue in front of an overloaded consumer just hides the overload for longer.
- "Async is faster." The response is faster; the work is not. Total completion time is usually *longer*, because it now includes queueing.
- "The queue guarantees delivery, so we cannot lose the work." It guarantees delivery of what it accepted. Publishing outside the transaction loses events regardless of what the broker promises (The Transactional Outbox).
- "Exactly-once delivery." Delivery is at-least-once in practice; exactly-once *processing* is your consumer's idempotency, and conflating the two is how duplicate charges happen.
- "Events decouple services." They decouple runtime availability and introduce schema coupling — every consumer depends on the event's shape, and there is no compiler to tell you who (Naming Events).
- "We should make everything event-driven." Some things need an answer. An operation whose result determines the response is synchronous, and dressing it as async just moves the wait somewhere less visible.
Operating it
- For synchronous calls: per-dependency latency percentiles, error rate, timeout rate and concurrency in flight. The gap between your latency and the sum of your dependencies' is your own work (The Critical Path Is the Only Path That Pays in Observability & Performance).
- For asynchronous work: queue age — the age of the oldest unprocessed message — rather than queue depth. Depth without age cannot distinguish a healthy burst from a stalled consumer (Depth Is Not an Emergency; Age Is in Observability & Performance).
- Consumer lag, retry counts and dead-letter arrival rate, each alerting separately. A rising dead-letter rate is a code bug; a rising lag is a capacity problem.
- End-to-end completion time for the whole operation, not just the request. The customer's experience of "order placed" includes the email, and no request metric contains that.
- At 10x, the asynchronous path degrades gracefully: the backlog grows, latency to completion rises, and nothing fails outright — provided consumers can eventually catch up and the queue is bounded (Backpressure).
- At 10x, the synchronous path degrades sharply: pools saturate, timeouts fire, retries amplify, and the failure spreads to callers that never touched the slow dependency (Cascading Failure).
- That difference is real and is where "async scales better" comes from. It is a statement about burst absorption and coupling, not throughput: a consumer that can process 500 messages per second processes 500 per second regardless of how deep the queue is, and if arrivals exceed that indefinitely the backlog is unbounded and async has made things worse, not better (Little's Law as Working Intuition in Observability & Performance).
- At 100x, both need explicit limits: concurrency caps and load shedding on the synchronous side, bounded queues and backpressure to producers on the asynchronous side.
- Async buys decoupling and pays in consistency: the system is correct eventually, and "eventually" is a window users and support staff will notice (Eventual Consistency in Practice).
- Async buys burst absorption and pays in debuggability: a request no longer has one trace, and cause and effect are separated in time.
- Sync buys immediacy and simple reasoning and pays in coupled availability and coupled latency.
- Async adds infrastructure — a broker to run, monitor, secure and upgrade — plus dead-letter handling and replay tooling nobody budgets for.
- Mixed is usual and correct, and it costs you two mental models in one request path. Be explicit about which half is which in the contract, or clients will assume the whole thing is synchronous.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe coupling analysis holds for any transport pair — HTTP versus a broker, gRPC versus a log, an in-process call versus an in-process event.
- PROTOCOL-SPECIFICGuarantees differ sharply by broker: a partitioned log gives ordering within a partition and consumer-controlled offsets; a classic work queue gives competing consumers and no ordering; a cloud queue may guarantee neither ordering nor deduplication except in a specific mode with a throughput cost. "The queue handles it" is only true of a specific queue in a specific configuration.
- SCALE-SPECIFICBelow a few requests per second, the synchronous version of nearly everything is correct and much simpler — a broker introduced at that scale is infrastructure without a problem. The burst-absorption argument only starts to pay when arrival rate is genuinely spiky relative to consumer capacity.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.