Coordination & Limits

The Sequential Await Trap

Three independent awaits in a row take the sum of their latencies for no reason at all. The fix is four characters of syntax and it is genuinely correct — but read the warning before you ship it, because the very next lesson is about the database this fix knocks over.

▶ Run the lab

The question this answers

The question

These three calls do not depend on each other — why does the endpoint take the sum of their latencies?

The work

A dashboard endpoint that awaits the user profile (80 ms), then their org (60 ms), then their notification count (40 ms), then their recent activity (120 ms) — four independent reads, one after another.

What is shared

Nothing between the four calls; they read disjoint data and touch no common state. That is precisely why serialising them buys nothing — and precisely why unserialising them is safe *for correctness* while being unsafe for load.

The invariant — what must stay true under every interleaving

Endpoint latency is bounded by the longest call it must make, not by the sum of the calls it happens to make; and any call that genuinely depends on an earlier result still runs after it.

Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.

WorkCan it overlap?Can it parallelise?What is shared?What ordering?What synchronization?Where is contention?What can deadlock?What can race?What is gained?What complexity?

Sum versus max, and where the 300 milliseconds went

Written in the obvious order, four independent awaits produce four sequential round trips: 80 + 60 + 40 + 120 = 300 ms. Started together, the endpoint takes as long as the slowest, which is 120 ms. Nothing about the calls changed; only the moment each one started.

The reason this is so common is that the sequential form is what the syntax encourages. await reads like a statement, statements read as ordered, and the order is invisible in a code review because it looks exactly like correct code — which it is. It is just slow, and the slowness is proportional to how many independent things the endpoint happens to need.

The tell in production is a latency graph where the endpoint's p50 tracks the sum of its dependency spans and each individual span is fast. In a trace waterfall it is unmistakable: a staircase, each span starting where the previous ended, with no overlap anywhere (Reading the Waterfall).

Four independent reads: awaited in sequence versus started together. Spans are shape, not measurement.ILLUSTRATIVE
Sequential — sum of latencies (300 ms)
profile 80ms
org 60ms
notifications 40ms
activity 120ms
Concurrent — max of latencies (120 ms)
all four in flight
Downstream database — what it sees
sequential: 1 concurrent query at a time
concurrent: 4 at once, for 120 ms
↑ concurrent version done↑ sequential version done
runningreadywaitingblockedidle1 tick ≈ 20 ms

The fix, and the dependency that must stay sequential

The mechanical change is to start every independent call before awaiting any of them. In JavaScript that means constructing the promises first, because construction is what starts the work; in Python it means wrapping each coroutine in a Task or handing them all to gather, because a bare coroutine is inert (Futures & Promises).

The important discipline is separating what is genuinely independent from what is not. A call that needs user.orgId cannot start before the user arrives, and no amount of restructuring changes that — the dependency is real and the latency is the critical path (Dependency Graphs). Parallelising a dependent call by guessing the input is not a performance fix; it is a correctness bug with a fast response time.

And now the warning, because it belongs here and not three paragraphs later. This refactor multiplies concurrent downstream load by the number of calls you unserialised. The endpoint that held one connection-pool slot for 300 ms now holds four for 120 ms. Your local benchmark improves; the database sees four times the concurrent query count at the same request rate. If the endpoint is hot, this is the change that takes the database down, and it will look like the database's fault. Read Parallelism Moves the Load Downstream before deploying it, and bound the concurrency if the fan-out is over a list rather than a fixed handful.

Four independent reads, serialised by habit: 300 ms
1async function dashboard(userId: string) {
2 const user = await getUser(userId) // 80ms
3 const org = await getOrg(orgIdFor(userId)) // 60ms — does not need `user`
4 const notifications = await getNotifications(userId) // 40ms — independent
5 const activity = await getActivity(userId) // 120ms — independent
6 return render({ user, org, notifications, activity })
7}
8// Total: 300ms. Trace shows a staircase. Each span is individually fast.
Independent calls started together, dependent call kept in order: 120 ms
1async function dashboard(userId: string, signal: AbortSignal) {
2 // Construction starts the work in JS. All four are in flight after this block.
3 const userP = getUser(userId, signal)
4 const orgP = getOrg(orgIdFor(userId), signal)
5 const notificationsP = getNotifications(userId, signal)
6 const activityP = getActivity(userId, signal)
7
8 const [user, org, notifications, activity] =
9 await Promise.all([userP, orgP, notificationsP, activityP])
10
11 // GENUINELY dependent: needs user.teamId. This one stays sequential,
12 // and it is now the critical path. Parallelising it would be a bug.
13 const team = await getTeam(user.teamId, signal)
14
15 return render({ user, org, notifications, activity, team })
16}
17
18// WARNING: this endpoint now holds 4 pool slots instead of 1.
19// At 200 req/s that is 800 concurrent queries where there were 200.
20// If the fan-out is over a LIST, bound it — do not map an array into all():
21async function enrich(ids: string[], signal: AbortSignal) {
22 return mapWithConcurrency(ids, 8, (id) => getDetail(id, signal))
23 // ^ a ceiling, chosen against the pool size
24}

Starting the independent calls together turns a sum into a maximum, and the dependent call stays where it belongs because its input does not exist any earlier. What the refactor also does — and what no compiler will tell you — is multiply concurrent downstream load by four. Fixed handfuls are usually fine; a fan-out over a list needs an explicit concurrency bound before it ships.

When the sequence was load-bearing

Not every staircase is a mistake. Sometimes the sequential order was carrying an invariant that nobody wrote down, and removing it produces a fast, wrong answer. The three cases to check before unserialising: a real data dependency (call B needs A's output), an ordering requirement on writes, and a shared resource that only tolerates one user at a time.

The schedule below is the second case, and it is the one that gets shipped. Two calls that "do not depend on each other" in the sense that neither uses the other's return value, but which write to and read from the same row. Sequential, the read observed the write. Concurrent, it observes whatever the database happened to have.

Unserialising a write and a read that shared a row. Neither used the other's return value.ILLUSTRATIVE
Invariant · The audit summary reflects every event recorded for this request
#recordEvent (write)buildSummary (read)Database rowState
1sequential version: INSERT audit event; commits··events=4 summaryCount=-
2·sequential version: SELECT count(*) → 4·events=4 summaryCount=4
3CONCURRENT version: INSERT issued at t=0, still in flight··events=3 summaryCount=-
4·CONCURRENT version: SELECT count(*) issued at t=0, returns 3·events=3 summaryCount=3
5··INSERT commits at t=15 msevents=4 summaryCount=3
✕ The summary says 3 events; 4 were recorded. Neither call used the other's return value, so the "independence" check passed — the dependency was through the database, not through the code.
6·response rendered with a stale count; no error anywhere·events=4 summaryCount=3
Independence means "shares no state and imposes no ordering", not "does not use the other's return value". Before unserialising, check for shared rows, shared caches, shared external resources and write-then-read pairs. When the ordering is real, keep it — and note that the sequential version was correct precisely because it was slow.

Key points

  • Independent awaits in sequence cost the sum of their latencies; started together they cost the maximum.
  • In JavaScript, constructing the promise starts the work — so the fix is to construct all of them before awaiting any.
  • In Python a bare coroutine is inert, so the fix is gather or create_task, not merely reordering awaits.
  • A trace waterfall showing a staircase of fast spans with no overlap is the signature.
  • The fix multiplies concurrent downstream load by the number of calls unserialised — one pool slot for 300 ms becomes four for 120 ms.
  • Independence means "shares no state and imposes no ordering", not "does not use the other's return value".
  • Fan-out over a list needs an explicit concurrency bound; never map a large array straight into Promise.all.

The loop, answered

Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.

How it works
  • Each await suspends the current task until that operation settles, so the next call is not even issued until the previous one has returned.
  • Total latency is therefore the sum of the round trips plus the local work between them.
  • Starting all the operations first puts them in flight simultaneously; the awaits then only collect results that are already on their way.
  • The executor is free during all of this either way — the sequential version is not using more CPU, it is using more wall-clock.
  • The aggregate settles when the last member settles, so the endpoint's latency becomes the maximum of the members plus the local work.
  • Downstream, the change converts one concurrent unit of work per request into N, held for a shorter time. Total work is identical; peak concurrency is N times higher.
Interleavings that matter
  • Sequential: profile (0–80 ms), org (80–140 ms), notifications (140–180 ms), activity (180–300 ms). One pool slot held throughout, response at 300 ms.
  • Concurrent: all four issued at t≈0, all four in flight, the slowest returns at 120 ms, response at 120 ms. Four pool slots held for 120 ms.
  • Concurrent with a genuine dependency mistakenly unserialised: getTeam(user.teamId) is started before user exists, so teamId is undefined and the call either 404s or — worse — returns a default team.
  • Concurrent write-then-read on the same row: INSERT and SELECT issued together, SELECT returns before the INSERT commits, and the summary is short by one with no error.
  • At 200 req/s: sequential holds ~60 concurrent connections on average, concurrent holds ~96 at four times the instantaneous rate — and the peak, not the average, is what exhausts a pool of 100.
  • Fan-out over a list of 500 with no bound: 500 simultaneous queries, 480 of them queued on a pool of 20, each one's timeout running from creation (Parallelism Moves the Load Downstream).
What it guarantees — and does not
  • Guaranteed: unserialising genuinely independent calls does not change the result. The correctness argument is sound when the independence is.
  • Guaranteed: latency drops from the sum to the maximum plus scheduling overhead.
  • Guaranteed: total downstream *work* is unchanged — the same queries, the same rows.
  • NOT guaranteed: that downstream can absorb the new concurrency. Peak concurrent load is multiplied by the fan-out width.
  • NOT guaranteed: that "does not use the other's return value" means independent. Shared rows, caches and external resources create dependencies the code does not show.
  • NOT guaranteed: any bound on concurrency. Promise.all over an array starts everything at once, always.
  • NOT guaranteed: improvement at all if the calls contend for the same bottleneck — four queries against a saturated pool finish no sooner than four in sequence.
Where contention appears
  • Peak concurrent downstream usage multiplies by the fan-out width, so the connection pool, the rate limiter and the thread pool all see N times the instantaneous demand.
  • If the four calls hit the same backend, they queue there instead of overlapping and the latency win shrinks toward zero while the load multiplication remains — the worst of both.
  • On an event loop, four continuations become ready close together, so the aggregate's resolution is a small burst rather than a spread.
  • Tail latency now belongs to the slowest member; a dependency with a bad p99 exports it to your endpoint the moment you parallelise (Fan-Out: Waiting for the Slowest of Seven).
How it fails
  • Unserialising a genuine data dependency: the second call runs with undefined input and returns wrong data or a plausible default.
  • Unserialising a write-then-read: the read observes state from before the write, with no error.
  • Connection-pool exhaustion after the deploy, because peak concurrency rose by the fan-out factor (Parallelism Moves the Load Downstream).
  • Rate-limit rejections from a third party that was fine with one call per request and is not fine with four at once.
  • Timeout storms: all N calls start their clocks together, so a slow backend fails all of them simultaneously instead of one at a time.
  • Unhandled rejection from an abandoned member after Promise.all rejects (Promise.all & gather).
When it helps
  • A fixed, small set of genuinely independent reads on a hot endpoint — the classic dashboard or product page.
  • Calls to *different* backends, where the concurrency is spread across systems rather than concentrated on one.
  • High-latency, low-cost dependencies: a 200 ms third-party API that costs nothing to run concurrently.
  • Anywhere the trace shows a staircase and the total is dominated by round trips rather than by work.
When it hurts
  • When the calls share a bottleneck — one database, one pool, one rate limit — so they queue instead of overlapping.
  • When the fan-out is over a list, because the width is now data-dependent and unbounded.
  • When the ordering was carrying an invariant that nobody documented.
  • When the endpoint is hot: the multiplication applies at your peak request rate, which is exactly when the dependency has the least headroom.
  • When the "independent" calls are cheap and local; you have added scheduling to save microseconds.
How you would know
  • Endpoint latency versus the sum of its dependency spans. Equal means fully serialised; equal to the max means fully overlapped.
  • The trace waterfall itself — a staircase is the diagnosis and it takes one look (Reading the Waterfall).
  • Before deploying the fix: peak concurrent downstream calls, connection-pool utilisation and pool wait time. Take the baseline first, because you will need it.
  • After deploying: the same three numbers, plus rate-limit rejection counts on any third party in the fan-out.
  • Downstream request rate divided by inbound request rate. It should rise by exactly the fan-out width; if it rises more, retries are compounding.
  • p99 of the endpoint, not p50. Parallelising moves you onto the slowest member's tail, and that shows up at p99 first.
Complexity it introduces
  • The code no longer reads top-to-bottom as a sequence of steps; the reader must work out which promises are in flight at each line.
  • Error handling changes shape: one failure now abandons in-flight siblings unless cancellation is plumbed through (Promise.all & gather).
  • A concurrency bound becomes a parameter someone must choose and justify against the pool size — and it is not derivable from first principles (Bounding Concurrency).
  • The dependency analysis is now load-bearing and lives only in the author's head; a later edit that adds a shared-row dependency between two parallelised calls reintroduces the race silently.
Simpler alternatives
  • One batch call instead of N concurrent ones: a single query with an IN clause, or a batch endpoint. Better latency *and* less load — strictly superior when it exists (batch-apis).
  • Bounded concurrency (mapWithConcurrency(items, K, fn)) whenever the fan-out is over a list, so the width is a decision rather than a consequence of the input size.
  • Cache the slow member. If activity is 120 ms and changes rarely, caching it removes it from the critical path without touching the concurrency at all.
  • Denormalise or precompute: if the same four things are always fetched together, one read of a materialised view beats four concurrent reads.
  • Leave it sequential when the endpoint is cold, the total is acceptable, and the dependency is fragile. 300 ms on an admin page is not worth multiplying database concurrency by four.

Sequential awaits vs Promise.all

Three independent awaits
None of the three calls needs another's result. Written one `await` per line they still run one at a time.
users-svc
90 ms
cart-svc
not issued yet
60 ms
promo-svc
not issued yet
120 ms
↑ response at 270 ms
runningreadywaitingblockedidlems
response time
270 ms
your throughput
148/s
peak downstream concurrency
40
downstream call rate
444/s
sequential   90 + 60 + 120 = 270 ms      peak downstream concurrency = 40 × 1 = 40
Promise.all  max(90, 60, 120) = 120 ms      peak downstream concurrency = 40 × 3 = 120

throughput   148/s → 333/s     downstream sees 444/s → 1000/s
270 ms of latency for 270 ms of waiting that could have overlapped. `await` means "suspend this chain until the value arrives", not "start this now". Three independent calls written on three lines run strictly in series, and the 2 later services sit idle while the first one is queried. The fix is `Promise.all` / `asyncio.gather` — but read the callout after you flip it, because the wall time you save is charged to the services you call.
SIMULATED

One request, N downstream calls

One request, N downstream calls
Fanning out turns N × latency into 1 × latency — and one request per second into N requests per second. The second number is the one that takes the downstream service down.
latency
unbounded
sequential would be
800 ms
peak downstream concurrency
200
downstream busy
over 100%
Latency by concurrency limit
1 at a time800.0 ms · 20 rounds · peak 10 downstream
2 at a time400.0 ms · 10 rounds · peak 20 downstream
4 at a time200.1 ms · 5 rounds · peak 40 downstream
8 at a time · peak 80 concurrent against 64 slots — no steady state
16 at a time · peak 160 concurrent against 64 slots — no steady state
20 at a time · peak 200 concurrent against 64 slots — no steady state
What the downstream sees
calls per parent request20 · each parent request multiplies into 20
concurrent calls at peak200 · 64 slots exist
queued at the downstream136 · these are connections, buffers and threads it did not budget for
Wait per call: unbounded
10 parent requests × 20 concurrent calls each = 200 simultaneous calls against 64 slots. The downstream has no steady state here: latency is not high, it is unbounded, and in a real system this appears as connection-pool exhaustion, timeouts and a service that was healthy until an unrelated caller shipped a loop. The best limit at this configuration is 4 at a time (200 ms) — and note that it is usually not 20. Raising the limit removes rounds, which is a linear win; it also raises peak downstream concurrency, which becomes a cliff the moment the peak crosses what the downstream can hold. A limit costs you a little latency in the good case and is the only thing standing between a routine traffic bump and a self-inflicted outage in the bad one. Bound it, and set the bound from the downstream capacity you were actually granted — not from the fan-out you happen to have today, which will be larger next quarter.
SIMULATEDA burst of 10 simultaneous parent requests against a downstream of 64 concurrent slots; waits from the engine's M/M/c approximation. Real fan-out also pays serialisation, connection setup and a tail latency that grows with N — the fastest of N calls does not set your latency, the slowest does.

Bounding concurrency with permits

Bounding concurrency — the permit count protects the dependency, not you
10K tasks behind a semaphore. The downstream service can serve a fixed number at once; the permit slider decides how many you throw at it.
permitsgoodputmean latencytimeoutsfailed of 10K
1 25/s43 ms0.00%0
5 125/s43 ms0.00%0
10 250/s43 ms0.00%0
25 625/s43 ms0.00%0
50 1000/s53 ms0.00%0
100 1000/s103 ms0.00%0
200 1000/s203 ms0.00%0
350 1000/s353 ms0.00%0
500 0/s503 ms100.0%10K
in flight
50
goodput
1000/s
queueing delay added
10 ms
tasks that time out
0
50 permits against a dependency that serves 40 at a time. The extra 10 requests are not being served faster — they are sitting in the dependency's queue adding 10 ms to every latency, and 0 of the 10K tasks time out because of it. Goodput is 1000/s against a peak of 1000/s: you added concurrency and got errors, not throughput. The permit count you want is the one that keeps in-flight work at the dependency's capacity — which you measure, you do not guess.
SIMULATED40 ms service · 400 ms client timeout

What people believe, and what is true

Claim

They do not use each other's return values, so they are independent.

Reality

Independence means no shared state and no required ordering. A write and a read on the same row use no return values from each other and are absolutely not independent.

Claim

Parallelising is free — it is the same work.

Reality

The same total work, arriving N times more concurrently. Peak concurrency is what exhausts pools and trips rate limits, and this refactor multiplies exactly that.

Claim

Moving the awaits around fixed it in Python too.

Reality

A Python coroutine does nothing until it is scheduled. Hoisting the calls above the awaits changes nothing at all; you need asyncio.gather or create_task.

Go deeper

Overview

Awaits in sequence cost the sum; started together they cost the maximum. Four independent 80 ms calls should not take 320 ms.

Practical

Look for a staircase in the trace. Start independent calls before awaiting any of them, keep genuinely dependent calls in order, and take a baseline of downstream pool utilisation before you deploy.

Advanced

The refactor is a load multiplier at your peak request rate. Fixed handfuls are usually safe; fan-out over a list needs an explicit bound, and the bound must be chosen against the downstream pool rather than against the input size.

Internals

Nothing changes in the executor: it was idle during the sequential waits too. What changes is the number of outstanding operations at the downstream resource, which is the quantity Little's law relates to queue length and wait time there.

Apply it