Structured Concurrency & Cancellation

Structured Concurrency

Child tasks belong to a scope, and the scope does not exit until every child has completed or been cancelled. The alternative — tasks that outlive the thing that started them — is how a request handler returns while three of its tasks are still writing to a response that is already closed.

▶ Run the lab

The question this answers

The question

Who owns this task, and what is guaranteed to have happened to it by the time the function that started it returns?

The work

A product-page handler that fans out to three services — inventory, pricing, reviews — assembles the results and returns. Reviews is slow and occasionally times out.

What is shared

The response object the handler is building, the request-scoped context (trace id, user, deadline), and any connection or transaction the handler holds. All three have a lifetime tied to the handler, and every child task holding one is holding something that may be recycled underneath it.

The invariant — what must stay true under every interleaving

When the handler returns, no task it started is still running. Every child has either completed, failed, or been cancelled and observed.

Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.

WorkCan it overlap?Can it parallelise?What is shared?What ordering?What synchronization?Where is contention?What can deadlock?What can race?What is gained?What complexity?

The rule, and what it buys

Structured concurrency is one rule: a task started inside a scope cannot outlive that scope. Concretely, the block that opens the scope does not exit — normally or by exception — until every task started in it has finished. If the block exits early because one child failed, the remaining children are cancelled first and awaited.

This is the same discipline that made structured programming work. goto let control leave a block arbitrarily; braces and functions constrained it, and reasoning became possible because you knew that when a block ended, everything inside it had ended too. Unstructured task spawning is goto for concurrency: a bare "start this in the background" call means control has left in a way the calling code cannot see or account for.

What the rule buys is a set of guarantees you otherwise have to establish by hand for every call site. Errors propagate: a child that throws surfaces at the scope, not into a void (Orphaned Tasks is what happens without this). Cancellation propagates: cancelling the scope cancels every child (Cancellation Propagation). Lifetimes are legible: a child cannot be holding your request context after you returned it to a pool. And leaks become structurally impossible rather than something you audit for.

Left: a scope with a boundary. Right: three tasks with no owner.
openjoinedjoinedcancelled on deadline, then joinedfire and forgetafter the handler returnedhandler(request)handler(request) — unstructuredscope: does not exit until all children settlespawn(reviews()) — no ownerreturn response — reviews() still runninginventory()pricing()reviews() — slowwrites to a recycled request contextreturn response — all children settled
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

What the shape looks like in code

The unstructured version is not obviously wrong, which is the problem. It spawns three tasks, awaits two of them, and returns. The third is still running — perhaps for four more seconds — holding a reference to a request context whose connection has already been returned to the pool. Its exception, when it comes, has nowhere to go and is swallowed by the runtime with at best a warning on a stream nobody reads.

The structured version differs in one visible way: there is a block, and the block is the join point. Every task started inside it is awaited at its close. If reviews throws, the scope cancels inventory and pricing and re-raises. If the caller cancels, the cancellation reaches all three. If the deadline fires, the same. None of that is code the handler wrote — it is a property of the scope.

Note the second-order benefit: the structured version cannot accidentally leak a task, so you do not need a lint rule, a code-review convention or a "did you await this?" checklist. The invariant is enforced by the construct rather than by attention, which is the only kind of enforcement that survives a growing team (Concurrency Anti-Patterns).

Three tasks, one owner, two joins — the third outlives the handler
1async def product_page(req):
2 inv = asyncio.create_task(inventory(req.sku))
3 prc = asyncio.create_task(pricing(req.sku))
4 rev = asyncio.create_task(reviews(req.sku)) # nobody awaits this
5
6 body = {"inventory": await inv, "pricing": await prc}
7 return render(body)
8 # rev is still running.
9 # - it holds req, whose connection is about to be recycled
10 # - if it raises, the exception is swallowed with a warning at GC time
11 # - if the client disconnected, nothing tells it to stop
12 # - under load these accumulate: unbounded concurrency, invisible
A scope: the block is the join point, and it is not optional
1async def product_page(req):
2 async with asyncio.TaskGroup() as tg: # the scope
3 inv = tg.create_task(inventory(req.sku))
4 prc = tg.create_task(pricing(req.sku))
5 rev = tg.create_task(reviews(req.sku))
6 # Control reaches here only when all three have settled.
7 # - any child exception propagates out of the block
8 # - a child raising cancels its siblings first
9 # - req is guaranteed unreferenced by the time we return
10 return render({
11 "inventory": inv.result(),
12 "pricing": prc.result(),
13 "reviews": rev.result(),
14 })

The unstructured version has no place where "all my tasks are done" is true, so no invariant can be stated about the handler's own lifetime. The structured version makes the closing brace that place. Everything else — error propagation, sibling cancellation, no leaked references — follows from having a boundary at all.

Failure policy, and what a scope does not solve

A scope has to decide what happens when one child fails while others are still running, and the choice is not universal. Fail-fast cancels the siblings and propagates immediately — right when the results are all needed, as here, because a page without pricing is not renderable and the other two calls are now wasted work. Wait-for-all collects every outcome and reports them together — right when partial results are usable, or when each child has a side effect that must complete. Libraries differ in their default, and using the wrong one produces either wasted work or lost results (Promise.all & gather covers the same choice at the API level).

The timeline shows fail-fast: reviews exceeds the deadline at tick 6, the scope cancels the still-running inventory retry, both are joined, and the handler returns at tick 7 rather than at tick 11. The wasted work saved is real, but note the second half of the picture — cancellation is cooperative, so inventory is only cancelled *at its next suspension point*, which is why its lane shows a short tail after the cancel marker (Cancellation Propagation).

And the honest limits. A scope does not make a child stop instantly — see the tail. It does not help with work handed to something outside the scope: an item pushed onto a global queue, a thread-pool submission, a callback registered with a driver. It does not survive process death; scopes are a structuring tool, not a durability one. And a scope that wraps a task which never yields cannot cancel it at all — Cancellation is where that limit is stated properly.

Fail-fast scope with a 600 ms deadline. Modelled.SIMULATED
Scope (handler)
open, start 3 children
awaiting children
cancel siblings, join
return 200 with partial + error
inventory()
call
retry after 503
cancellation delivered at next await
pricing()
call
done
reviews()
call
deadline exceeded → raises
Same work, unstructured
handler returns at tick 4
reviews() still running, unowned
↑ deadline — scope cancels siblings↑ all children settled; scope exits
runningreadywaitingblockedidle1 tick ≈ 100 ms

Key points

  • One rule: a task started in a scope cannot outlive it. The closing brace is a join point, and it is not optional.
  • Error propagation, sibling cancellation, and the absence of leaked references all follow from having a boundary — none of them is separate code.
  • Fail-fast versus wait-for-all is a real choice; library defaults differ, and picking wrong costs either wasted work or discarded results.
  • A scope does not stop a child instantly. Cancellation is cooperative, so children stop at their next suspension point.
  • Work handed outside the scope — a global queue, a pool submission, a driver callback — is outside the guarantee, and that is where leaks re-enter.
  • The value is that the invariant is enforced by the construct rather than by review discipline, which is what makes it hold as a team grows.

The loop, answered

Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.

How it works
  • Opening a scope creates an owner object that tracks every task started through it.
  • Starting a child registers it with the scope; there is no API for starting an unregistered child inside the block.
  • The block body runs; children run concurrently with it and with each other.
  • At the closing of the block the scope waits for every child to settle before allowing control past it.
  • If a child fails, the failure policy runs: fail-fast cancels the remaining children, waits for them to observe it, and then propagates the original error.
  • If the scope itself is cancelled from outside, that cancellation is delivered to every child, and the scope still waits for all of them before exiting.
Interleavings that matter
  • Unstructured: handler awaits inv and prc, returns at tick 4, and rev continues to tick 11 holding req. At tick 6 the request context is recycled for a different request; rev writes trace attributes onto someone else's span.
  • Unstructured with a failure: rev raises at tick 11. No frame is waiting on it. The runtime logs "task exception was never retrieved" at collection time, minutes later, with no request context attached.
  • Structured fail-fast: rev raises at tick 6; the scope cancels inv and prc; inv observes cancellation at its next await; the scope joins both and re-raises at tick 7. Total elapsed 700 ms instead of 1 100 ms.
  • Structured wait-for-all: rev raises at tick 6; inv and prc run to completion; the scope exits at tick 11 reporting one failure and two successes. Slower, and correct if the siblings had side effects that must finish.
  • The pathological case: a child running a tight CPU loop with no suspension point. The scope cancels it; the cancellation is recorded and never observed; the scope waits forever. Structured concurrency did not fail — cooperative cancellation did (Cancellation).
What it guarantees — and does not
  • A scope guarantees that when control passes its closing point, every task started within it has settled — completed, failed, or observed its cancellation.
  • It guarantees a child's exception reaches a frame that can handle it, rather than being discarded.
  • It guarantees no child holds a reference to scope-local state after the scope exits — which is what makes pooled connections and request contexts safe to recycle.
  • It does NOT guarantee children stop promptly. Cancellation delivery is cooperative and bounded only by how often children reach a suspension point.
  • It does NOT guarantee anything about work the child handed to something outside the scope before it stopped.
  • It does NOT make concurrent code correct. Two children mutating shared state still race; the scope governs lifetime, not access (Shared Mutable State).
Where contention appears
  • The scope's child registry is touched on every spawn and every completion; for a scope with thousands of children that becomes a real shared structure rather than bookkeeping.
  • Fail-fast cancellation wakes every sibling at once, producing a burst of scheduler work at exactly the moment the system is already in a bad state.
  • A scope offers no concurrency limit by itself: spawning a child per item over a 50 000-item list creates 50 000 concurrent tasks, all structured and all overwhelming the downstream (Unbounded Concurrency, Bounding Concurrency).
  • Nested scopes serialise cancellation delivery down the tree, so a deep hierarchy takes measurably longer to unwind than a flat one.
How it fails
  • Scope hang: one child never reaches a suspension point, so the scope waits forever for a cancellation it can never observe.
  • Lost sibling results under fail-fast when partial results were in fact usable.
  • Wasted work under wait-for-all when the first failure already made the rest pointless.
  • Escaped work: a child enqueues onto a global queue or submits to a shared pool before stopping, and that work outlives the scope anyway.
  • Unbounded fan-out: structurally correct, operationally a denial-of-service against a downstream (Parallelism Moves the Load Downstream).
  • Cancellation swallowed by a child that catches a broad exception type and continues, converting a cancel into an unlogged error (Cancellation).
When it helps
  • On any request path that fans out, because the request context, connection and deadline all have the handler's lifetime and children must not outlive it.
  • Whenever a failure in one branch makes the others pointless — fail-fast turns "three slow calls and an error" into "one error, fast".
  • In long-lived services, because leaked tasks accumulate: one per request at a few thousand requests a minute is a memory and load problem within the hour.
  • When a team is larger than the set of people who remember the conventions — the construct enforces what the convention could not.
When it hurts
  • When the work genuinely should outlive the request: an audit write or a cache warm that must complete regardless. Those need a *longer-lived* scope owned by the application, not no scope (Orphaned Tasks).
  • When a child cannot be made cancellable — a blocking library call, a C extension, a synchronous socket read — because the scope then guarantees a hang rather than a leak.
  • When the overhead matters: scope bookkeeping per task is small but not zero, and for millions of tiny tasks it is measurable (Parallel Overhead).
  • When the language has no support and you emulate it by hand; a hand-rolled scope that gets cancellation propagation wrong is worse than an honest fire-and-forget with a tracked handle.
How you would know
  • Live task count by scope or by parent operation. A count that grows with uptime rather than with concurrency is a leak, and it is the primary signal.
  • "Task exception was never retrieved" warnings, or the runtime's equivalent — treat them as errors, because each one is a swallowed failure.
  • Scope exit duration: the gap between the last useful result and the scope actually closing is the cost of cancellation delivery.
  • Handler duration against the sum of its children's durations; if the handler is consistently shorter than its slowest child, you have unstructured tasks.
  • Request-context reuse errors — trace attributes on the wrong span, connections used after return — which are the visible symptom of children outliving their scope (Carrying the Trace Across the Gap in Performance).
Complexity it introduces
  • Every fan-out site gains a block, and the failure policy at each becomes an explicit decision rather than a default.
  • Children must be written to be cancellable, which pushes the requirement down into libraries you may not control.
  • Work that legitimately outlives the request needs an explicit longer-lived owner, so the application grows a task-supervision concept it did not have.
  • Debugging changes: a hang is now "which child is not yielding" rather than "which task leaked", which is a better question but still needs task dumps to answer (Task Dumps: When the Threads Look Idle and Nothing Is Moving).
Simpler alternatives
  • Await everything you start, manually. Structured concurrency by discipline — correct, and it fails the first time someone adds a fourth call in a hurry.
  • A supervised background pool with an explicit lifetime for work that should outlive the request, with its own bound and its own shutdown (Draining a Pipeline).
  • A durable queue for anything that must survive the process, which is the correct answer whenever "must complete" really means "must complete even if we crash".
  • Do it sequentially. If the calls are fast and the fan-out saves 20 ms, sequential code has no lifetime problem at all (The Sequential Await Trap is about the opposite mistake — know which one you have).

Three children, and the moment the parent returns

Three children, and the moment the parent returns
The parent starts three tasks. The only question is whether the parent is allowed to return while they are still running — and whether anybody is left to hear it when one of them fails.
1/12 · t0
spawn(fetch_user)                  # nobody holds the handle
spawn(fetch_orders)
spawn(build_report)
return "ok"                        # the children outlive this frame
Parent
spawn 3 tasks
return
Child A · user
fetch user
Child B · orders
fetch orders
raises TimeoutError
Child C · report
build report — nobody is waiting for it
↑ parent returns
runningreadywaitingblockedidlemodel ticks
parent returns at
t2
children alive after that
3
orphans running now
0
exceptions reaching the caller
0 of 1
t0spawn(A); spawn(B); spawn(C) # fire and forget
t1parent returns "ok" to its caller — before any child has finished
t3A completes. Its result is written to a future nobody holds.
t4B raises TimeoutError. There is no awaiter, so the exception is swallowed — at best a line in a log nobody reads.
t12+C is still running, still holding a DB connection, long after the request it belonged to was answered. Nothing will ever join it or cancel it.
The parent returned at t2 and told its caller everything was fine. At t0 the orphan has finally stopped, or has not — you cannot tell from here: Child C is an orphan, holding a connection that belongs to a request that has already been answered, and it will keep running until it finishes, the process exits, or it leaks forever. Child B's TimeoutError went nowhere at all — an exception raised in a task nobody awaits has no propagation path, so it is swallowed or logged into a void, and the caller was told "ok". This is why the failure shows up as a metric that does not add up rather than as a stack trace. The structural claim is worth stating plainly: concurrency without a scope is a goto for lifetimes. It breaks the property every other control-flow construct gives you — that when a block ends, what it started has ended too — and with it goes error propagation, cancellation, timeouts and the ability to reason about resources at all. The price is that a scope must wait, so a genuinely background task needs an explicitly longer-lived scope that somebody owns, not a detached spawn nobody does.
ILLUSTRATIVETicks are model time, not measurements. Exactly how cancellation is delivered is runtime-specific: cooperative cancellation points in Python and Kotlin, an AbortSignal in JavaScript, a context in Go, a stop token in C++.

What people believe, and what is true

Claim

Structured concurrency makes concurrent code safe.

Reality

It governs lifetime and error propagation. Two children mutating the same dictionary still race; you still need the whole of the shared-state module.

Claim

The scope kills children when it exits.

Reality

It cancels them and then waits. A child that never checks for cancellation is not killed — the scope blocks on it, turning a leak into a hang.

Claim

Awaiting all my tasks is the same thing.

Reality

It is, on the happy path. It differs on the error path, which is where it matters: an exception between spawn and await skips the await entirely and leaves the task running.

Apply it