The question this answers
This task has no one waiting for it. Who stops it, who bounds it, and where does its exception go?
A signup handler that returns immediately and starts three background tasks: send a welcome email, warm the user's recommendation cache, and write an analytics event. None is awaited.
Whatever the orphaned tasks captured when they were created: the request context, a database connection, a user object the handler may still mutate. All of it now has two lifetimes — the handler's, and the task's.
Every started task has an owner that will observe its outcome, bound its concurrency, and be able to stop it; and no failure disappears without being recorded.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The three properties you lose at once
Starting a task without keeping its handle loses three things simultaneously, and teams usually only notice the first one. Nobody awaits it, so its result — including its failure — has no destination. Nobody cancels it, so it is unreachable by shutdown, by the request's cancellation, by anything. Nobody bounds it, so at 400 signups a second you have created 1 200 concurrent tasks and nothing anywhere caps that number (Unbounded Concurrency).
The swallowed exception is the one worth dwelling on, because it is genuinely silent. In a promise-based runtime an unhandled rejection may be reported to a global handler or may terminate the process depending on version and configuration; in Python asyncio, a task that raises and is never awaited logs "Task exception was never retrieved" — at garbage-collection time, minutes later, with no request context, no trace id and no user id. Nobody has ever been paged by that log line. The welcome email has been failing for six weeks and the metric that would show it does not exist, because the code path that would increment it is inside the exception nobody catches.
The second-order damage is the captured state. The task holds the request context after the handler returned it to a pool; it holds a database connection past the transaction's scope; it writes trace attributes onto a span that has already ended, or worse, onto a recycled one belonging to a different user. These produce symptoms — wrong user in a trace, a connection "used after close" — that get investigated as driver bugs for a long time before anyone looks at the fire-and-forget line (Carrying the Trace Across the Gap in Performance).
| # | Signup handler | Email task (orphan) | Cache-warm task (orphan) | Runtime | State |
|---|---|---|---|---|---|
| 1 | create email task, cache task, analytics task; keep no handles | · | · | · | live_tasks=3 owned=0 |
| 2 | return 201; request context released to the pool | · | · | · | live_tasks=3 ctx=recycled |
| 3 | · | · | writes a span attribute using the captured context | · | live_tasks=3 ctx=now belongs to request #8812 ✕ Trace data for one user is attached to another user's request. No error is raised; the trace is simply wrong. |
| 4 | · | SMTP provider returns 421 rate-limited; raises | · | · | live_tasks=2 errors_visible=0 ✕ The exception has no awaiting frame. It is stored on the task object and goes nowhere. |
| 5 | · | · | · | garbage-collects the task minutes later | errors_visible=0 |
| 6 | · | · | · | logs "Task exception was never retrieved" with no request context | errors_visible=0 |
| 7 | SIGTERM arrives; shutdown drains the queue and joins the pool | · | · | · | live_tasks=1 joined=pool only ✕ The orphans are not in any registry, so shutdown does not know they exist and cannot wait for or cancel them. Whatever they were doing is lost at exit. |
Giving the work an owner
The fix is not "await it" — the whole point was that the handler should return without waiting. The fix is that the work gets an *owner whose lifetime is the application's* rather than the request's. A long-lived supervised scope, or a bounded background pool, with three properties: it holds a handle to every task, it observes every outcome, and it participates in shutdown.
That gives you back all three lost properties. Failures reach a callback that increments a metric and logs with context. Concurrency is bounded by the pool, so 400 signups a second cannot become 1 200 concurrent tasks. And shutdown knows about the work, so it can drain, cancel or abandon it deliberately (Draining a Pipeline).
It also forces the question the fire-and-forget was avoiding: does this work have to happen? The analytics event does not — losing one on a deploy is fine, and a bounded pool with an abandon policy is right. The welcome email does — a user who never receives it will notice, and that means it belongs in a durable queue, not in process memory, because no in-process ownership survives a crash. The distinction between "should not block the response" and "must eventually happen" is the design decision, and fire-and-forget hides it by treating both the same way (Background Jobs and Workers in Architecture).
1async def signup(req):2 user = await create_user(req.body)3 asyncio.create_task(send_welcome_email(user)) # no handle kept4 asyncio.create_task(warm_recommendations(user)) # no handle kept5 asyncio.create_task(record_signup_event(user)) # no handle kept6 return json(201, user)7 8# All three: unawaited, uncancellable, unbounded, and their exceptions9# surface only as "Task exception was never retrieved" at GC time.10# CPython may also garbage-collect a task nobody references mid-execution,11# so the work can simply stop partway through with no log at all.1class BackgroundWork:2 def __init__(self, limit: int = 50):3 self._tasks: set[asyncio.Task] = set() # hold strong refs4 self._sem = asyncio.Semaphore(limit) # bound concurrency5 6 def spawn(self, coro, name: str) -> None:7 async def runner():8 async with self._sem:9 try:10 await coro11 metrics.bg_ok.inc(name)12 except asyncio.CancelledError:13 raise # never swallow14 except Exception:15 metrics.bg_failed.inc(name) # the metric that was missing16 log.exception("background task failed", task=name)17 t = asyncio.create_task(runner(), name=name)18 self._tasks.add(t)19 t.add_done_callback(self._tasks.discard)20 21 async def drain(self, timeout: float) -> None:22 await asyncio.wait(self._tasks, timeout=timeout)23 for t in self._tasks:24 t.cancel() # policy, chosen explicitly25 26async def signup(req, bg: BackgroundWork):27 user = await create_user(req.body)28 bg.spawn(warm_recommendations(user), "cache-warm")29 bg.spawn(record_signup_event(user), "analytics")30 await outbox.enqueue("welcome-email", user.id) # durable: must happen31 return json(201, user)Same non-blocking behaviour, three properties restored. Note the last line: the email moved to a durable outbox rather than the pool, because "must eventually happen" is not something any in-process owner can guarantee across a crash.
Finding them in code you already have
Orphans are easy to spot once you know the shapes. A task-creating call whose result is discarded. An async function called without await in a language where that produces a floating promise. A .then() chain with no .catch(). A thread or worker started and never joined. A submission to an executor whose returned future is dropped. Most ecosystems have a lint rule for at least the floating-promise case, and turning it on is usually a one-day cleanup with a surprisingly large payoff.
The runtime signals are just as reliable. Live task or thread counts that grow with uptime rather than with concurrency. "Exception was never retrieved" warnings, which should be treated as errors rather than noise. A gap between a handler's measured duration and the duration of everything it started. And background work that silently stops on every deploy — the orphan that shutdown never knew about.
The subtle one is CPython-specific and worth stating plainly: asyncio.create_task returns a task the event loop only holds a *weak* reference to. If you keep no strong reference, the task can be garbage-collected while it is still running, and the work simply stops partway with no error at all. The self._tasks set in the fixed version is not bookkeeping; it is load-bearing.
# 1. Live task count grows with uptime, not with load
asyncio_tasks_live{service="signup"}
09:00 142 rps=380
12:00 1,918 rps=372 <- concurrency flat, tasks 13x
15:00 4,406 rps=390 <- nothing bounds this
# 2. The log line nobody reads. No trace_id, no user_id, minutes late.
15:02:11 ERROR asyncio Task exception was never retrieved
future: <Task finished coro=<send_welcome_email()>
exception=SMTPResponseException(421, b'4.7.0 rate limited')>
# 3. Handler duration vs. what it actually started
http_request_duration_p99{route="/signup"} 41ms
background_work_duration_p99{name="cache-warm"} 2,900ms <- unowned
# 4. Deploy-shaped cliff: work that shutdown never knew existed
welcome_emails_sent ..1,204 1,198 1,211 [DEPLOY] 0 0 0
# not zero because of the deploy - zero for six weeks, and the
# deploy is simply the first place anyone looked at the graph.Key points
- Dropping the handle loses three properties at once: nobody awaits it, nobody cancels it, nobody bounds it.
- The swallowed exception is the expensive one — a failure that surfaces only as a context-free warning at collection time, or not at all.
- Orphans capture request-scoped state and keep using it after the request returned it, producing "impossible" bugs in traces and connection pools.
- The fix is an owner with the application's lifetime: holds handles, observes outcomes, bounds concurrency, participates in shutdown.
- Fire-and-forget hides the real question: "must not block the response" and "must eventually happen" are different requirements, and only the second one needs durability.
- In CPython, a task with no strong reference can be garbage-collected while running — the handle set is load-bearing, not bookkeeping.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Creating a task registers it with the runtime's scheduler and returns a handle carrying its eventual result or exception.
- • Discarding the handle means nothing will ever read that result; the exception is stored on the task object and reachable by nobody.
- • When the task is finally collected, the runtime may emit a diagnostic — a warning, an unhandled-rejection event, or nothing, depending on runtime and configuration.
- • Because the task is in no registry, shutdown cannot enumerate it, cancellation cannot reach it, and no limiter counts it.
- • The fixed version wraps every spawn: acquire from a bounded semaphore, run inside try/except that records the outcome, keep a strong reference, and remove it on completion.
- • Shutdown drains the owner with a budget and then applies an explicit policy — cancel, abandon, or persist — to whatever is left.
- • Handler returns at 41 ms and releases the request context; the cache-warm task writes a span attribute at 2.9 s onto a context now serving a different request. Trace data is attributed to the wrong user, with no error anywhere.
- • Email task raises at 300 ms; no frame awaits it; the exception sits on the task object until collection, then produces a warning with no request context. Six weeks of failures, zero alerts.
- • Deploy: SIGTERM, the queue drains, the worker pool joins, the process exits. Forty orphaned tasks are destroyed mid-flight. Nothing logged, because nothing knew.
- • Load spike: 400 signups per second, three tasks each, average 2.9 s duration — roughly 3 500 concurrent tasks, all hammering the recommendation service, which now fails, which makes the tasks slower, which raises the count further (Unbounded Concurrency).
- • CPython collection: no strong reference is held, a collection cycle runs mid-execution, and the task is destroyed halfway through its work. The write is half-done and nothing is logged.
- • The owned version: the semaphore caps concurrency at 50; the 51st spawn waits; every failure increments a metric with a name; shutdown drains for 5 s and cancels the remainder, logging what it cancelled.
- • An orphaned task guarantees precisely one thing: the caller does not wait for it.
- • It does NOT guarantee the task runs to completion — in CPython it may be collected mid-flight, and in every runtime it is destroyed at process exit.
- • It does NOT guarantee failures are visible. That is the defining property of the bug.
- • It does NOT bound how many exist, so its resource usage is a function of arrival rate with no ceiling.
- • An owning pool guarantees outcomes are observed, concurrency is bounded, and shutdown can act. It does NOT guarantee the work survives a crash — only durable storage does that.
- • Even a supervised pool does NOT guarantee prompt cancellation; that remains cooperative (Cancellation).
- • Unbounded orphans contend for everything at once — connections, sockets, CPU — with no limiter and no visibility, and they compete with the request path that actually has a user waiting.
- • Because they are invisible to any concurrency limiter, they defeat capacity planning: measured request concurrency no longer reflects actual load (Capacity Planning: Traffic to Machines in Performance).
- • Orphans holding pooled connections past the handler's lifetime is a direct cause of pool exhaustion that profiles attribute to the wrong code (Connection Pool Saturation: Waiting in Front of an Idle Database in Performance).
- • A bounded owner converts that contention into a visible queue at the semaphore, which is the whole point: contention you can see is contention you can size.
- • Swallowed exception: a failure that produces no metric, no alert and no attributable log line.
- • Task leak: live task count growing with uptime until memory or a downstream limit fails.
- • Use-after-release of request-scoped state, producing cross-request data corruption in traces and logs.
- • Silent work loss at shutdown, because the task is in no registry.
- • Mid-flight garbage collection in CPython when no strong reference is retained.
- • Unbounded fan-out against a downstream that has no idea where the load is coming from (Parallelism Moves the Load Downstream).
- • Retry loops inside orphans, which can spin indefinitely with nobody to cancel them.
- • It genuinely helps for work that is cheap, idempotent, losable, and whose failure is unimportant — and that set is much smaller than it appears at code-review time.
- • The *owned* version helps whenever the response must not wait for secondary work: cache warms, non-critical enrichment, best-effort notifications.
- • It helps as a deliberate throughput trade when the alternative is holding a request open for work the user does not care about.
- • Even then, "owned and bounded" costs about fifteen lines more than "fire and forget" and removes every failure mode in this lesson.
- • Whenever the work must actually happen. An in-process task is not a delivery guarantee, and a welcome email that fails silently is a product bug (Background Jobs and Workers in Architecture).
- • On any high-rate path, because unbounded task creation is a load generator with no throttle.
- • When the task captures request-scoped resources, which it almost always does because it was written inside the handler.
- • When the team believes background work is happening. An invisible failure is worse than a visible one precisely because it is planned around.
- • Live task or thread count, plotted against request concurrency. Divergence is the leak signal and it is unambiguous.
- • "Exception never retrieved" / unhandled-rejection counts, promoted from log noise to an actual metric with an alert.
- • Per-task-type success and failure counters — impossible without an owner, which is why the owner is what makes the work observable at all.
- • Background work duration p99 compared with the handler's p99; a large gap tells you how long orphans outlive their request.
- • A step change to zero on any background metric at a deploy boundary, which is work being destroyed at shutdown ("What Changed?" — Deploy Markers and the Invisible Deploys in Performance).
- • An application-lifetime owner is new infrastructure: it must be constructed, injected, drained at shutdown and configured with a bound.
- • Every spawn site now names its work, which is a small tax that pays for itself the first time a metric identifies which background task is failing.
- • The durable path — an outbox or queue for must-happen work — is genuinely more machinery, and it is the correct machinery for that requirement.
- • Bounding introduces a new decision at saturation: does spawn block, drop, or reject? The same question as every other bounded queue (Bounded vs Unbounded Queues).
- • Await it. If the work takes 5 ms, the whole discussion is unnecessary and sequential code has no lifetime problem.
- • A durable queue or transactional outbox for work that must happen, which survives crashes as no in-process owner can (Message Queues in Architecture).
- • A scheduled reconciliation job that fixes up whatever the best-effort path missed — often simpler and more reliable than making the inline path perfect.
- • Do not do the work. A surprising share of fire-and-forget tasks are for data nobody consumes, and deleting them is the cheapest available fix.
What people believe, and what is true
Fire-and-forget is fine because the work is not critical.
Then it should be bounded and counted, so you know it is not happening. "Not critical" and "invisible" are different properties, and only the first one was chosen.
The runtime will report it if the task fails.
It reports a context-free warning at collection time, if at all. No trace id, no user, no metric, no alert — which is why these run broken for months.
Keeping a reference to the task is just for tidiness.
In CPython it is required: without a strong reference the task can be collected mid-execution and the work simply stops, silently.