Worker Processes
Running N copies of your service in one machine buys cores and isolation, and multiplies every per-process resource by N.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What actually changes when one process becomes eight, on the same machine?
The service runs on a multi-core instance and uses one core. We want to use the machine we are paying for without changing the application.
Set workers to the number of cores. Same code, more throughput, nothing else changes.
The database refuses connections. Each worker built its own pool, so eight workers with a pool of ten is eighty connections from one instance (Connection Pools).
- The database refuses connections. Each worker built its own pool, so eight workers with a pool of ten is eighty connections from one instance (Connection Pools).
- The in-memory cache hit rate collapses: there are now eight independent caches, each with an eighth of the traffic warming it (Local vs Distributed Cache).
- A scheduled job runs eight times, once per worker, because the schedule lives in the process (Scheduled Jobs).
- An in-process rate limiter allows eight times the intended limit (Rate Limiting).
- Memory use is far above the estimate, because each worker holds its own heap, its own caches and its own loaded modules (Stateless Services).
- A local lock that protected a critical section now protects one-eighth of the executions (Pessimistic Locking).
What is actually happening
- A master process binds the listening socket and forks N workers, each a full copy of the application with its own memory, its own connections and its own event loop or thread pool (Accepting Connections).
- Workers share the listening socket, so the kernel distributes accepted connections among them. Which worker gets which connection is the kernel's or the runtime's decision, and it is not perfectly even.
- On a fork-based system, the children initially share the parent's memory pages copy-on-write. In practice the savings erode as each worker writes — in CPython, even reading an object touches its reference count, so shared pages are copied sooner than the model suggests.
- Isolation is the second benefit and often the bigger one: a worker that crashes, leaks or segfaults takes down one of N. The master restarts it and the service stays up (Error Boundaries: Three Translations, Not One).
- Everything per-process is now per-worker: connection pools, in-memory caches, rate-limit counters, scheduled timers, warm JIT state, prepared statements. Multiply before deploying.
- A graceful reload works by starting new workers and retiring old ones once they finish their in-flight requests, which is how a process manager achieves a zero-downtime restart on one machine (Graceful Shutdown).
- In a container-orchestrated deployment this same choice appears one level up: N workers inside one container, or N containers with one worker each. The multiplication arithmetic is identical; the operational story is not (Containerizing a Backend).
What gets multiplied
The pattern behind every failure in this lesson is the same: something that was one thing is now N things, and the code that assumed one thing did not change.
Go through the list before increasing a worker count. Two of these rows are the ones that cause real incidents — connections and anything that was supposed to happen once.
| Per-process thing | With N workers | Consequence |
|---|---|---|
| Connection pool of size P | N x P connections | Database connection limit reached; other services cannot connect (Connection Pool Exhaustion) |
| In-memory cache | N independent caches | Hit rate falls; N times the memory for the same working set |
| Rate-limit counter | N counters | Published limit multiplied by N (Rate Limiting) |
| In-process lock | N locks | Protects nothing across workers (Pessimistic Locking) |
| Scheduled timer | N timers | The job runs N times (Scheduled Jobs) |
| Heap and loaded modules | N heaps | Container memory limit sized for one worker kills all of them |
| Warm-up cost (JIT, caches) | N warm-ups | Slower and lumpier recovery after a deploy or a recycle |
| A crash | 1 of N | The one thing that gets better: isolation and restart granularity |
Master and workers, and the shared socket
The structure is worth picturing because it explains both the isolation benefit and the accept race. One process owns the listening socket; the workers inherit it and all call accept on the same queue.
It also explains graceful reload: the master can start new workers on the same socket and retire the old ones once their in-flight requests finish, so the socket never stops being served (Rolling Deployments).
Do the multiplication before the deploy
This is the lesson's one actionable rule, and it takes a minute. Every per-worker resource has a total, and the total is what the shared systems experience.
The arithmetic below is the version worth putting in a runbook, because the failure it prevents shows up as an outage in a service that has nothing to do with the deploy that caused it.
Instances (or pods) : 6 Workers per instance : 8 Pool size per worker : 10 --------------------------------------------------- Connections to the database : 6 x 8 x 10 = 480 Postgres max_connections : 100 <-- refused, hard And remember the ones that are NOT the database: in-memory caches : 48 independent caches scheduled jobs : each cron entry fires 48 times in-process rate limits : published limit x 48 memory : 8 x per-worker heap, per instance Fixes, in the order to try them: 1. shrink pool per worker (10 -> 2) => 96 connections 2. move the schedule out of the process entirely 3. move counters and locks to a shared store 4. put a connection proxy (e.g. PgBouncer) in front of the database 5. only then reconsider the worker count itself
How to build it
Most important first.
- Compute the fan-out explicitly: instances x workers x pool size, checked against the database's connection limit, before the first deploy. Write it down somewhere that gets reviewed (Connection Pool Exhaustion).
- Shrink the per-worker pool as you add workers. Eight workers with a pool of two is often better than two workers with a pool of eight, because the constraint is total connections, not per-worker capacity.
- Move anything that must happen once out of the workers: scheduled jobs to a scheduler or a leader-elected job, counters and locks to a shared store (Atomic Operations).
- Handle
SIGTERMin the worker: stop accepting, finish in-flight requests, close pools, then exit — and make sure the grace period exceeds your longest normal request (Graceful Shutdown). - Create connections after the fork, never before. A file descriptor shared across processes by forking is a corruption bug that appears under load (Connection Pools).
- Set worker recycling (
max_requestswith jitter, or a maximum age) as a containment measure for slow leaks, and keep looking for the leak (Memory Leaks in Backend Services). - Decide deliberately between "N workers in one container" and "N containers with one worker": one container per worker gives the orchestrator direct control over scheduling and limits; N workers per container amortises the base image and startup cost (Deployment Models).
What can go wrong
- Connection exhaustion at a shared database, which typically surfaces first in a completely different service that can no longer connect.
- Memory limit exceeded in a container because the limit was sized for one worker's heap; the orchestrator kills the container, not the greedy worker.
- Thundering herd on accept: many workers woken for one connection, most losing the race and going back to sleep, wasting CPU under high connection rates.
- Uneven load distribution: some workers hot, others idle, because connections are long-lived and were distributed unevenly at accept time (Keep-Alive and Connection Reuse).
- A worker stuck rather than crashed — deadlocked or blocked forever — silently reducing capacity with no restart, which is what a worker timeout is for.
- The mitigation failing: worker recycling set aggressively enough that workers spend a meaningful share of their lives starting up, with cold pools and cold caches.
- Thundering herd: several workers wake on one incoming connection and race to accept it.
SO_REUSEPORTgives each worker its own accept queue, which reduces contention and changes the fairness characteristics (Accepting Connections). - During a rolling reload, old and new workers serve concurrently, so two code versions run against one database at the same time — which is precisely why migrations must be backward compatible (Expand and Contract Migrations).
- Anything in-process that assumed exclusivity — a lock, a counter, a "run once at startup" side effect — now happens N times concurrently (Backend Races).
- A connection created before the fork is shared by every worker, so their reads and writes interleave on one socket and corrupt the protocol.
- Worker isolation is a genuine security boundary: separate address spaces mean a memory-disclosure bug in one worker cannot read another worker's heap. Threads inside a worker have no such boundary.
- Recycling workers bounds the lifetime of any state an attacker manages to plant in memory, which is a small but real containment property.
- Secrets loaded before fork are copied into every worker. That is usually fine and is worth knowing when reasoning about memory disclosure (Secrets Are Not Configuration).
- Per-worker rate limiting is a security failure, not just a correctness one: your published limit is silently multiplied by the worker count (Rate Limiting).
- "Workers are threads." They are processes with separate memory. Anything you expected to share is not shared.
- "Set workers to core count and move on." Core count is a starting point for CPU-bound work. For I/O-bound services the right number depends on memory per worker and on the downstream connection budget.
- "Copy-on-write means workers are nearly free." Memory sharing erodes quickly as workers run, and in CPython reference counting erodes it faster than most people expect.
- "More workers means more throughput." Only until a shared resource saturates — usually the database — after which more workers make throughput worse by adding contention.
- "The process manager handles graceful shutdown." It sends a signal. Whether in-flight requests finish is your handler's behaviour (Graceful Shutdown).
Operating it
- Metrics must carry a worker or process identifier, or aggregation hides the case where one worker is broken and the rest are fine.
- Busy workers versus total workers is the saturation signal; without it, saturation is indistinguishable from a slow dependency (Why Is My API Slow?).
- Worker restart count and restart reason. A steady trickle of restarts is a leak, a crash or an over-aggressive timeout, and all three are worth different responses.
- Requests handled per worker: a skewed distribution means uneven connection assignment rather than uneven work.
- Total database connections from the application versus the database's limit — the number that the multiplication produces, measured rather than calculated.
- At 10x, worker count per instance stops being the interesting number and instances times workers becomes it. Connection multiplication is usually the first hard wall.
- At 100x, a connection proxy in front of the database becomes necessary, because thousands of application processes cannot each hold connections to a database that supports hundreds (Connection Pools).
- The isolation benefit scales well and the memory cost scales linearly: at large fleet sizes, per-worker memory is a substantial and often overlooked share of the bill (Horizontal vs Vertical Scaling).
- Isolation and multi-core use, paid for in memory per worker and multiplied downstream connections.
- No shared in-process cache: each worker warms its own, so hit rates fall and a shared cache becomes more attractive — with a network hop attached (Local vs Distributed Cache).
- Fewer, bigger workers keep pools smaller and concentrate blast radius; more, smaller workers spread risk and multiply resources. There is no default that is right at both ends.
- One worker per container gives the orchestrator precise control and multiplies base memory and startup cost across containers.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- RUNTIME-SPECIFICUniversal in shape, different in motivation. Python and Node use workers to get more than one core for application code at all, because each process runs application code on one thread; a Go or JVM service already uses every core in one process, so workers there are about isolation and restart granularity rather than parallelism.
- CLOUD-SPECIFICUnder a container orchestrator the same decision is usually expressed as replicas rather than workers, and the platform prefers one process per container so it can schedule, limit and restart at that granularity — which makes the per-worker resource multiplication a cluster-wide number rather than a machine-local one (Running a Backend on Kubernetes).
- GENERALThe multiplication rule — every per-process resource is multiplied by the process count — holds regardless of language, platform or orchestrator.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.