The question this answers
How do I stop one tenant, one dependency or one bad query from consuming the capacity everyone else needs?
A stated maximum share of a resource per class of work, so that exhaustion by one class cannot starve another. It bounds the *resource*, not the outcome: a class that exhausts its own partition still fails completely, and the partition is only real if the underlying resource is genuinely separate.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows its own pool occupancy per class. It does not know whether the resource behind the pool is truly independent — two "separate" connection pools may reach the same database, two pods may share a host, two partitions may share a garbage collector — so a node cannot verify its own isolation. That verification is an infrastructure question, and it is where most bulkhead designs turn out to be decorative.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The cost of the cut, in one calculation
Take 200 connections and four dependencies. One shared pool means any dependency can use all 200 — excellent utilisation, and one slow dependency parks all 200. Four pools of 50 means a slow dependency parks 50 and the other three keep working. The isolation is real and so is the cost: each dependency’s ceiling has dropped from 200 to 50, so a legitimate burst on one dependency now fails while 150 connections sit idle.
This is the general shape. Sizing k partitions for the peak of each means provisioning k × peak where a shared pool needed peak_of_sum, and because peaks rarely coincide, peak_of_sum is much less than sum_of_peaks. You are paying for statistical multiplexing you have chosen to give up. For uncorrelated demand across k partitions, the aggregate peak grows roughly with √k while partitioned capacity grows with k.
The way out is not to choose one extreme. Reserve a floor per class and let a shared surplus be borrowed: 30 reserved per dependency (120) plus 80 shared. Every class is guaranteed 30 no matter what anyone else does, and a burst can reach 110. Isolation for the worst case, utilisation for the common one. Nearly every good bulkhead design has this two-part shape.
| Design | Normal-case ceiling per dep | Available to others when dep A hangs | Idle capacity at peak |
|---|---|---|---|
| One shared pooltypical | 200 | 0 | none |
| Four pools of 50typical | 50 | 150 | up to 150 |
| 30 reserved each + 80 sharedtypical | 110 | 90 (their reservations + shared) | small |
| Weighted fair share, no fixed splitassumption | 200 | 150 (A capped at its share) | none — requires a scheduler |
What to isolate on, and the layer where it becomes real
Isolation has to be applied to the resource that actually saturates, and teams routinely pick the wrong one. Partition connections and a slow dependency still consumes threads. Partition threads and a memory-hungry tenant still exhausts the heap. Partition CPU and a single query still locks the table everyone reads. The first job is identifying which resource is the binding constraint under the failure you care about — and it is often a different resource than the one that binds during normal operation.
The dimension you partition on matters just as much. By dependency is the classic bulkhead: one pool per downstream, which stops a slow dependency from freezing unrelated work. By tenant protects multi-tenant systems from a noisy neighbour. By criticality keeps a lane free for critical work — and reserving a lane for health checks and admin endpoints costs almost nothing and prevents the orchestrator from killing instances mid-incident. By operation type separates cheap reads from expensive writes or reports.
And the isolation is only as strong as the weakest layer beneath it. Two thread pools share a heap and a garbage collector: a memory-exhausting workload in one crashes both. Two pods on one node share a kernel, a page cache and a network interface. Two database users on one instance share a buffer pool and a WAL. Each level of the stack has to be isolated separately, and the effective boundary is the lowest level that is still shared — which is exactly the reasoning in Fault Domains: What Fails Together, applied to capacity instead of to crashes.
1class PartitionedPool {2 private readonly used = new Map<string, number>()3 private sharedUsed = 04 5 constructor(6 private readonly reserved: Map<string, number>, // guaranteed floor per class7 private readonly sharedCapacity: number, // borrowable surplus8 ) {}9 10 // A class always gets its reservation; beyond that it competes for the surplus.11 // The reservation is what makes the guarantee statable: class B can name a12 // number it will always have, regardless of what class A does.13 tryAcquire(cls: string): 'reserved' | 'shared' | null {14 const floor = this.reserved.get(cls) ?? 015 const inUse = this.used.get(cls) ?? 016 17 if (inUse < floor) {18 this.used.set(cls, inUse + 1)19 return 'reserved'20 }21 if (this.sharedUsed < this.sharedCapacity) {22 this.used.set(cls, inUse + 1)23 this.sharedUsed++24 return 'shared'25 }26 return null // this class is at its limit; others are unaffected27 }28}29 30// Health checks and admin endpoints get their own reservation. It costs almost31// nothing and it is what stops the orchestrator killing healthy instances32// during an overload.33const pool = new PartitionedPool(34 new Map([['checkout', 40], ['search', 30], ['reports', 10], ['control-plane', 5]]),35 115,36)Shuffle sharding: better isolation than partitioning, for free
Straight partitioning of tenants across n nodes gives each tenant one node, so a tenant that takes its node down affects everyone assigned to that node — with 8 nodes and 800 tenants, one bad tenant fully breaks 99 others.
Shuffle sharding assigns each tenant a random *subset* of nodes instead. With 8 nodes and a subset size of 2, there are C(8,2) = 28 distinct assignments. A bad tenant degrades both of its nodes, and another tenant is fully affected only if it drew *exactly the same pair* — roughly 1 in 28. The rest overlap on at most one node and keep working at reduced capacity if the client can retry to its other node.
The scaling is what makes it striking: 100 nodes with a subset of 5 gives C(100,5) ≈ 75 million combinations. The probability that a given tenant shares a full shard with a specific bad tenant is about 1 in 75 million — while every node still serves many tenants, so utilisation stays high. You buy near-per-tenant isolation at near-shared-pool utilisation, and the only cost is routing. That is an unusually good trade, and it is why the technique appears throughout large multi-tenant infrastructure.
Two honest caveats. It relies on the client being able to use its other shard members when one is bad, so it needs retry-to-a-different-endpoint to work. And it isolates *only* against per-tenant faults: a bad deploy, a poisoned config or a shared-dependency outage hits every node regardless of how you drew the subsets.
nodes=8, tenants=800 fully affected partially affected shared (all tenants, all nodes) 799 799 partitioned (1 node per tenant) 99 99 shuffle shard, subset=2 (C=28) ~28 ~171 nodes=100, tenants=10,000, subset=5 C(100,5) = 75,287,520 combinations expected tenants sharing an identical shard with a given tenant: ~0.0001
Key points
- Isolation and utilisation trade directly:
kpartitions each sized for peak costsk × peakwhere a shared pool needed the peak of the sum. - The two-part design — a reserved floor per class plus a borrowable shared surplus — gets most of the isolation at a fraction of the cost.
- Partition the resource that actually saturates under the failure you care about; connections, threads, memory and CPU each fail differently.
- Isolation is bounded by the lowest shared layer: two pools in one process share a heap; two pods on a node share a kernel.
- Shuffle sharding delivers near-per-tenant isolation at near-shared utilisation, and requires the client to be able to use its other shard members.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Identify the resource that saturates first under the failure being contained, and the dimension worth separating (dependency, tenant, criticality, operation type).
- • Set a reserved floor per class, sized from the class’s minimum viable throughput rather than its peak.
- • Leave the remainder as a shared surplus that any class may borrow when it is free.
- • Reject immediately when a class is at its ceiling, rather than queueing across the boundary — a shared wait queue re-couples what the pools separated.
- • Verify at every layer beneath: separate processes, hosts, node groups or database instances where the guarantee needs to hold through a crash, not just through saturation.
- • The partitions share a lower layer — heap, kernel, buffer pool, network path — and the isolation exists only on paper.
- • Reservations sum to more than the resource, so the guarantee is unfundable and the first class to ask simply loses.
- • A shared wait queue in front of the partitions reintroduces head-of-line blocking across classes.
- • Partitions are sized once and never revisited, so a class that has grown 10× is throttled while another class’s reservation sits idle.
- • Isolation is per process, and the multi-process deployment multiplies every limit by the worker count.
- • Isolation that is not: one dependency hangs and every class degrades together. The thread dump shows all workers in one call, or the heap dump shows one class’s objects dominating — the boundary was above the resource that actually saturated.
- • Utilisation collapse: p99 rises and rejections appear while overall resource usage sits at 40%. Somewhere a class is at its ceiling next to a large idle reservation, which is the signature of partitions sized for peak.
- • Control-plane starvation: health checks and admin endpoints share the request pool, so an overloaded instance fails its probes and is killed, reducing capacity mid-incident.
- • Cross-tenant correlation despite sharding: many tenants degrade at once and shuffle sharding was supposed to prevent it. The cause is a fault that is not per-tenant — a bad deploy or a shared dependency — against which shard assignment does nothing.
- • Local pool partitioning needs no coordination and gives a per-instance guarantee only; the fleet-wide guarantee is that number times the instance count.
- • Fair-share scheduling gives better utilisation than fixed partitions but needs a scheduler that observes all classes — an extra component on the hot path.
- • Shuffle sharding needs agreement only on the assignment function, which can be a deterministic hash of the tenant id. That makes it coordination-free at request time, which is most of why it scales.
- • Classes within their reservation keep their full guarantees while another class is exhausted — the claim, and the reason to accept the utilisation cost.
- • A class that exhausts its own partition fails completely; isolation contains the failure, it does not prevent it.
- • Borrowed shared capacity disappears first under contention, so the effective ceiling drops to the reservation exactly when load is highest — the reservation must be sized for that moment, not for the quiet one.
- • Detect: per-class utilisation against per-class limits. Aggregate utilisation is actively misleading here — 40% overall with one class at 100% is the case you must be able to see.
- • Contain: the partition contains by construction. The incident lever is to raise the ceiling of the starved class, not to remove the boundary.
- • Recover: an exhausted class recovers on its own as its work drains; nothing needs to be re-enabled if rejections were clean.
- • Reconcile: work rejected at a class boundary must be re-driven by its owner or explicitly abandoned — a bulkhead assumes someone has a plan for the rejection.
- • Verify: saturate one class in a load test and confirm the others show no latency change at all. Any correlation reveals a shared layer you did not account for.
- • Per-class pool utilisation and per-class rejections, which is the only view that distinguishes "system is full" from "one lane is full".
- • Reservation versus actual usage per class over time — the input for resizing, and the evidence for whether the split still matches reality.
- • Shared-surplus occupancy: consistently at 100% means the reservations are doing all the work and utilisation is suffering.
- • Cross-class latency correlation during a single-class saturation test. It should be zero; anything else is a leak in the boundary.
- • Multi-tenant systems where one tenant can generate pathological load, and shuffle sharding is available.
- • Services calling several dependencies with different reliability characteristics, where one hung dependency must not freeze the rest.
- • Any system where control-plane traffic must survive data-plane overload — the cheapest and highest-value reservation there is.
- • Small deployments where partitioning leaves each class too small to serve its normal traffic, and the isolation causes the outage it was meant to prevent.
- • Highly variable workloads where peaks genuinely do not coincide: a shared pool is both cheaper and more resilient, and partitioning throws away the multiplexing that was protecting you.
- • When the partitions are nominal because the resource beneath is shared, in which case you have paid the utilisation cost and bought nothing.
- • Per-class rate limits instead of per-class pools: cheaper to implement, bounds arrivals rather than occupancy, and does not handle a class whose requests are individually slow.
- • Fair-share scheduling with no fixed partitions, which keeps utilisation high at the cost of a scheduler on the request path.
- • Separate processes or separate deployments per class — the strongest isolation available, and the most expensive in operations and cost.
- • Do nothing, and rely on Decide at the Door Whether the Capacity Exists to keep total load bounded. Correct when all work is genuinely equivalent and no class needs protecting from another.
Shuffle sharding: a random subset per tenant
| Assignment | Fully affected | Partially affected | Utilisation |
|---|---|---|---|
| shared — every tenant on every node | 799 | 799 | best |
| partitioned — 1 node per tenant | 134 | 134 | poor — each node sized for its own peak |
| shuffle shard, subset 2 | ~29 | ~371 | near-shared — every node still serves many tenants |
What people believe, and what is true
We have separate connection pools, so the dependencies are isolated.
Only against connection exhaustion. If the caller uses one thread pool, a hung dependency still parks every thread — isolate the resource that actually saturates.
More partitions means more isolation, so partition everything.
Each partition sized for its own peak multiplies your provisioning and can cause the very rejections it was meant to prevent. Reserve a floor and share the surplus.
Shuffle sharding is just sharding.
Sharding gives each tenant one node; shuffle sharding gives each a random subset, so the number of distinct blast radii is combinatorial rather than linear. That is the entire effect.
Isolation at the application layer is enough.
Two pools in one process share a heap and a garbage collector; two pods share a kernel and a NIC. The effective boundary is the lowest shared layer, wherever that turns out to be.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Give each class of work its own slice of the pool so one class cannot consume everything. You pay for it in utilisation, because slices sized for peak sit idle most of the time.
Practical
Find the resource that saturates first, split it by dependency or tenant with a reserved floor plus a shared surplus, always reserve a small lane for health checks and admin traffic, and reject at the class ceiling rather than queueing across it. Verify by saturating one class and checking the others do not move.
Advanced
Use shuffle sharding where the fault is per-tenant: C(n,k) distinct assignments turn a linear blast radius into a combinatorial one, so 100 nodes with a subset of 5 gives ~75 million shards while each node still serves many tenants. Then be precise about what it does not cover — correlated faults such as a bad deploy or a shared dependency ignore shard assignment entirely, and those are the outages that actually take the whole system down.
Apply it
- 💬 You split one 200-connection pool into four pools of 50 and rejections went up while CPU went down. What happened?
- 💬 Explain shuffle sharding to someone who already understands sharding, and say what it does not protect against.
- 💬 Which resource should you isolate to protect against a dependency that becomes slow rather than failing?