Load Shedding
Deciding in advance what to drop when demand exceeds capacity, so the system fails in the shape you chose.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
When there is more work arriving than the system can serve, what should it stop doing?
An overloaded system without a shedding plan still sheds — by timing out on everything equally, which is the least useful distribution of failure available.
Accept every request and let them queue. Dropping a user's request feels like giving up, and the queue will drain when the burst passes.
Unbounded queues convert an overload into a latency collapse. Requests wait past the client's timeout, so the work is done and then thrown away — the system is fully busy producing nothing.
- Unbounded queues convert an overload into a latency collapse. Requests wait past the client's timeout, so the work is done and then thrown away — the system is fully busy producing nothing.
- Clients that time out retry, and the retries arrive on top of the load that caused the timeouts. Accepting everything is what makes a retry storm possible (Capacity Management).
- Without prioritisation, a bulk export and a checkout request are served in arrival order, so the cheap high-value request waits behind the expensive low-value one.
- Resources are held while queued work waits — connections, memory, threads — so the queue itself consumes the capacity that would have served it.
- When the system does start failing, it fails at the deepest, most expensive point: after the database work, not before it.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Shedding is admission control. Something decides, as early and as cheaply as possible, that a unit of work will not be served, and says so immediately.
- The value of shedding comes almost entirely from where it happens. A request rejected at the edge costs a connection; the same request rejected after a database round trip has already spent the resource you were trying to protect.
- Bounding the queue is the simplest form: a fixed limit on in-flight or queued work, with immediate rejection past it. This converts unbounded latency into bounded latency plus visible errors, which is a strictly better failure mode because it is legible.
- Prioritisation makes shedding a choice rather than a lottery. Requests carry a class — critical path, background, bulk, retry — and classes are shed in reverse order of value.
- Shedding must be visible to the client in a way it can act on: a status that means "try later", with a hint of how much later, so the retry does not arrive immediately and make things worse.
- The mechanism only works if the shed path is cheap. A rejection that runs authentication, database lookups and full logging is not shedding; it is serving an error at full cost.
Shed early or pay twice
The same rejection has wildly different costs depending on where it happens. This is the entire design principle: push the decision as far towards the client as the information allows.
- 1At the edge or CDN
Rejects by client, path or crude rate before a connection reaches your fleet.
fails by Cannot see per-user class or session context, so it is blunt.
evidence Edge rejection count, with origin request rate flat behind it.
- 2At the load balancer
Bounds concurrent connections and surge queue depth per target.
fails by Rejects by connection, not by value — a checkout and a bulk export look identical.
evidence Surge queue length and rejected connection count.
- 3At service admission
Applies a concurrency limit and a request class before any work begins.
fails by Needs the class to be present on the request; missing class means default treatment.
evidence Shed rate by class, and in-flight count against the limit.
- 4Before the expensive dependency
Refuses work that would queue on a saturated pool or a tripped breaker.
fails by Some request cost has already been paid — parsing, auth, allocation.
evidence Breaker state and pool wait time, from the caller side.
- 5Inside the dependency
The database or broker rejects because it is at its own limit.
fails by Most expensive possible rejection, and it affects every caller at once.
evidence Connection refusals and server-side error counts.
- 6Nowhere — client timeout
The request is served, slowly, into a socket nobody is listening on.
fails by Full cost, zero value, and the client is already retrying.
evidence Latency above the client timeout with a low server-side error rate — the classic signature.
A queue that cannot say no
The single most common overload bug is a queue with no bound. It looks harmless in review because the queue is nearly always empty, and it is the reason overloads turn into collapses rather than into degradations.
request -> queue (no limit) wait grows without bound client times out at T server finishes at T + delta response discarded client has already retried -> offered load rises -> wait grows faster
request -> admission check
in-flight < limit ?
yes -> serve, bounded wait
no -> reject immediately
retryable status + backoff
-> latency stays bounded
-> shed rate rises visibly
-> downstream load stays flatBoth systems fail at the same arrival rate. The first fails invisibly and amplifies, spending full resources on responses nobody receives; the second fails legibly and protects everything behind it. The difference is not capacity — it is whether the system is allowed to say no.
What goes wrong with shedding itself
Shedding is a mitigation, and mitigations have failure modes of their own. These are the ones that turn a protection into an amplifier.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Limit applied to all traffic uniformly | Checkout fails at the same rate as bulk export | No request class, so shedding is a lottery | Classify at the entry point and shed in reverse value order |
| Health checks subject to the limit | Instances marked unhealthy and removed during overload | Shedding the traffic that proves the instance is alive (Probes: Readiness, Liveness and Startup) | Exempt health and control-plane paths from admission limits |
| Client retries immediately on rejection | Shed rate and arrival rate both climb together | Rejection is cheap enough to retry hard, with no backoff | Return backoff guidance; enforce jittered backoff in shared clients (Retryability: Telling Clients What To Do Next in APIs) |
| Fixed limit, service got slower | Saturation with the limit never reached | The limit was tuned against an older service time | Use adaptive concurrency, or re-tune whenever service time shifts materially |
| Shedding at the service, not the edge | Fleet CPU high while rejecting almost everything | Rejection path runs auth, parsing and full logging | Move the check earlier; make the shed path allocate almost nothing |
| Sustained shedding accepted as normal | Steady low-level error rate for weeks | Shedding masked chronic under-capacity, so nobody escalated | Alert on sustained shed rate as a capacity signal, not as an error (An Alert Should Demand Action) |
How to do it properly
Most important first.
- Bound every queue and every pool. An unbounded queue is a deferred outage with extra steps (Operating Queues and Scheduled Work).
- Shed at the outermost layer that can make the decision — the load balancer or gateway if the class is knowable there, the service entry point otherwise.
- Classify traffic before you need to. Retries, background jobs, bulk API consumers and internal callers are the natural first candidates to drop.
- Return a retryable status with backoff guidance rather than a timeout, and make sure your own clients honour it — Backend and APIs own the client-facing contract for rate limits (Production Anti-Patterns).
- Prefer concurrency limits to request-rate limits for protecting a service, because concurrency is what actually correlates with resource exhaustion.
- Test the shed path under load. It is code that only ever runs on your worst day, which is the code most likely to be broken.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Shedding is itself the containment mechanism — the risk is shedding the wrong class or the wrong share, which a per-class limit and a shed-rate dashboard bound.
What can go wrong
- Shedding so late in the request path that the rejection costs nearly as much as the success would have.
- A limit set once and never revisited, so it sheds during normal peaks after the service got faster, or never triggers after it got slower.
- Shedding the wrong class: dropping health checks or control-plane traffic, so the platform decides the instance is dead and removes it, reducing capacity further (Probes: Readiness, Liveness and Startup).
- Clients that treat a shed response as a hard failure and retry immediately, turning shedding into amplification.
- The mitigation failing: shedding hides sustained under-capacity, so the system runs permanently degraded and nobody escalates because there is no outage.
- "Shedding means we under-provisioned." It means you bounded the failure. Every system has a limit; shedding decides what happens at it.
- "Rate limiting is load shedding." Rate limiting enforces a contract per client, continuously. Shedding is a response to your own saturation, applied to whoever is unlucky. Systems need both, for different reasons.
- "Queue it, do not drop it." Only if a delayed answer is still worth something. For a synchronous request whose caller has already timed out, queueing is throwing away capacity.
- "We shed, so we are protected." You are protected up to the cost of shedding. Past a high enough arrival rate, even rejecting is work you cannot keep up with.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A shed-rate metric, by class, on the operator dashboard — separate from the error rate, because shedding is a decision and not a fault.
- Latency staying bounded during an overload event while shed rate rises: the signature of shedding working.
- A load test taken past the limit that shows the service returning quick rejections instead of timing out.
- Downstream dependencies showing flat load during the overload — proof that shedding protected them.
- Limits are configuration and should be revertible without a deploy, because the first time you tune them will be during an incident (A Config Change Is a Production Change).
- Raising a limit during an incident is a decision to accept more load; write down that you did it, because a limit raised at 3am and never lowered is how the next incident starts.
- Automate the mechanism: adaptive concurrency limits that track observed service time are more robust than a fixed number someone chose a year ago.
- Automate the shed-rate alert, since sustained shedding means the capacity conversation has been deferred rather than settled.
- Keep class policy human. Which customers or features get dropped first is a product and contract decision, not an engineering default (Quotas vs Rate Limits in APIs owns the contract).
- Shedding trades errors for availability of the rest. Some fraction of users get a clear failure so the majority get service — a choice that should be made deliberately and communicated.
- Prioritisation needs a class on every request, which means plumbing through the whole call graph and keeping it accurate as the system changes.
- Adaptive limits are harder to reason about during an incident than fixed ones: the system's behaviour is now a control loop with its own dynamics.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALAdmission control is universal. Where it can be implemented is not: a managed gateway may offer concurrency limits directly, while a plain VM fleet needs it inside the application or at a proxy in front of it.
- ORG-SPECIFICWhich traffic is shed first encodes a commercial policy — internal before external, free tier before paid, bulk before interactive. There is no technically correct ordering, only one your contracts and product owner agree with.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — why every client in a fan-out retrying at once produces a load spike no amount of capacity absorbs.