Overload & Backpressure

Inside one process, backpressure is a blocking call: the producer stops because the consumer will not take the item. Across machines, "stop" is a message. It takes time to arrive, it can be ignored, and it usually reaches a queue rather than a producer — which is not backpressure, it is buffering.

Q · My service is overloaded. How does the pressure actually get back to whoever is generating the work?

Above capacity you will not serve every request. The only question is whether the system chooses which ones to drop, or lets timeouts choose at random. Shedding is that choice made deliberately — and it only works if a rejection costs far less than a success.

Q · I cannot serve everything. Which requests do I drop, and how do I drop them without spending the capacity I am trying to save?

An unbounded queue does not absorb overload; it postpones it and makes it worse. First latency climbs past every deadline, so the work you finally do is worthless. Then memory runs out. Admission control is the decision to find out at the door instead.

Q · Should this request be let in at all, given what I already owe?

A single user request becomes 20 service calls becomes 200 database calls. Now let each tier retry up to three times. The database does not see 3× the load, it sees 27× — because retries at independent tiers multiply rather than add, and every tier thinks it is being modest.

Q · Every tier retries three times, which seems reasonable. Why is the bottom of my stack seeing thirty times the traffic?

"Retry up to three times" sounds like a limit. It is not: it bounds one request while leaving the aggregate unbounded, because the number of requests is not something you control. A budget bounds the thing that actually hurts — total retry traffic as a share of original traffic.

Q · What is the right limit on retries, if "three per request" does not bound anything that matters?

Exponential backoff spaces out one client’s attempts. It does nothing about the fact that ten thousand clients all failed at the same instant and are all now counting down the same interval. Backoff without jitter reproduces the spike; it just reproduces it one second later.

Q · I added exponential backoff and the load spikes are still there, just further apart. Why?

You split the monolith into forty services, so a failure in one should stay in one. It does not, because the blast radius follows shared resources — a thread pool, a connection pool, a node pool, a database — and service boundaries drawn on a diagram do not cut any of those.

Q · One service is failing. Why is the failure spreading, and what actually stops it?

One shared pool gives the best utilisation and the worst isolation: whoever misbehaves takes everyone with them. Separate pools give the opposite. The engineering question is not which is better, it is how finely to cut — and the arithmetic of that cut is unforgiving in both directions.

Q · How do I stop one tenant, one dependency or one bad query from consuming the capacity everyone else needs?