Resource Limits
Every finite resource needs an explicit limit, or the system discovers its own — at the worst possible moment, in the worst possible way.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Which limits does my service have, and which of them did I actually choose?
The service should stay up under a traffic spike, and degrade in a way we can see and reason about instead of falling over.
Set the connection pool size, since that is the one the framework asks about, and rely on autoscaling for everything else.
A request body with no size limit lets one upload allocate hundreds of megabytes, and the instance is killed for memory rather than for load (Request Bodies and Streaming).
- A request body with no size limit lets one upload allocate hundreds of megabytes, and the instance is killed for memory rather than for load (Request Bodies and Streaming).
- A query with no
LIMITreturns a million rows for the largest tenant, and the serialization of that result is what exhausts memory (Pagination That Survives a Large Table). - A fan-out with no concurrency cap opens thousands of sockets from a single request (Unbounded Concurrency).
- No timeout on an external call means a stalled dependency holds request slots until every one is occupied (Timeouts).
- Autoscaling adds instances that all connect to the same database, so scaling out converts a compute shortage into a connection shortage (Serverless and Database Connections).
What is actually happening
- Every resource in a backend is finite: memory, connections, file descriptors, threads, sockets, CPU, disk, and the capacity of everything you call. A limit exists for each whether or not you chose it.
- An unchosen limit is discovered at runtime and fails badly: an OOM kill takes down every in-flight request, a file-descriptor exhaustion refuses new connections, a full pool blocks endpoints unrelated to the one causing the load.
- A chosen limit converts exhaustion into rejection or queueing, both of which are observable, boundable and shed-able. That is the entire value: it does not make the resource larger, it makes running out predictable (Backpressure).
- Limits compose in a chain, and the effective limit is the tightest one. A service permitting 500 concurrent requests with a 20-connection pool is a 20-concurrency service with 480 requests queued somewhere less visible (Little's Law as Working Intuition).
- Queueing is not free. A request waiting for a resource still occupies memory and a slot, and a long queue is indistinguishable from an outage from the client's side — which is why bounded queues with timeouts beat unbounded ones (Bounded vs Unbounded Queues).
- Limits belong at several layers because each catches what the others cannot: the edge caps request rate and size, the application caps concurrency and payloads, the database caps connections and statement duration, and the platform caps memory and CPU.
The inventory nobody has
The most useful exercise in this lesson is not conceptual. It is to sit down and list every finite resource the service touches, write next to each one what its limit is, and mark whether anyone chose that number. The unmarked rows are the future incidents, in roughly the order of how close they are to binding.
Most teams find two or three limits they did not know existed, one framework default that is wildly wrong for their payloads, and at least one resource with no limit at all — usually a collection size or a fan-out.
The inventory is also what makes capacity conversations concrete. "Can we handle 10x?" is unanswerable in general and straightforward once you know which limit is at 40% utilisation and which is at 4%.
| Resource | Where the limit is set | What happens with no limit | Leading indicator |
|---|---|---|---|
| Request body size | Per route, at the edge and in the app | One upload exhausts memory | 413 rate; body size distribution |
| Result set size | LIMIT and enforced page size | Largest tenant OOMs the instance | Rows returned per query, p99 |
| In-flight fan-out | Semaphore at the call site | Sockets and buffers until OOM (Unbounded Concurrency) | In-flight gauge |
| DB connections | Pool size, per instance | Refused connections at the server | Acquire wait time (Connection Pools) |
| Request concurrency | Server or middleware limit | Memory per queued request; latency collapse | Active requests over the cap |
| External call duration | Client timeout | Slots held by a stalled dependency | p99 duration vs timeout (Timeouts) |
| Query duration | statement_timeout | One query holds a connection for minutes | Long-running query count |
| Lock wait | lock_timeout | Unbounded waits, pool exhaustion | Lock wait time (Pessimistic Locking) |
| Process memory | Container limit | Host-level pressure or OOM kill | RSS over limit; OOM kill count |
| Queue depth | Bounded queue + shed policy | Backlog grows until storage or memory fails | Queue age (Queue Backlog) |
| Requests per caller | Rate limiter | One caller consumes everything | Rejections per caller (Rate Limiting) |
Limits compose, and the tightest one wins
Limits are not independent settings; they form a chain from the edge to the database, and the system's real concurrency is the minimum along it. Raising a limit that is not the binding one changes nothing except where the queue forms.
This is why capacity work so often feels futile. The server concurrency is raised from 100 to 500, and throughput does not move, because the 20-connection pool was the constraint and now there are simply more requests waiting for it — in a place with worse observability than before.
The corollary is that the interesting number is not any single limit but the utilisation of each, expressed as a fraction. The one nearest to 1.0 is the system's capacity, and it is the only one worth changing.
- 1Edge rate limit
Caps requests per caller per window.
fails by Keyed on a forgeable header; or absent, so one caller consumes everything (Rate Limiting).
- 2Body size limit
Rejects oversized payloads before they are buffered.
fails by A framework default sized for JSON while the route accepts uploads.
- 3Server concurrency
Caps simultaneously-processed requests.
fails by Set far above the pool size, so the queue moves somewhere unmeasured.
- 4Application fan-out bound
Caps work started per request.
fails by Absent — the single most common gap (Unbounded Concurrency).
- 5Connection pool
Caps concurrent database work.
fails by Usually the true system limit, and usually reasoned about last.
- 6Statement timeout
Caps how long one query holds a connection.
fails by Unset, so one bad plan occupies a connection indefinitely.
- 7External call timeout
Caps how long a dependency holds a slot.
fails by Longer than the caller's own timeout, so work outlives its client.
- 8Container memory
Caps the process.
fails by Reached first when an earlier limit is missing; the kill takes everything in flight.
Effective concurrency is the minimum across this chain. Raising any other number relocates the queue rather than raising capacity.
Queue, reject, or shed
Retry-After is sane, that shedding drops what you intended (Performance Testing a Backend).Once a limit binds, the system must do something with the excess, and there are only three options: make it wait, refuse it, or drop lower-value work to protect the rest. The choice is a product decision as much as a technical one.
Queueing is right when the wait is short and the caller is patient — a background job, an internal batch. It becomes wrong quietly: a queue that grows past the client's timeout is doing work nobody will receive, which is worse than having refused it (Queue Backlog).
Rejecting is right for interactive traffic, where a fast, clear failure with Retry-After lets the client back off intelligently. It requires the honesty to return 429 or 503 rather than letting requests pile up and time out, which looks better on a dashboard and is worse for everyone (Backpressure).
Shedding is the mature version: when saturated, drop the work that matters least — analytics writes, prefetches, non-critical enrichment — so the checkout path keeps working. It requires knowing which requests are which, which is a design decision made long before the incident.
Can the caller wait, and does this work still have value when it completes?
when Short waits, patient callers, work that keeps its value: jobs, batch, async.
cost Latency, and memory per queued item. Unbounded queues turn saturation invisible (Bounded vs Unbounded Queues).
when Interactive requests where a fast failure beats a slow one.
cost Clients must handle it; a badly-behaved client retries immediately and makes it worse (Retry Storms).
when Mixed traffic where some work is clearly less important.
cost Requires classifying every request, and the classification must be trustworthy.
when A partial answer is useful: cached data, fewer fields, no enrichment.
cost Two code paths, and the degraded one is exercised only under stress — so it is rarely tested (Circuit Breakers).
when The constraint is per-instance and the spike outlasts the scaling delay.
cost Latency before it helps; useless or harmful when the constraint is shared (Autoscaling a Backend).
when Never, deliberately — but this is the default when nothing else is chosen.
cost The failure mode is whichever resource runs out first, and it takes in-flight work with it.
How to build it
Most important first.
- Write down every limit the service has and mark which were chosen. The unmarked ones are your incidents (The Backend Security Checklist has a related discipline).
- Set a request body size limit per route, sized to real payloads rather than to a framework default (Request Bodies and Streaming).
- Cap every collection: page sizes, batch sizes, array lengths in request bodies, result set sizes. Validate them at the edge and clamp rather than trusting the client (Transport Validation).
- Bound concurrency wherever work fans out, and size the bound from the tightest downstream constraint (Unbounded Concurrency).
- Give every external call a timeout, and make the timeouts shorter as you go deeper so an inner call cannot outlive its caller (Timeouts).
- Set database statement timeouts and lock timeouts, so one pathological query cannot hold resources indefinitely (Pessimistic Locking).
- Isolate resources by class of work: separate pools or instances for user traffic, batch jobs and webhooks, so one cannot starve the others (Bulkheads).
- Set container memory and CPU limits deliberately, and know which signal the platform sends when they are hit (Containerizing a Backend).
- Load-test to find where each limit binds, so the first time you learn the answer is not during an incident (Performance Testing a Backend).
What can go wrong
- Limits set at one layer and not another, so the tightest one is a resource nobody instrumented.
- A limit too high to ever bind, which is the same as no limit but harder to notice because a number is present.
- A limit too low, rejecting legitimate traffic — the failure mode people fear, which is why limits are so often set high enough to be useless.
- Unbounded queues in front of a bounded resource, converting fast failure into slow timeouts and hiding the saturation (Queue Backlog).
- A timeout longer than the caller's timeout, so work continues for a client that has already given up (The Request Lifecycle).
- Autoscaling reacting to a symptom rather than the constraint: adding instances when the bottleneck is a shared database only makes the contention worse (Autoscaling a Backend).
- Limits enforced per instance and reasoned about globally, so the effective total scales with the deploy size.
- No limit on the number of concurrent long-lived connections, so a slow-client attack occupies every slot (Keep-Alive and Connection Reuse).
- Concurrent requests all passing a limit check that is a read-then-increment, so the limit is exceeded by the width of the window (Atomic Operations).
- Per-instance limits enforced independently while the resource they protect is shared, so the effective limit is N times what was configured.
- A resource freed and immediately re-acquired by a waiter, so a shed decision made a moment earlier is stale.
- Autoscaling adding instances while a shared limit is already saturated, so new instances make contention worse before they help (Cascading Failure).
- A semaphore permit leaked on an error path, tightening the limit over time until throughput collapses (Semaphores: Counting Permits as a Resource Limit).
- Every uncapped input is a denial-of-service vector: body size, array length, page size, upload count, header size, query complexity. Capping them is a security control, not a performance tweak (Every Input Surface).
- Resource limits are what turn a resource-exhaustion attack from an outage into rejected requests. The attacker's goal is to find the resource with no limit.
- Per-tenant limits prevent one customer, compromised or merely enthusiastic, from consuming everything (Multi-Tenancy).
- Rate limits must be enforced server-side and keyed on something the caller cannot forge; a limit keyed on an unfiltered
X-Forwarded-Foris not a limit (Rate Limiting). - Slow-client attacks work by holding connections rather than by sending volume, so connection count and idle timeouts matter as much as request rate.
- "Autoscaling means we do not need limits." Autoscaling changes how much of a resource you have. It does nothing about resources that are shared, and it reacts on a timescale slower than a spike (Autoscaling a Backend).
- "The framework has sensible defaults." Defaults are chosen without knowledge of your payloads, dependencies or traffic. A default is a starting point, not a decision (Configuration: Separating Code From Environment).
- "We have a rate limit, so we are protected." A rate limit bounds requests. It does not bound the work each request creates (Unbounded Concurrency).
- "Limits are premature optimisation." They are failure-mode selection. The question is not whether the system will hit a limit but whether you choose which one and what happens.
- "More capacity fixes it." More capacity moves the bottleneck. Knowing where it moves to is the useful part (Why Is My API Slow?).
Operating it
- Utilisation of each limited resource as a fraction of its limit — pool in use over pool size, in-flight over the concurrency cap, memory over the container limit. Absolute numbers do not tell you how close you are.
- Wait time at each bound: pool acquire time, semaphore wait, queue age. Waits rise before errors, so they are the leading indicator (Saturation: The Reading Utilization Cannot Give You).
- Rejection counts by limit, which tell you which bound is binding right now.
- Container OOM kills and restart counts, which are silent in application logs because the process does not get to write one (Memory Leaks in Backend Services).
- A dashboard listing every limit with its current utilisation. The exercise of building it usually finds two or three limits nobody knew existed.
- Limits that never bind at 1x bind at 10x, in an order you can predict by measuring utilisation now.
- At 100x the binding limit is usually shared and external — the database's connection ceiling, a partner's rate limit — and no amount of application-side capacity changes it (Read Replicas From the Application).
- Horizontal scaling multiplies per-instance limits and does not multiply shared ones, so scaling out moves the bottleneck onto whatever is shared (Horizontal vs Vertical Scaling).
- Autoscaling needs a signal that reflects the actual constraint. Scaling on CPU when the bottleneck is pool wait adds instances that make the contention worse (Autoscaling Signals).
- Every limit rejects something. Set them too tight and you fail legitimate work; too loose and they do not protect. Being roughly right and instrumented beats being precise and unmeasured.
- Isolation by bulkhead reserves capacity per class of work, which means capacity sits idle when that class is quiet.
- Queueing improves utilisation and adds latency; rejecting improves latency and discards work. Which is right depends on whether the caller can retry (Backpressure).
- Per-tenant limits are fairer and require tracking state per tenant, which is itself a resource.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALEvery backend has finite resources and therefore has limits; the only choice is whether you set them.
- RUNTIME-SPECIFICWhat binds first differs by runtime. Node typically exhausts memory or file descriptors on one loop thread, so the symptom is an OOM kill or refused connections. A thread-per-request JVM or Python service exhausts the thread pool first, so the symptom is queueing while memory looks fine. Go tends to reach the connection pool or the remote service before goroutines cost anything.
- CLOUD-SPECIFICManaged platforms impose their own ceilings you cannot exceed from inside the application — load balancer idle and request timeouts, per-function concurrency and duration limits, per-account connection quotas. These are typically the tightest limits in the system and the least visible from application code; the specific values differ by provider and product and must be looked up rather than assumed.
- SCALE-SPECIFICAt low volume most limits never bind and setting them is cheap insurance. The order in which they start binding is measurable now, by watching utilisation fractions, long before any of them is a problem.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — load shedding and admission control as system-wide properties rather than per-service settings.