The Backlog Arithmetic: Four Levers and a Drain Time
10,000 jobs/s arriving, 8,000/s processed, backlog growing at 2,000/s. The gap is arithmetic, not opinion — and there are exactly four things you can do about it. Time-to-drain is the number to put in the incident channel.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The arithmetic, and the number people actually want
A growing backlog is one subtraction. Arrivals λ at 10,000/s, service μ at 8,000/s, so the backlog grows at λ − μ = 2,000/s. After twenty minutes that is 2.4 million queued jobs. This is not a modelling question or a judgement call; it is the only part of an incident where you can state the future with confidence.
The number to report is time-to-drain, and it depends on what happens next. If arrivals stay at 10,000/s and you do nothing, drain time is infinite — the backlog grows forever, and saying "it will catch up overnight" is a guess with no arithmetic behind it. If arrivals fall to 6,000/s at the end of peak, the queue drains at μ − λ = 2,000/s and 2.4 million jobs take 20 minutes to clear. If you double workers to 16,000/s while arrivals hold, it drains at 6,000/s and clears in under seven minutes.
Notice what that arithmetic exposes: the backlog you have already accumulated has to be paid for *on top of* the incoming work. Capacity that exactly matches arrival rate never drains a backlog — it just stops it growing. This is the single most common miscalculation in a queue incident, and it is why "we scaled to match traffic" is followed an hour later by "why is it still behind?".
1lambda = 10_000 # arrivals per second2mu = 8_000 # processing per second, current capacity3depth = 2_400_0004 5# 1. Are we still falling behind, and how fast?6growth = lambda - mu # +2,000/s -> yes, still growing7 8# 2. If nothing changes, when does it drain?9# mu <= lambda -> never. Say "never", not "eventually".10drain_now = depth / (mu - lambda) if mu > lambda else INFINITY11 12# 3. What capacity clears it inside the deadline we owe someone?13# Backlog must be paid on top of incoming work:14deadline_s = 900 # 15 minutes15mu_required = lambda + depth / deadline_s16# = 10,000 + 2,400,000/90017# = 12,667/s -> ~1.6x current capacity, not 1.25x18 19# The trap: scaling to 10,000/s stops the growth and drains nothing.Four levers, and what each one costs
Once the gap is quantified, the option space is small and finite. You can raise μ by adding workers. You can raise μ by making each job cheaper. You can lower λ by reducing arrivals. Or you can change *which* work is in the queue at all — shed the non-critical, prioritize the rest. Everything proposed in an incident channel is one of these four wearing different clothes.
Adding workers is the reflex and it is the right first move surprisingly often — but only if the workers are actually the constraint. If each worker spends 90% of its time waiting on a database that is already saturated, doubling workers doubles the pressure on the database and μ barely moves; you have converted a queue problem into a database problem (see Twenty Workers, All Busy, Five Hundred Waiting for how to tell before you scale, and The Bottleneck Moves After Every Fix for what happens after).
Shedding is the lever teams reach for last and should often reach for second. A queue mixing "send password reset" with "recompute weekly analytics" has an obvious answer: the analytics jobs can wait an hour and the password resets cannot. Dropping or deferring the low-value work is instant, costs nothing, and buys the time to do the other three levers properly. Its price is honesty — you need to have decided in advance which work is droppable, because deciding that at 3am under pressure goes badly.
| Lever | Mechanism | Speed | Fails when | Cost |
|---|---|---|---|---|
| Add workers (raise μ) | More consumers processing in parallel | Minutes — bounded by Autoscaling Lag: The Gap Where the Outage Lives and warm-up | Workers are not the constraint; a shared downstream is | Money, plus more load on whatever the workers call |
| Cheaper work (raise μ) | Optimize the job: batch DB writes, drop an N+1, skip redundant work | Hours to days — this is a code change | The cost is inherent to the work, not accidental | Engineering time; risk of a new bug during an incident |
| Reduce arrivals (lower λ) | Rate-limit producers, coalesce duplicate jobs, debounce | Minutes | Arrivals are genuine user demand you cannot refuse | Producers now handle rejection; upstream complexity |
| Shed / prioritize | Drop droppable work, or serve high-priority classes first | Immediate | Nothing in the queue is actually droppable | Requires deciding value *before* the incident |
The lever that is not on the list
Increasing the queue's retention or maximum depth is not a lever — it is a decision to make the incident longer and quieter. A bigger buffer absorbs more backlog before anything visibly breaks, which sounds like relief and is actually the removal of your last honest signal. Queue depth limits, like timeouts, exist to convert a slow silent failure into a fast loud one.
The same applies to raising consumer timeouts so that slow jobs stop failing. The jobs were not failing because the timeout was wrong; they were failing because the work now takes longer than the promise. Extending the promise silently is a product decision being made by an on-call engineer at 3am, and it will be discovered a month later by a customer.
This connects directly to Concurrency Limits: An Unbounded Server Is a Slower Server and Architecture → Backpressure. A queue with a bounded depth that rejects producers when full is telling the truth: the system cannot absorb this work right now. An unbounded queue tells nobody anything until memory runs out or the oldest job is a day old. Bounded, loud and early beats unbounded, silent and late.
Key points
- Backlog growth is λ − μ: one subtraction that lets you state the future instead of guessing at it.
- Time-to-drain is the number the incident channel wants; if μ ≤ λ the honest answer is "never", not "eventually".
- Capacity matching arrival rate stops growth and drains nothing — the accumulated backlog must be paid on top of incoming work.
- There are exactly four levers: more workers, cheaper work, fewer arrivals, or less work in the queue. Everything else is one of these renamed.
- Raising queue limits or consumer timeouts is not a lever — it converts a loud failure into a silent one that lasts longer.
Progressive depth
Overview
Work arrives at one rate and is processed at another. If arrivals exceed processing, the difference piles up. That pile is the backlog, and it grows until one of the two rates changes.
Practical
Quantify it: growth is λ − μ, and time-to-drain is depth ÷ (μ − λ) once μ exceeds λ. To clear a backlog inside a deadline you need μ_required = λ + depth/deadline — always larger than λ alone. Report drain time, not depth.
Advanced
Scaling μ rarely scales linearly, because workers share downstream dependencies. Doubling workers against a saturated database raises μ by a fraction and degrades the database, moving the bottleneck rather than removing it (The Bottleneck Moves After Every Fix). Check worker utilization against downstream saturation before choosing the lever.
Internals
Under the arithmetic sits queueing behaviour: as utilization ρ = λ/μ approaches 1, waiting time grows non-linearly even *before* λ exceeds μ, because arrivals are bursty rather than evenly spaced. A queue at 95% utilization is already producing long waits with no backlog growth at all — which is why Queueing: Why Systems Get Slow Before They Get Broken and Saturation: The Reading Utilization Cannot Give You matter for capacity planning, and why targeting ρ near 1 is a trap.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Producers → queue: λ rises to 10,000/s at peak; unique job ids rise proportionally, so this is demand, not Retry Storms: The Load You Generated Yourself.
- 2Queue → workers: μ holds at 8,000/s; per-worker throughput unchanged, worker count unchanged — capacity simply does not meet demand.
- 3λ − μ → depth: backlog accumulates at 2,000/s, reaching 2.4M after twenty minutes; the growth line is straight, confirming a steady shortfall rather than a stall.
- 4Depth → head age: oldest job crosses four minutes, breaching the one-minute promise on user-facing work sharing this queue.
- 5Scaling attempt → database: doubling workers raises μ to only 11,000/s, not 16,000/s, because the shared database becomes the new constraint — the bottleneck moved.
- • "We scaled workers to match 10,000/s, so we are fine" — matching λ halts growth but drains zero backlog; you need λ + depth/deadline.
- • "It will catch up overnight" — only true if λ falls below μ overnight. State the assumption, or the number is fiction.
- • "The backlog is growing, so the workers must be degraded" — check per-worker throughput first; usually μ is unchanged and λ moved.
- • "Doubling workers doubles throughput" — only when workers are the constraint. If they are waiting on a shared dependency, μ grows sublinearly and the dependency degrades.
- • "Raise the queue depth limit so it stops rejecting" — that removes the signal, not the problem, and lengthens the outage.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • λ and μ on one graph in the same units, so the gap is visible as a gap rather than inferred from two dashboards.
- • Current depth, to feed the drain-time arithmetic: `depth / (μ − λ)` once μ exceeds λ.
- • Required capacity for a deadline: `μ_required = λ + depth / deadline_seconds` — compute it before choosing a scaling target.
- • Per-priority-class depth and head age, so "what can we shed?" has an answer with numbers attached.
- • Worker utilization and downstream saturation together, to check whether workers are the constraint before scaling them ([[worker-pool-saturation]]).
- • Shed or defer the droppable work first — it is immediate, free, and buys time for the other levers. This requires a pre-agreed priority classification.
- • Compute `μ_required = λ + depth/deadline` and scale to *that*, not to λ — then verify the scaling actually raised μ rather than moving the bottleneck.
- • Rate-limit or coalesce producers where the arrivals are duplicates or low-value, lowering λ at the source.
- • Make the job cheaper (batch writes, remove per-job N+1 queries) — slowest to deliver, but it is the only lever that permanently changes the capacity equation.
- • Split priority classes into separate queues so user-facing work is never behind a batch backlog again.
- • After scaling, confirm μ actually rose by the expected factor on the processing-rate graph — sublinear growth means the constraint moved downstream.
- • Watch depth's *slope* turn negative, not just its value; a falling backlog is the proof, a lower number could be a purge.
- • Confirm head-of-queue age returns below the class SLO and stays there through the next peak, which is the user-facing definition of recovered.
- • Re-run the drain arithmetic with the post-fix μ and check the observed drain time matches the prediction — if it does not, the model is missing a constraint.
- • Scaling workers costs money continuously and pushes load onto shared dependencies that may have less headroom than the workers do.
- • Shedding work means someone does not get their result — acceptable for analytics recomputation, unacceptable for payments, and the classification has to be explicit.
- • Splitting queues by priority multiplies operational surface: more consumers, alerts and dashboards, and more ways for one to be misconfigured.
- • Making jobs cheaper is a code change during an incident, carrying the risk of shipping a new bug into the system that is already degraded.
- • An alert on sustained λ > μ for N minutes, which fires while there is still time to act instead of after the backlog is visible.
- • A load test at projected peak λ against current worker capacity, run before each seasonal peak, with the drain arithmetic in the report (Load Testing: What Question Is This Test Answering?).
- • A documented priority classification for every job type, agreed in daylight, so shedding is a lookup rather than a debate at 3am.
- • Capacity headroom tracked as a ratio (μ / peak λ) with a target, so the gap is visible on a dashboard before it is visible to users (Headroom: The Capacity You Deliberately Do Not Use).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 10k/8k/2.4M figures exist to make the subtraction legible. The arithmetic transfers to any scale; the specific numbers do not.
- ESTIMATEDDrain-time projections assume λ and μ hold constant. Real arrival rates follow daily curves, so a projection is only as good as the traffic assumption stated alongside it.