The Bottleneck Moves After Every Fix
You removed the CPU bottleneck and the system is still slow — because the constraint moved to the database, where it had been hiding behind the CPU limit all along. This is what success looks like, and predicting the next constraint is what separates a plan from a sequence of surprises.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
One constraint at a time, by definition
A system's throughput is set by its most constrained resource, and only by that one. Everything else has slack. This has an unintuitive consequence: until you relieve the current constraint, improvements to anything else produce approximately zero end-to-end benefit — and the moment you *do* relieve it, some other resource becomes the limit, immediately and without warning.
So "I fixed the bottleneck and it is still slow" is not a failed optimization. It is the expected outcome of a successful one, observed before the next iteration. The mistake is treating it as failure and reverting, or as a mystery and re-investigating from scratch. The correct response is to ask which resource is the constraint *now* — a question the same instrumentation answers in a minute, because the utilization ranking was already there before the fix, showing you what was in second place.
The reason it feels like a surprise is that the second-place resource is genuinely invisible while the first-place one saturates. If CPU pins at 100%, the application never issues enough concurrent database queries to reveal that the connection pool is one size too small. Relieving CPU raises the offered load on everything downstream, and the pool saturates the same day (Connection Pool Saturation: Waiting in Front of an Idle Database). The database problem did not appear; it was always there, masked.
Predicting where it moves
Rank every resource in the request path by utilization *before* you ship the fix. The one in second place is your next constraint, and knowing it converts a surprise into a plan. You can then decide whether the first fix is even worth shipping alone: if CPU is at 98% and the pool is at 94%, relieving CPU buys you very little before the pool binds, and the honest answer is to fix both or neither.
The prediction needs one correction, because resources do not scale linearly. Relieving a constraint raises offered load on downstream resources, and a resource at 70% utilization can pass its knee and behave far worse than "70% plus a bit" once it receives 40% more work (Queueing: Why Systems Get Slow Before They Get Broken, Saturation: The Reading Utilization Cannot Give You). So estimate the *post-fix* load on each downstream resource, not its current utilization. A database at 60% CPU that will receive twice the query rate is a much more imminent constraint than the ranking suggests.
This is also the honest answer to "how much faster will this make us?". The arithmetic that says "serialization is 55% of CPU samples, so removing it makes us 2x faster" is only true if CPU remains the constraint afterwards — and it usually does not. Predicting the next constraint lets you state the expected improvement as "roughly 15% until the pool binds, then flat", which is both more accurate and much more useful to whoever is deciding whether to fund the work (Self Time, Total Time, and Where the CPU Went).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| App CPU | 96% (constraint) | Currently binding. This is what the fix targets. | smoking gun |
| DB connection pool | 94% of connections busy | Second place, and it will receive more concurrent queries the instant CPU stops limiting. Next constraint. | suspect |
| DB CPU | 38% | Comfortable now, but will roughly double when the pool stops throttling query concurrency. Third. | suspect |
| Cache hit rate | 97% | Healthy and unlikely to move; not a candidate. | normal |
| Network bandwidth | 11% of link | Nowhere near binding. | normal |
| Worker queue depth | ~0, stable | Async path has ample headroom. | normal |
Knowing when to stop
Bottleneck migration is a loop, and loops need a termination condition. The condition is not "no bottleneck remains" — there is always a constraint, by definition. It is the SLO: stop when the user-facing objective is met with sufficient headroom, and spend the remaining engineering effort elsewhere (SLOs: A Target, a Window, and a Reason, Headroom: The Capacity You Deliberately Do Not Use).
Each iteration also costs more than the last. The first fix is usually a hot spot that a profile hands you. The third is a design change. The fifth is a rewrite or a re-architecture, and the improvement per iteration shrinks while the risk grows. Watching the cost-per-iteration curve is how teams avoid spending a quarter chasing a constraint that a slightly relaxed SLO would have made irrelevant (Every Optimization Buys Something and Sells Something).
There is a specific stopping signal worth naming: when the constraint moves *outside your system* — to a third-party dependency, a provider's network, the speed of light between regions (Cross-Region Latency Is Physics, Not Configuration) — the optimization loop as such is over. What remains is architectural: cache it, parallelize around it, move closer to it, do it asynchronously, or renegotiate the requirement. Recognising this transition early saves a great deal of effort aimed at a resource you do not own.
| Iteration | Constraint | Fix | p99 after | Effort | Next constraint (predicted) |
|---|---|---|---|---|---|
| Baseline | App CPU (96%) | — | 2.40s | — | Connection pool (94% busy) |
| 1 | App CPU | Replace reflective serializer on the list path | 2.05s | 2 days | Connection pool — bound almost immediately |
| 2 | Connection pool | Size from Little's Law; add wait-time metric | 1.35s | 1 day | DB CPU — query rate roughly doubled |
| 3 | DB CPU | Covering index removing the orders scan | 0.72s | 3 days | Payment dependency p99 |
| 4 | Payment dependency | Not ours. Made async with a job + webhook | 0.31s | 2 weeks | None binding — SLO met with headroom |
| Stop | SLO is 500ms p99 | Remaining work redirected to reliability | 0.31s | — | Re-evaluate at 2x traffic |
Key points
- Throughput is set by the single most constrained resource; improvements elsewhere produce approximately no end-to-end benefit until it is relieved.
- "Fixed it and it is still slow" is the expected outcome of a successful fix, observed before the next iteration — not a failed optimization.
- Rank resources by utilization before shipping; second place is your next constraint, and knowing it turns a surprise into a plan.
- Correct the prediction for the load increase your fix causes downstream — a resource at 70% can pass its knee under 40% more work.
- Terminate on the SLO with headroom, not on "no bottleneck remains"; when the constraint leaves your system, the work becomes architectural.
Progressive depth
Overview
Systems have one binding constraint at a time. Fix it and another takes its place — that is success, not failure.
Practical
Rank every resource by utilization before you ship. Second place is your next constraint. If first and second are close, fixing only the first buys almost nothing.
Advanced
Correct the ranking for load redistribution: relieving a constraint raises offered load downstream, and a resource at 70% utilization can cross its queueing knee under 40% more work, degrading far more than linearly (Queueing: Why Systems Get Slow Before They Get Broken). Predict post-fix load, not current utilization.
Internals
Constraints mask each other through concurrency limits that are often implicit. A saturated CPU limits how many database queries are in flight; a small connection pool limits how much work reaches the database; a bounded queue limits how much reaches the workers. Each of these is an admission-control mechanism nobody designed as one (Concurrency Limits: An Unbounded Server Is a Slower Server). Removing one raises concurrency everywhere downstream, which is why the effects are so abrupt — and why explicit concurrency limits make the whole system more predictable, since the limiter is then a parameter you chose rather than an accident of resource exhaustion.
Watch the Bottleneck Move
Change an input and watch which number moves — and which one does not.
The handler is expensive and the fleet is small. CPU is pinned.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Constraint → system: the most utilized resource caps throughput; every other resource runs with slack and looks healthy.
- 2Engineer → fix: the constraint is profiled, understood and relieved; its utilization drops sharply and the fix validates cleanly in isolation.
- 3Fix → downstream: offered load on every downstream resource rises, because the upstream limiter is no longer throttling concurrency.
- 4Downstream → new constraint: the previously second-place resource saturates, often within the same deploy, and end-to-end latency barely improves.
- 5Team → conclusion: "the optimization did not work" — when in fact it worked exactly as designed and revealed the constraint that was always behind it.
- • "The optimization did not work" — check the resource you targeted. If its utilization dropped as predicted, it worked; the constraint moved and the next iteration is the work.
- • "We have a new problem" — the new constraint was there all along, masked by the old one. Framing it as new leads to re-investigating from scratch instead of consulting the ranking you already had.
- • "CPU is at 40% now, so we have headroom" — headroom on the relieved resource says nothing; the binding resource is elsewhere and is the only one that matters (Headroom: The Capacity You Deliberately Do Not Use).
- • "Serialization is 55% of the profile, so we will be 2x faster" — only if CPU remains the constraint afterwards, which it usually does not (Self Time, Total Time, and Where the CPU Went).
- • "We should keep optimizing until nothing is saturated" — something is always the constraint. The termination condition is the SLO, not the absence of a bottleneck.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Utilization and saturation for every resource in the request path — CPU, memory, pool occupancy, queue depth, disk, network, dependency latency — captured as a ranked list before and after each fix ([[use-method]]).
- • End-to-end p50/p95/p99 as the outcome metric, since the resource metric you optimized will improve regardless of whether users benefit.
- • Request-duration attribution across layers from traces, so the shift in where time accumulates is directly visible ([[critical-path]]).
- • Offered load on each downstream resource after the fix — not its prior utilization, since relieving the constraint raises the load it receives.
- • Effort and elapsed time per iteration, so the diminishing-returns curve is visible to whoever is funding the work.
- • Rank every resource in the request path by utilization and predicted post-fix load before shipping, and write down the expected next constraint.
- • When two resources are close to binding, fix both together or accept that the first fix alone will produce little user-visible benefit.
- • State the expected improvement as a range that ends where the next constraint binds, rather than as the arithmetic of the resource being fixed.
- • Re-measure the full ranking after each iteration; the ranking, not intuition, names the next target.
- • Stop when the SLO is met with headroom, and redirect the remaining effort — the loop has no natural end of its own.
- • The targeted resource's utilization drops by roughly the predicted amount — this validates the fix itself, independently of end-to-end effect.
- • End-to-end p99 improves by roughly the amount predicted by the model that accounted for the next constraint; a much smaller gain means the next constraint bound sooner than expected.
- • The predicted next constraint is in fact the one that binds — a correct prediction validates the model and makes the next iteration cheap to plan.
- • Trace attribution shows time shifting from the fixed layer to the new constraint, rather than simply disappearing ([[trace-waterfall]]).
- • Fixing several constraints at once reduces the number of iterations and destroys attribution — you no longer know which change bought what.
- • Each iteration costs more and returns less; a quarter spent on iteration five may be worth less than a slightly relaxed SLO.
- • Optimizing for the current constraint can add complexity that becomes technical debt once the constraint moves elsewhere.
- • Predicting downstream load requires a model of the system that takes real effort to build, and the effort is only repaid across several iterations.
- • Alert on saturation for every resource in the path, not just the one that was historically the problem — the constraint moves, and alerting follows it too slowly.
- • Keep the utilization ranking on a dashboard so the current constraint is always one glance away, not an investigation (Dashboards Built Around Questions).
- • Re-run the ranking at traffic milestones (1.5x, 2x), since growth changes the ordering even with no code change (Capacity Planning: Traffic to Machines).
- • Record each iteration in a performance log with predicted versus actual outcomes; the accuracy of your predictions is itself a measure of how well you understand the system.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe four-iteration campaign, its latencies and its effort estimates are a constructed example. Real campaigns rarely have this clean a sequence, and the effort per iteration varies enormously by system and team.
- WORKLOAD-SPECIFICWhich resource sits in second place, and how much load a fix shifts downstream, depends entirely on your workload mix. The ranking must be measured per system; it cannot be inferred from architecture diagrams.