6 lessons

Capacity & Cost

How much load the system can take, how much headroom is left, whether autoscaling arrives in time, and what a request actually costs — capacity and efficiency as separate questions.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

Capacity Planning: Traffic to Machines
▶ lab

Average traffic, a peak multiplier, per-request cost and a latency target become a instance count through five multiplications — each one an assumption you can name, challenge and re-measure. The output is an estimate, and saying so is what makes it useful.

Symptom · Nobody in the planning meeting can say what happens at three times current traffic. The last scaling decision was "double it and see", and the one before that was an incident.
Headroom: The Capacity You Deliberately Do Not Use

Running at 100% utilization means every burst, every failed instance and every deploy becomes an incident. How much headroom is a real decision with a real cost, and the number comes from burst shape, scale-up time and blast radius — not from a convention.

Symptom · The service is comfortable at steady state and falls over during deploys, instance replacements and modest traffic bumps — events that are all entirely predictable.
Autoscaling: Scaling on the Right Signal

An autoscaling policy is a claim about what your bottleneck is. Scale on CPU and you have claimed the service is CPU-bound; when it is actually waiting on a database, the policy never fires while users time out.

Symptom · Latency climbs and requests time out, but the fleet size never changes. The scaling dashboard shows the policy at rest, well below its threshold, throughout the entire incident.
Autoscaling Lag: The Gap Where the Outage Lives

Between a traffic spike and a new instance serving real traffic sit five delays: the metric window, the evaluation interval, provisioning, boot, and warm-up. Add them up honestly and you often find the spike ends before the capacity arrives.

Symptom · Traffic spikes, latency degrades for two to four minutes, then recovers. The scaling dashboard shows instances arriving right around the time things got better, and everyone credits autoscaling for a recovery it did not cause.
Cost per Request: The Other Performance Metric

Compute, database, cache, bandwidth, third-party calls and inference add up to a number per request. Track it next to latency and a whole category of "optimizations" reveals itself as buying a small latency win with a large permanent bill.

Symptom · The infrastructure bill grows faster than traffic. Nobody can say which endpoint is expensive, and every performance improvement so far has involved adding a replica, a cache tier or a bigger instance.
Capacity or Efficiency: Which Problem Are You Solving?

Capacity asks how much load the system can take. Efficiency asks how much resource each unit of work consumes. A system can scale beautifully while wasting most of what it buys — and the two problems have different fixes, different costs and different urgency.

Symptom · The fleet keeps growing to keep up with traffic. Each scale-out works, so nothing looks broken, but the ratio of machines to users has been drifting upward for a year and nobody planned for that.