Cloud & Infrastructure Practice
Read the incident report, decide what is worth inspecting, and commit to a diagnosis before anything is confirmed. Only some of the evidence is revealing and nothing tells you which — that is the skill being practised. Every wrong turn carries the reason it fails.
Beginner
One layer at a time — reachability, a route, a permission, a health check. The symptom and the cause are usually one hop apart.
A service was redeployed at 11:20 and none of its pods stay up: 0/1 Ready, status CrashLoopBackOff, restart count climbing. The image is byte-for-byte the one that has been running since yesterday. The only thing in the merge was a ConfigMap update.
Roughly half of a service's pods are Running and the other half are stuck in ImagePullBackOff. The deployment went out four hours ago and was fine. This morning the cluster gained a second node pool for capacity, and every pod that lands on the new nodes fails to pull the image.
Intermediate
Two components interacting: a rollout against a readiness probe, a security group against an application tier, a limit against a runtime.
The checkout API started returning 500s at 09:14. Every request logs connect ETIMEDOUT 10.0.3.41:5432 after exactly 30 seconds. Nothing was deployed this morning — the last release was Thursday. The database console shows the instance available, CPU near idle, and zero connections from the API.
Scaling a Deployment from 6 to 10 replicas has left four pods Pending for twenty minutes. The cluster dashboard reads 38% CPU and 41% memory used across fourteen nodes, so there is obviously room. Two more nodes were added an hour ago and the pods are still Pending.
One pod of the report-rendering service restarts most nights between 03:00 and 03:20. No alert has ever fired, because the Deployment always has other replicas ready and the restart is over in ten seconds. Someone finally looked at the pod: Restart Count: 46, last state Terminated, Exit Code: 137.
A routine terraform plan on the platform repository — no code change, just a rebase before some unrelated work — reports Plan: 1 to add, 0 to change, 0 to destroy. The module declares three application instances. The console shows two. Nobody remembers deleting anything and the service has been serving traffic normally all week.
Finance flags that last month's infrastructure spend is roughly three times the month before. Engineering shipped no new services, instance count is unchanged, and traffic is up about 15%. The leading theory in the thread is that the provider raised prices.
At 00:00 UTC the mobile app stopped working for every user: SSL handshake failed. The web app shows a browser interstitial about an invalid certificate. Every dashboard is green — instances healthy, all load balancer targets healthy, application error rate zero, latency normal. The first thing the on-call engineer checked, an internal monitoring endpoint, returned 200 OK.
Advanced
The evidence contradicts itself, the obvious suspect is innocent, and the reassuring signal is the one that is lying.
The nightly export job failed at 02:10 with AccessDenied: not authorized to perform s3:PutObject on arn:aws:s3:::exports-prod/nightly/2026-08-24.csv. The on-call engineer attached a policy granting s3:* on that bucket to the job's role and retried. Same denial. It ran successfully every night for eight months and nothing in the job changed.
A routine deploy started at 16:05. By 16:15 the service is serving 502s for about a third of requests, the fleet is down from 12 instances to 4, and the deployment is still "in progress". A rollback was requested at 16:16 and it is also stuck. Latency on the four surviving instances is eight seconds.
Between 12:00 and 12:40 the checkout service fails about one payment in six with ETIMEDOUT calling the payment provider. Every dashboard is green: instances healthy, all load balancer targets healthy, CPU 35%, database fine, error budget barely moving. The provider says the requests they receive all succeed — they just receive fewer than we say we sent.
A bad migration at 14:05 dropped a column and rewrote 40 million rows in the orders table. The decision to restore from last night's backup was made at 14:20. It is now 17:30, the restore is still running, the estimated finish keeps moving, and somebody has finally asked how long this is supposed to take. The runbook says "RTO: 1 hour".
Every weekday at 09:00 the service sheds requests for three to four minutes: latency climbs to six seconds, a few percent of requests time out, then everything recovers on its own. The autoscaling policy fires correctly every morning and adds the right number of instances. By the time they serve traffic the peak is over. The team has lowered the threshold twice and nothing changed.
Expert
Cross-cutting failures — identity, delivery, state and drift together — where the fix is cheap and finding it is not.
Practise under pressure
Three ways to be wrong before production is: predict a failure, size a system, and design one from parts.
Predict what users see before the impact is revealed, then read the signal that lied.
The same system at four loads. At the first rung the correct answer is to change nothing.
Place components, then have the analyzer tell you what you exposed and what you over-built.