The Common Backend Failures
Thirteen failures that account for most backend incidents, each with the symptom that identifies it and the first diagnostic to run.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
When production breaks, what is it usually, and what is the first thing to check for each?
On-call needs something better than "read the logs and think hard". A short list of what actually goes wrong, indexed by what you can see from a dashboard at 02:00.
Every incident is unique, so keep a general debugging mindset and work from first principles each time.
It is not true. A small set of failures recurs across every backend in every language, and recognising one in the first two minutes is worth more than any amount of general cleverness.
- It is not true. A small set of failures recurs across every backend in every language, and recognising one in the first two minutes is worth more than any amount of general cleverness.
- First principles at 02:00 with an audience is a bad environment for careful reasoning. Recognition is robust under stress; derivation is not.
- Without a list, junior responders check what they know and stop, so the same causes go undiagnosed repeatedly.
- Post-incident reviews produce thirteen unrelated action items instead of noticing that four incidents shared one cause.
What is actually happening
- Most backend failures are resource exhaustion, feedback amplification, or a change with an unconsidered consequence — and the three present very differently.
- Exhaustion (pool, memory, file descriptors, worker slots) shows as queueing then timeouts, usually with a ramp rather than a step.
- Amplification (retry storms, stampedes, cascading failure) shows as a sharp non-linear spike, often after a small trigger, and gets worse when you add capacity to the caller.
- Change (deploy, migration, expiry, flag) shows as a step at a specific minute, usually fleet-wide.
- The symptom rarely names the cause, because backends are chains: an exhausted pool in service A presents as timeouts in service B and as a queue backlog in service C (Cascading Failure).
The list, with the first thing to check
Read this column-wise during an incident: find the row whose symptom matches what you are seeing, then run the diagnostic in the last column before you accept the cause. The confirmation step is what stops the list from becoming a source of confident wrong answers.
Nothing here requires deep knowledge of any framework. Every diagnostic is a metric, a database view or a single command.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Concurrency exceeds pool size, or connections held too long | Endpoint latency up, database query time flat, some requests time out waiting | Connection pool exhausted (Connection Pool Exhaustion) | Check pool waiters and acquire duration; then find the longest connection hold |
| A list endpoint loads related rows lazily | Latency proportional to result count; database busy with many tiny queries | N+1 query pattern (The N+1 Query Problem) | Count queries per request in a trace; look for one statement repeated with different ids |
| CPU-heavy synchronous work on the request-serving thread | Every endpoint on the process slows together, including trivial ones | Event loop blocked (Blocking the Event Loop) | Measure loop lag (or thread state); correlate with the endpoint that started it |
| References accumulate in a long-lived structure | Memory climbs over hours; GC runs harder; eventually OOM or restart | Memory leak (Memory Leaks in Backend Services) | Plot live heap after collection, not total heap; then take two snapshots and diff |
| A dependency degrades and every caller retries | Downstream request rate multiplies while success rate falls | Retry storm (Retry Storms) | Compare outbound attempts to outbound requests; look for retries of retries |
| Producers outrun consumers, or a consumer stalls | Async effects arrive minutes or hours late; queue age climbing | Queue backlog (Queue Backlog) | Compare enqueue rate with completion rate; check oldest message age and consumer errors |
| Write path does not invalidate, or invalidation fails silently | Users see old data; support reports "it fixed itself" after some minutes | Stale cache (Cache Invalidation) | Read the same key from cache and from the source of truth and compare |
| Two transactions take the same locks in different orders | Intermittent errors under load; database reports deadlock victims | Deadlock (Deadlocks in Application Code) | Read the engine's deadlock log; identify the two statements and their lock order |
| A client or queue retries a non-idempotent operation | Duplicate orders, double charges, doubled counters | Duplicate processing (Idempotency Keys) | Look for two records with the same business intent seconds apart, same correlation id |
| A credential or certificate reaches its expiry date | Sudden fleet-wide auth failures against one dependency, at a round timestamp | Expired secret or certificate (Secrets Are Not Configuration) | Check the credential's expiry; the error text usually names it if anyone reads it |
| Traffic growth or a new caller exceeds a provider quota | 429s from a dependency, or your own limiter rejecting legitimate users | Rate limit breached (Rate Limiting) | Read the response headers for remaining quota and reset window |
| A dependency stops responding and no timeout is set | Requests hang; threads or connections accumulate; the whole service degrades | External timeout missing (Timeouts) | Check the client's configured timeout — the default is often infinite |
| A schema change is applied that the running version cannot tolerate | Errors on some instances only, during or just after a rolling deploy | Bad migration (Expand and Contract Migrations) | Compare schema version to deployed application versions per instance |
The pairs that look identical from the dashboard
Several of these produce the same first-glance signal, and picking wrong sends the investigation in the opposite direction. The distinguishing evidence is usually one extra metric — which is exactly why it is worth having that metric before you need it.
| Looks like | Could be | Or could be | What separates them |
|---|---|---|---|
| Slow endpoint, busy database | One slow query | Hundreds of fast queries (N+1) | Query count per request |
| Slow endpoint, calm database | Pool exhaustion | Blocked event loop | Pool wait time vs loop lag |
| Memory climbing | Leak | Cache filling to its bound | Live heap after GC, and whether it plateaus |
| Dependency errors rising | Their outage | Your retry storm making it their outage | Their request rate from your side |
| Queue growing | Not enough consumers | Poison message stalling one partition | Consumer error rate and oldest-message age |
| Sudden fleet-wide failure | Bad deploy | Expired credential | Whether the timestamp matches a deploy or a round expiry |
| Duplicate records | Client retry | Consumer redelivery after a slow ack | Whether the duplicates share a correlation id or a message id |
Instrument for the list before you need it
Every diagnostic in the first table assumes a signal exists. Most services emit request latency and error rate and nothing else, which is exactly enough to know that something is wrong and not enough to know what.
The set below is small, cheap and almost entirely made of gauges the runtime and drivers already track. Adding them is an afternoon of work; adding them during an incident is impossible.
- 1Pool gauges
in-use, idle, waiting, acquire duration histogram
fails by Absent, so pool exhaustion is indistinguishable from a slow database
- 2Runtime saturation
Event-loop lag, thread-pool queue depth or worker busy ratio
fails by Absent, so a blocked process looks like a slow dependency
- 3Query count per request
A counter incremented by the data-access layer, attached to the trace
fails by Absent, so N+1 is invisible until someone reads a trace by hand
- 4Outbound attempts vs requests
Separates retries from first attempts per dependency
fails by Absent, so a retry storm looks like a traffic spike
- 5Queue depth and age
Backlog size and oldest unprocessed message
fails by Depth alone hides a stalled consumer with no new arrivals
- 6Live heap after GC
Distinguishes growth from leak
fails by Total-heap graphs make every managed runtime look like it is leaking
- 7Cache hit rate and evictions
Explains database load changes with no traffic change
fails by Absent, so stampedes are attributed to the database
- 8Deploy and migration markers
Puts change events on the same axis as symptoms
fails by Absent, so the highest-prior cause is the hardest one to check (Deploys Are the First Suspect)
How to build it
Most important first.
- Keep the list somewhere on-call can read it in ten seconds — this lesson is meant to be that artefact.
- For each entry, know the one metric that confirms it. Recognition without confirmation is how anchoring starts.
- Instrument for the list ahead of time: pool waiters, queue age, retry counts, heap after GC, cache hit rate, migration state. Every one of them is cheap and none of them can be added during the incident.
- Write the confirmed cause into a runbook entry after the incident, with the exact query or dashboard link that settled it.
- Prefer mitigations that do not destroy evidence — drain an instance rather than restarting it, so a dump can still be taken.
What can go wrong
- Pattern-matching to the last incident rather than to the evidence. The previous cause is the single most over-weighted hypothesis in any incident.
- Two of these happening at once — a deploy that caused a leak, or a dependency outage that triggered a retry storm — where fixing one leaves the symptom.
- A runbook that lists causes but not confirmations, so it produces confident wrong answers faster.
- Alerting on the symptom only (latency, error rate) so every one of these thirteen looks identical at the moment the page fires (The Metrics a Backend Must Emit).
- Duplicate processing is a race between a retry and the original request, or between two consumers of the same message (Duplicate Detection).
- Deadlocks require concurrency by definition and are therefore invisible in single-threaded testing (Deadlocks in Application Code).
- A cache stampede is many concurrent misses on one key racing to recompute it (Cache Stampede).
- Expired secrets and certificates are in this list because they are a *reliability* failure with a security cause; treat rotation as a scheduled operation with an alert before expiry, not an incident (Secrets Are Not Configuration).
- A bad migration can silently drop a constraint that was the last line of defence against a data integrity or authorization bug (Database Constraints).
- Duplicate processing on a payment or entitlement path is a financial and access-control problem, not only a correctness one (Idempotency in Backends).
- Incident-time log level increases and ad-hoc production queries are the most common route for personal data to leave its boundary (Secrets in Logs).
- "The database is down" — far more often the database is fine and your pool, your locks or your query count are not.
- "It is a memory leak" said about any process whose memory grows. Managed runtimes grow heap by design until pressure forces collection (Memory Leaks in Backend Services).
- "The queue is backed up so we need more workers." Sometimes. Also consistent with a poison message, a stalled consumer, or a downstream that is now the real limit (Queue Backlog).
- "Retries make us resilient." Retries make a *healthy* dependency's transient failures survivable and make an *unhealthy* one's outage worse (Retry Storms).
- "The deploy was fine, it has been running for an hour." Leaks, cache degradation and unbounded growth all take an hour to show.
Operating it
- Pool: in-use, waiting, acquire duration (Connection Pools).
- Queue: depth and age. Depth alone hides a stalled consumer that is not receiving new work.
- Retries: outbound attempt count versus outbound request count. A ratio that moves is a storm forming (Retries).
- Memory: heap size *after* collection. Live heap growing across collections is a leak; total heap growing is not (Memory Leaks in Backend Services).
- Cache: hit rate, and eviction count. A hit-rate cliff explains a database spike with no traffic change.
- Migration: schema version per instance during a rolling deploy (Expand and Contract Migrations).
- Locks: blocked query count and lock wait time from the database's own views (Deadlocks in Application Code).
- At scale the same failures arrive sooner and with less warning, because utilisation is higher and headroom is thinner.
- Amplification failures scale super-linearly: a retry storm across 200 instances is not ten times a storm across 20.
- Some of these are effectively scale-free — expired secrets and bad migrations hit a two-instance service exactly as hard.
- A checklist speeds recognition and encourages premature closure. The countermeasure is the confirmation column, not abandoning the list.
- Instrumenting for all thirteen costs metric cardinality and some engineering time up front, spent long before the incident that justifies it.
- Defensive measures interact: timeouts plus retries plus a breaker is more configuration surface, and misconfigured together they cause the incident they were meant to prevent.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe thirteen recur across languages and frameworks; only the tool that confirms each one differs.
- RUNTIME-SPECIFICEvent-loop blocking exists as such only on single-loop runtimes like Node; a thread-per-request JVM or Go service degrades gradually as the pool saturates instead, and CPython's GIL produces a third shape where one worker cannot run two handlers at all.
- DATABASE-SPECIFICDeadlock detection, lock granularity and what a long transaction blocks differ by engine: Postgres reports deadlocks and rolls back a victim, MySQL/InnoDB uses different lock ordering, and both differ from an engine with table-level locking.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — turning each confirmed cause into a runbook entry and an alert that fires before the symptom reaches users.