Distributed Systems Incidents
Real-shaped incidents worked the way they actually arrive: a symptom, several parties who each saw part of the truth, and a cause that is invisible until you know it and obvious afterwards. You commit to a diagnosis before the cause, the misreadings and the fixes are revealed.
Forty-seven customers were charged twice for the same order on a Tuesday afternoon. Both charges appear on the card statement 2 to 3 seconds apart. The payments service reported a 0% error rate for the entire window, and the checkout service logged no exceptions. The only signal was support tickets.
Users update their display name, see a success toast, and land on a profile page still showing the old name. Refreshing sometimes fixes it, sometimes not. It affects roughly 4% of updates and is completely unreproducible in staging.
After a 90-second network event between two racks, the inventory service reports negative stock for eleven SKUs. Two nodes both logged "elected leader" with no error in between. Every write in the window was acknowledged to a client.
A nightly ledger export produced a file with 340 duplicated rows and a truncated tail. The export is guarded by a distributed lock with a 10-second TTL and has run cleanly for two years. Both workers logged "acquired lock" — 11 seconds apart.
The recommendations service degraded from 80ms to 900ms after a cache node was replaced. Within four minutes it was returning nothing at all, and the two tiers in front of it were also down. Total outage: 26 minutes, of which the original cause lasted 90 seconds.
One shard of a 16-shard event store runs at 94% CPU while the other fifteen sit at 20%. Writes to that shard time out during peak. The team doubled the cluster to 32 shards over a weekend. The hot shard is still hot, and overall p99 got worse.
A 12-consumer group on a 48-partition topic stopped making progress. Lag grew to 4 million messages over two hours. CPU on every consumer was under 10%. Each consumer logged "partitions assigned" between 40 and 60 times per minute.
A finance reconciliation found 812 refunds marked issued in the orders database with no corresponding payment-provider refund. The oldest was 19 days old. Every service involved reported healthy for the entire period.
After a 40-minute transatlantic link degradation, 3,100 customer records show a shipping address from one region and a phone number from the other, in combinations that were never entered together. Four hundred records lost an edit entirely.
While debugging a checkout failure, the merged log timeline shows the payment service’s "charge authorised" line 700ms *before* the order service’s "calling payments" line. Several traces show child spans starting before their parents. Two engineers spent a day building a theory around a rogue background job.
A travel booking saga failed at the final step. The compensating transactions ran. Two hours later the customer had no hotel, no flight, a charge on their card, and a loyalty account with 40,000 points that were spent and then returned twice. The saga log says COMPENSATED_OK.
Customers report clicking search results that 404. Others report that a product they can reach by direct link never appears in search. An audit finds 2,300 documents in the search index with no matching row, and 1,700 rows with no document. The index has never been alerted on.
For 6 to 9 seconds at the top of every hour, the homepage p99 goes from 90ms to 11 seconds and roughly 2% of requests fail. Between those windows everything is perfect. The pattern started when the site got popular and has survived three rounds of database tuning.
An automated outreach agent sent 1,100 duplicate emails over a weekend, some recipients receiving the same message four times with different message ids. The agent’s run log shows 1,100 successful runs and no retries. The email provider shows 4,400 accepted sends.
A customer changed their shipping address at 09:14 and confirmed it on screen. The order that shipped at 11:00 went to the old address. The audit log contains both versions, with the *new* address recorded as having been superseded by the *old* one.
A feature-flag store configured N=3, W=2, R=2 served a flag value that had been overwritten 40 minutes earlier, to 8% of readers, for six hours. The team’s design document proves that R + W > N guarantees the read sees the write.
A cross-database transfer held row locks on two databases for 47 minutes. Every query touching those rows blocked. The two participants each reported the transaction as PREPARED. The coordinator process was not running.
A stream processor with exactly-once semantics enabled produced duplicate push notifications for 0.3% of events during a rolling restart. The processor’s own output topic contains exactly one record per input event, verified by audit.
The checkout service returned errors for 22 minutes. Every instance passed its health check throughout, so the load balancer kept sending traffic to all of them and no instance was ever replaced. The dashboard stayed green for the entire outage.
A 40-node storage cluster spent nine hours moving data between nodes without any node having failed. Network egress was saturated, client p99 tripled, and the cluster reported "rebalancing" continuously. Node uptime was 100% across the fleet.
During a traffic surge the search backend ran at 100% CPU for 40 minutes while the user-facing p99 was 3 seconds and rising. Analysis afterwards showed that 71% of the queries the backend executed had no client left to receive the answer.
The product page p99 is 1.9 seconds. It calls 30 backend services in parallel; each reports a p99 of 60ms and a p50 of 8ms. Three teams have each verified their own service is well within its SLO. The page has been slow for a year.
A tax-calculation vendor’s API went from 40ms to 9 seconds. Within three minutes the entire application was down — including the home page, the login flow and the help centre, none of which use tax calculation.
Hourly revenue aggregates are 2 to 6% below the transactional database, but only for the 00:00–01:00 window and only after weekends. Everything else matches to the cent. The streaming job reports no errors and no lag.
After a routine node drain, 14,000 order-confirmation emails were never sent. The consumer group’s lag was zero throughout — the offsets say every message was consumed. There is no gap in the log and no entry in the DLQ.
A price change from £40 to £30 was applied, the cache was invalidated, and within seconds roughly 15% of page views showed £40 again. Re-invalidating fixed it for a few seconds each time. It took eleven manual invalidations before it stuck.
A single-field change to the user service’s response was deployed at 14:05. By 14:07 five other services were failing. The change had been reviewed, tested against the user service’s test suite, and deployed with a canary that stayed green for its full ten minutes.
A migration renamed status to order_status on the orders table at 03:00. The orders service deployed cleanly. At 08:00 the fulfilment service began marking every order as unshippable, and the analytics warehouse silently loaded nulls for five days before anyone noticed.
A regional outage triggered automatic failover to the secondary region. Failover completed in 4 minutes and the service came back cleanly. Two days later reconciliation found 1,842 orders that had confirmation emails, charges on the card, and no record in the database.
A nightly aggregation job that used to take 12 minutes now takes 54, blowing its window and delaying every downstream report. The cluster is 90% idle for the last 45 minutes of every run. Adding workers changed nothing.
A single failed disk triggered automatic replacement of one node in a 24-node cluster. Within eight minutes read p99 across the *entire* cluster went from 15ms to 4 seconds, and stayed there for six hours while data was restreamed.
A support-automation system runs a pool of agents that claim tickets from a shared queue. Over one week, 62 tickets were worked by two agents simultaneously. Eleven of those produced duplicate refunds. The claim mechanism reported no conflicts.
A video-transcoding queue produced two output files for 4% of jobs, always the longest ones. Both workers logged a successful transcode. The queue’s delivery mode is "at-least-once" and the team believed duplicates would only occur on worker crashes, which had not happened.
A storage node lost its NVMe device. Recovery from the most recent checkpoint plus the write-ahead log was expected to take 4 minutes and lose nothing. It took 6 hours and lost 40 minutes of writes. The runbook had been tested quarterly and always passed.