Practice: Debug & Design

Support tickets: customers see two identical card charges for one order, a few seconds apart.

The payment provider became slow (p99 from 180 ms to ~8 s). Four minutes later checkout availability was 2%.

Database CPU hits 100% from 00:00:00 to about 00:01:30 every night; product-page p99 goes from 40 ms to 6 s.

Users report being logged out at random, often mid-checkout; shopping carts vanish and reappear.

Queue depth is 2.4 M and the oldest message is 7.4 hours old; both numbers have risen every day since Monday.

Finance reconciliation found 41 orders in state `SHIPPED` whose payment record is `failed` (card declined). The goods left the warehouse; nobody was charged.

The order search index shows orders with a status but no items, and some orders are missing entirely.

Gateway p99 has gone from 90 ms to 210 ms over six months; adding instances (6 → 12) changed nothing.

recommendations-svc p99 went from 180 ms to 45 s (its vector store was compacting).

After cancelling an order the order page still shows `CONFIRMED` for one to three seconds; a refresh later shows `CANCELLED`.

Callers of inventory-svc see ~33% `ECONNREFUSED` for about five minutes, then it clears — several times every evening.

cache-c runs at 95% CPU and serves ~45% of all cache reads; cache-a, -b and -d sit at 10–25%.

`GET /orders/:id` p50 is 210 ms and p99 600 ms; the order team’s explanation is "the database is slow".

Releases are a four-hour coordinated event every second Tuesday in which all nine services deploy together; the last one that shipped a single service alone (November) broke pricing.

A service that turns a long URL into a 7-character code and redirects anyone who opens it. Small feature, complete system: it forces the ID-generation decision, a read-heavy cache, the 301-vs-302 analytics tradeoff and a data set that outgrows one database.

One-to-one and group messaging with delivery and read receipts, presence, and push notifications when the recipient is offline. The core difficulty is not storage, it is routing: a stateful WebSocket connection lives on one server, and the recipient is on another.

Users follow other users and see a timeline of their posts. The naive query is a join across 200 followees at 70k reads/s; the fix is to precompute timelines — until one account has 20M followers, which breaks the precomputation. The design is the hybrid that survives both.

Catalogue, search, cart, checkout, inventory, payments and order fulfilment. The interesting parts are the ones where money and stock meet retries: reserving the last unit, charging exactly once through a provider that times out, and keeping an order’s state honest while five downstream systems react to it.

One service that every other service calls to reach a user by email, SMS or push — with preferences, templates, rate limits, retries and delivery status. It is the cleanest example of queue + workers + external providers, and of why "just call the email API" stops working.

Upload, store, share and download files of any size — a Dropbox-style service. The design separates two very different kinds of data (small hot metadata, huge cold bytes), moves bytes off the API servers with presigned URLs, and handles the realities of 2 GB uploads on flaky networks, duplicate content and malware.