Distributed Systems Practice

You move a function from inside your service into a new service and call it over HTTP. The code at the call site looks almost identical. What has actually changed about that call?

The failure · After the split, a customer is charged twice on a day when the network between the two services is briefly congested. Nothing in the code changed except that the call became remote.

A team says "we are not distributed, we run a single application server". That server talks to Postgres, Redis, S3 and Stripe. Are they distributed?

The failure · The handler writes a row to Postgres, then uploads the file to S3. A deploy restarts the process between the two, and the system now has a row pointing at an object that does not exist.

Two servers in the same rack log an event each. Server A logs 10:00:00.120, server B logs 10:00:00.090. Can you conclude B's event happened first?

A team wants to split a module out of the monolith into its own service because "it will scale better and deploy independently". What do you want to know before agreeing?

The failure · Six months after the split, both services must be deployed together in a fixed order, and an outage in the new service takes down checkout — which it never did as a module.

Your system must never let a user hold more than one active subscription. Users live in the Accounts service, subscriptions in the Billing service. Where does that invariant live, and what does that decide?

The failure · A user double-clicks Subscribe. Two requests hit two Billing instances 4ms apart, both read "no active subscription", and the user is billed twice.

A service keeps a per-user rate-limit counter in process memory. It works perfectly on one instance. What breaks when you run twelve instances behind a load balancer, and what are the honest options?

The failure · An autoscaler doubles the fleet during a traffic spike; the rate limit silently doubles with it, and the upstream API the limiter was protecting starts returning 429s to everyone.

A batch endpoint accepts 500 records. Halfway through, the downstream store starts rejecting writes. What is the shape of the problem, and what must the API tell the caller?

The failure · The caller receives a 500 after 312 records were written, retries the whole batch, and the report now shows 812 records.

Your health check marks a node unhealthy after it misses three 1-second heartbeats. The node is actually alive but stuck in a 6-second garbage-collection pause. What did your system just conclude, and was it entitled to?

The failure · The failover promotes a replica. Four seconds later the original node resumes, completes its in-flight write, and two nodes have written different values for the same key.

Someone proposes adding checksums and signature verification between your own internal services "for correctness". What failure model are they assuming, and is it the right one here?

A database is configured with three replicas and the team reports "we can survive two failures". You discover all three run on hosts in the same rack, backed by the same storage array. What is your assessment?

The failure · The rack loses power. All three replicas go down together; the "two failure tolerance" turns out to be zero.

Walk through a node that has just been partitioned from the rest of the cluster. Enumerate what it knows and what it merely believes, and explain why the distinction changes what it is allowed to do.

The failure · The old leader keeps serving reads for 30 seconds after a new leader was elected. Users see a value that was overwritten half a minute ago and no error is logged anywhere.

One node in a ten-node cluster is not down, but it responds to 4% of requests with 30-second latency. Dashboards are green, the node passes health checks, and overall error rate is 0.1%. Why is this harder than an outright crash, and how do you contain it?

The failure · Callers with a 30-second client timeout exhaust their connection pools waiting on the one bad node; the outage is reported as "the whole API is down" while nine of ten nodes are perfectly healthy.

Service A calls Service B and the call times out. Did B execute the request?

The failure · Your service calls a payment provider and times out. The provider's dashboard later shows the charge succeeded. Your retry created a second charge, and the reconciliation job that would have caught it runs nightly.

A job measures its own duration by taking the wall-clock time before and after. Occasionally it logs a negative duration. What happened, and what should it have used?

On a social feed, a user posts "I lost my keys" and then a reply "found them!". Some readers see the reply before the post. What ordering property is missing, and is a total order needed to fix it?

The failure · A moderator deletes a comment and then posts an explanation. A replica applies the explanation first, so users briefly see an explanation for a deletion that has not happened.

You have added Lamport timestamps to every event so you can order them. A colleague says this lets you detect concurrent updates. Are they right?

A lease-based leader holds a 10-second lease and serves reads locally without contacting peers, arguing that the lease guarantees it is still leader. Under what assumptions is that sound, and how would you strengthen it?

The failure · The leader host is live-migrated and pauses for 12 seconds. It resumes mid-read, serves a value from before a completed failover, and the client uses it to make a billing decision.

A team proposes putting every state change through a single totally-ordered log so that "all replicas stay identical". What have they actually built, what does it buy, and what does it cost?

The failure · One slow follower drags commit latency to 400ms for every write in the product, including a like button, because the log is one stream and the quorum waits for the slowest of the majority.

Name the reasons to keep more than one copy of your data, and say which of them a read replica actually delivers.

Your database can acknowledge a write after the leader has it, or after one follower has confirmed it. Describe the trade you are making in each direction.

The failure · A leader dies with 900ms of unshipped writes. The new leader is promoted, and 40 orders that returned 201 Created to customers no longer exist.

A team configures N=3, W=2, R=2 and concludes "R+W>N, so reads always see the latest write". What assumptions are hiding in that conclusion?

A user refreshes a page twice. The first refresh shows a comment; the second shows it gone; the third shows it again. Nothing was deleted. Explain what the load balancer has to do with it.

The failure · Support cannot reproduce the bug because their requests happen to land on the current replica; the reporter sees it every few minutes.

A user updates their profile, gets a 200, and immediately reloads the page — which shows the old value. Nothing is broken. Explain why this happens and how you would fix it.

The failure · After a deploy, replica lag rises to 4 seconds. Support tickets spike with "my changes did not save", and every one of those changes did in fact save.

To cut write latency you enable multi-leader replication across three regions. What class of problem have you accepted, and what must the application now provide that it did not before?

The failure · A customer edits their shipping address from a plane, hitting the EU region, while support edits it from the US region. Both writes succeed; LWW keeps the one with the later clock reading, which is the support edit made 200ms earlier.

A vendor describes their datastore as "strongly consistent". What do you need them to say before that sentence means anything?

A database is serializable. A colleague concludes that a read issued after a completed write will definitely see it. Are they right?

Your product manager hears "eventually consistent" and asks whether that means the data might be wrong forever. What is the honest answer, and what should you actually promise them?

The failure · A replica stops receiving updates because of a stuck replication slot. Nothing alerts, because reads still succeed. Divergence is discovered three weeks later by a customer.

Explain the CAP theorem.

The failure · A link between two data centres degrades. The CP store on one side stops accepting writes and the on-call engineer, seeing a healthy cluster with no errors in the logs, spends 40 minutes looking for a bug that is not there.

Take a ride-hailing app: driver location updates, ride assignment, surge pricing, ride history, and payment. Assign a consistency requirement to each and justify the differences.

Your system has had no partition in two years. A colleague argues CAP is therefore irrelevant to your design. Respond.

The failure · A routine security-group change blocks traffic between two subnets for 90 seconds. The "never partitioned" cluster loses quorum, writes fail, and nobody has ever seen the resulting error path.

You must decide whether a store really is linearizable for single-key operations. You cannot read its source. How do you find out?

Two users edit the same record at nearly the same moment from different replicas. The system keeps the one with the later timestamp. What just happened to the other edit, and when is that acceptable?

The failure · A user adds an item on their phone while another item is added from the desktop. The cart shows one item; the customer checks out without noticing the other is gone.

How does a replica tell the difference between "this write supersedes what I have" and "this write is concurrent with what I have"?

A team wants to use CRDTs so that "conflicts resolve themselves". For which parts of a collaborative task app does that work, and where does it break down?

The failure · Two offline clients each add a member to a 4-member project. On sync, the project has 6 members, and the business rule has been violated by a system that never made an incorrect decision.

Your replicas have converged: every one holds byte-identical state. A customer reports that their withdrawal is missing. Both statements are true. How?

The failure · A deposit and a withdrawal are processed concurrently on two replicas, each writing an absolute new balance. LWW keeps one. All replicas agree on a balance that reflects only one of the two transactions.

Why do consensus systems require a majority rather than, say, any two nodes out of five?

A Raft cluster elects a new leader while the old one is partitioned away. The old leader still has clients pointed at it. What stops it from causing damage, and what does not?

The failure · A dashboard served by the partitioned leader shows a two-minute-old inventory count. A human makes a purchasing decision on it. No error appeared anywhere.

A three-node etcd cluster is re-electing a leader every few seconds. Writes intermittently fail. Where do you look, and what is the usual cause?

The failure · The consensus store shares a disk with a log-shipping process. During log rotation, fsync latency hits 900ms, the leader misses heartbeats, and the whole control plane becomes unavailable for 15 seconds every hour.

What problem does consensus solve?

The failure · A team routes every application write through a Raft group for safety. Write latency triples, throughput caps at one ordered stream, and an unrelated slow follower makes the product feel broken.

For each of these, say whether consensus is required: assigning unique order ids; deciding which node runs a cron job; keeping a counter of page views; deciding whether a customer can spend from their balance; publishing a feature-flag change.

A correctly implemented consensus system elects exactly one leader per term. Explain why the storage system it writes to still needs to check something.

The failure · A checkpointing job holds a lock, pauses for 25 seconds, and writes its checkpoint over the one written by its successor. The corruption is discovered on the next restart, hours later.

Under what conditions does a Raft cluster fail to make progress even though every node is running and no node has crashed?

The failure · A misconfigured firewall rule allows outbound but blocks inbound on one node. The cluster elects and re-elects for 40 minutes; every node reports itself healthy and every dashboard is green.

An order must reserve inventory and charge a card. They live in two different services. Why can you not just wrap this in a transaction, and what do you do instead?

The failure · The orchestrator crashes after reserving inventory and before charging. Nothing releases the reservation; the item shows as out of stock for a customer who was never charged.

In two-phase commit, a participant votes yes and then loses contact with the coordinator. What is it allowed to do, and why is that the worst part of the protocol?

The failure · A coordinator host is terminated by an autoscaler. Three participant databases hold row locks for 40 minutes; the incident presents as unrelated queries timing out across the whole product.

A saga compensates a failed booking by issuing a refund. A colleague calls this "rolling back". What is wrong with that description, and what does it change about the design?

The failure · The confirmation email is sent at step 2 of 5. Step 4 fails, the booking is compensated, and the customer has a confirmation email for a trip that does not exist and a refund arriving in five business days.

Your saga has six steps across five services. Would you orchestrate it centrally or let each service react to events, and what would change your mind?

The failure · With choreography, a service is deployed with a changed event schema. Three downstream handlers silently stop matching, orders stall at step 4, and no error is raised anywhere because nothing failed — things simply did not happen.

Sagas give up isolation as well as atomicity. Name the anomalies that creates and how you would contain them.

The failure · A refund saga sets the order status to REFUNDING at step 1. A concurrent support action sets it to CLOSED. The saga completes and writes REFUNDED, and the closed order reopens in the support queue with no explanation.

Which of these are idempotent, and which only look it: DELETE /orders/5; POST /orders; balance = 100; balance += 10; "send this email"?

You are adding idempotency keys to a payments API. Who generates the key, what is stored against it, how long is it kept, and what happens on a concurrent duplicate?

The failure · A mobile client retries on a flaky connection and both requests reach different API instances 12ms apart. Both check for the key, both find nothing, both charge.

A broker offers at-most-once and at-least-once delivery. Why are those the only two on offer, and how do you choose?

A consumer deduplicates by keeping message ids in Redis with a 5-minute TTL. Under what circumstances does this fail, and what would you build instead?

The failure · An incident is resolved by replaying two hours of messages from a DLQ. Every one of them is outside the dedupe window; 40,000 emails are sent a second time.

A handler is described as idempotent. It writes a row, publishes an event, and increments a metric. Is the handler idempotent?

The failure · A retried handler writes no duplicate row (the constraint holds) but publishes a second OrderPlaced event. A downstream service ships a second package.

How do you guarantee exactly-once processing?

The failure · A team enables Kafka's exactly-once configuration and removes the deduplication from their consumer. The consumer writes to an external CRM; after the next rebalance, the CRM has duplicate records that the broker's guarantee never covered.

A synchronous HTTP call from the checkout service to the email service occasionally makes checkout slow and sometimes fails it. Someone proposes a queue. What does the queue actually change?

When would you use a work queue rather than publish/subscribe, and what goes wrong if you pick the wrong one?

A queue has a 30-second visibility timeout. Processing usually takes 5 seconds but occasionally 90. What is happening to those slow messages, and how do you fix it properly?

The failure · A nightly report job runs for 4 minutes under a 60-second visibility timeout. Four workers produce four copies of the report and the finance team receives four conflicting emails.

One message in a queue causes the consumer to crash on every attempt. Describe what happens without a dead-letter queue, and what a good DLQ policy looks like.

The failure · A single message with an unparseable date crashes the consumer. The queue backs up for six hours; the on-call engineer sees a crash-looping pod and rolls back a deploy that was not the cause.

A team relies on message order to apply account updates. What order does their broker actually guarantee, and what would you check before trusting it?

The failure · An account is suspended and then reinstated. The two messages hash to different partitions after a partition-count change; the consumer applies them in the wrong order and the account stays suspended.

Is Kafka just a message queue?

The failure · A team migrates a task queue to Kafka for "scale". One poison message blocks its partition; a fifth of all tasks stop being processed while the other four fifths look perfectly healthy.

A dashboard shows "orders in the last hour" and the number is wrong after a network hiccup upstream. What are the two different clocks in play, and which one should the dashboard use?

The failure · A mobile client buffers events offline and uploads a day later. The processing-time dashboard shows a 40% sales spike on a random Tuesday.

Your stream job emits an hourly aggregate when the watermark passes the end of the hour. What is the watermark actually asserting, and what happens to data that arrives afterwards?

A Kafka consumer group is rebalancing every few minutes and lag is climbing. What causes a rebalance, and why does it make things worse rather than better?

The failure · A deploy rolls 12 consumer pods one at a time. Each restart triggers a full group rebalance; the group spends more time rebalancing than processing, and lag reaches four hours during a routine deploy.

Where exactly should a stream consumer commit its offset relative to doing its work, and what guarantee does each choice give?

The failure · Auto-commit fires while a batch of 500 records is half processed. The pod is evicted; 240 records are never processed and never reported as failed.

A stateful stream job maintains a per-user running total. The job is restarted after a crash. What has to be true for the totals to be correct afterwards?

The failure · A job restores state from a 10-minute-old checkpoint but the consumer group offset was committed 30 seconds ago. Nine and a half minutes of transactions vanish from every user total, and the numbers are simply wrong from then on.

You are sharding a table of events by a timestamp key. Hash or range? Argue both sides.

A cache uses hash(key) % N to pick a node. What happens when you add the eleventh node to a ten-node cluster, and what would you use instead?

The failure · A cache node is added at 9am. Ninety percent of keys miss simultaneously, the database saturates, and the "capacity increase" causes the outage it was meant to prevent.

One shard out of 32 is at 90% CPU while the others sit at 15%. Walk through your diagnosis and your options.

The failure · A viral post makes one row the target of 200,000 reads per second. The shard holding it saturates and every unrelated user on that shard sees timeouts.

After sharding by user id, a feature needs "all orders over £1000 across all users, sorted by date". What does that cost now, and what are the options?

You must move a partition from node A to node B in a live system without losing writes or serving stale reads. Describe the handover.

The failure · A rebalance updates the routing table before the drain completes. Writes routed to B for 400ms coexist with in-flight writes at A; the partition ends up with two divergent histories and no way to decide which is right.

Why is "coordination is slow" an incomplete description of what coordination costs?

Give three concrete techniques for enforcing a business rule without coordinating on every operation, and say what each gives up.

What is the difference between a lock and a lease, and why does a distributed system almost always want the second?

Your services use ZooKeeper for leader election, configuration and service discovery. ZooKeeper becomes unavailable for ten minutes. What should happen, and what usually does?

The failure · A ZooKeeper quorum loss lasts 8 minutes. Every service that consults it per request returns 503, and the incident is reported as a total platform outage caused by a component holding no user data.

You need globally unique identifiers across many nodes. Compare the options, and say which requires coordination.

Why can a distributed lock still be unsafe, even when the lock service is correct?

The failure · A nightly job takes a 30-second lock to rebuild an index. Its JVM pauses for 45 seconds. A second instance starts, rebuilds, and finishes; the first wakes and writes its half-built index over the good one.

A service instance is terminated. For how long can clients still try to send it traffic, and what should happen to those requests?

A 500-node cluster tracks membership by gossip rather than a central registry. What did that buy, and what did it cost?

Two replicas have been partitioned for an hour and hold millions of keys. How do you find and repair the differences without transferring everything?

The failure · A replica silently stops receiving writes for three weeks because of a stuck replication stream. Reads succeed, no alarm fires, and the divergence is only found when anti-entropy is finally run.

Design the failure detector for a cluster where evicting a healthy node is expensive and keeping a dead one is also expensive. What do you build?

The failure · A brief network blip makes every node suspect every other. The cluster evicts itself, triggers mass re-replication, saturates the network, and the re-replication traffic keeps the nodes unreachable.

A service adds automatic retries — three attempts on any failure — and reliability gets worse during the next incident. Explain.

The failure · A database slows down by 3x. Retries at the ORM, the service client, the API gateway and the mobile app turn a slowdown into a 30x load increase, and the database never recovers until traffic is manually shed.

Demand exceeds capacity. Your service can queue the excess or reject it. Which is better, and how do you decide the threshold?

The failure · An unbounded queue absorbs a 5-minute traffic spike. Requests are served 90 seconds later, by which time every client has timed out and retried; the service spends the next hour serving responses nobody is waiting for.

A fast producer feeds a slow consumer through a queue. The queue depth grows for hours. What is missing, and where should the signal go?

A single dependency slows from 20ms to 900ms. Twenty minutes later the whole platform is down, including services that never call it. Trace the mechanism.

The failure · A recommendations service — explicitly optional, degradable, non-critical — slows down. Because the product page waits on it synchronously with a 3-second timeout and no concurrency cap, it takes down checkout.

A system was healthy at 60% load. A brief spike pushed it to 100%. The spike ended ten minutes ago, load is back to 60%, and the system is still down. Why does removing the cause not fix it?

The failure · A 90-second database failover empties an application cache. On recovery, every request misses, the database cannot serve the miss rate, requests time out and retry, and the outage continues for two hours at normal traffic levels.

What is wrong with a 30-second default timeout on an internal service call, and how would you choose a better one?

A user request has a 2-second budget and passes through four services, each with its own 5-second timeout. What goes wrong, and what should the design be?

The failure · During a slowdown, 60% of the compute fleet is busy computing responses for requests whose clients disconnected minutes ago. Adding capacity makes no difference because the new capacity does the same wasted work.

A request fans out to 50 shards and waits for all of them. Each shard has a p99 of 100ms and a median of 10ms. What is the request latency, and what do you do about it?

A client cancels a request. What can the server actually guarantee about the work, and what should the API contract say?

The failure · A user cancels an export. The UI says cancelled; the job completes 40 seconds later and emails them a 2GB file containing data they were mid-way through deleting.

Adding a cache is usually described as a performance change. What consistency question does it introduce, and who has to answer it?

A popular key expires. Within 50ms, 4,000 requests miss and all query the database. Describe the fixes and their trade-offs.

The failure · A cache node restarts during peak. Every key it held misses simultaneously; the database saturates and the outage lasts far longer than the cache restart did.

You must invalidate a cached value across 200 application instances, each holding a local in-process copy. How, and what remains broken?

The failure · A price change is broadcast while two instances are mid-deploy. They come up from a snapshot with the old price and serve it for the next four hours because the TTL is set to a day.

Your cache hit rate is 97% and the cache cluster fails. Is the system fine on the remaining 3%, and what should have been designed in?

The failure · A Redis failover takes 20 seconds. In those 20 seconds the database receives more queries than in the previous hour, saturates, and stays saturated for 90 minutes after Redis comes back.

A write returns success. What has actually been guaranteed, and what questions would you ask to find out?

A team uses an object store as a database: they list a prefix, read the objects, and write updates back. What properties are they assuming that an object store may not provide?

The failure · Two workers update the same manifest object 300ms apart. Both read the old version, both write; one worker's entire batch of additions vanishes with no error anywhere.

Explain why a system that keeps a durable log still needs checkpoints, and what determines how often to take one.

You need a backup of a system whose state is spread across twelve shards. Taking a snapshot of each at slightly different moments — what could go wrong?

The failure · A restore from twelve independent snapshots brings the system back with 340 orders that have payments and no inventory reservation, and 12 with reservations and no payment.

Why do distributed compute frameworks try to run the computation where the data already is, rather than fetching the data to the computation?

In a map-reduce style job, why is the shuffle usually the expensive part?

A job with 10,000 tasks completes 9,997 in four minutes and the last three take an hour. What causes that, and what does speculative execution actually fix?

The failure · Speculative execution is enabled for a job whose tasks write files directly to a shared path. Two attempts of the same task write simultaneously and the output file is a mix of both.

A cluster shows 60% CPU utilisation but jobs are queued and cannot start. What is going on, and what would you change?

A stakeholder asks why a write from Sydney to a database in Frankfurt cannot be made to feel instant with better engineering. What do you tell them?

Your DR plan is an active-passive standby in a second region with asynchronous replication. What are you actually promising, and what will go wrong on the day?

The failure · A region fails at 03:00. Failover is declared at 03:40 after debate, promotion takes 6 minutes, and the standby — sized at 40% of production to save money — collapses under the redirected load.

EU customer data must not leave the EU. How does that constraint propagate through a system that was designed as a single global deployment?

You run active-active in three regions. A partial failure in one region — not a clean outage, but 30% error rates — begins affecting the other two. How is that possible?

The failure · A bad configuration push reaches all three regions through the shared control plane in 90 seconds. Active-active did not help, because the failure travelled the same path the coordination did.

You need low-latency writes in both Europe and the US, and a strict global uniqueness constraint on a user-chosen handle. What trade-off appears, and how do you resolve it?

The failure · A multi-leader deployment lets both regions accept handle claims. Two users are both told they own @alex; the conflict is resolved by timestamp hours later, and one of them has already printed business cards.

A product page shows inventory, reviews, recommendations and price. The reviews service is down. What should the page do, and what decision had to be made in advance?

A user reports that one request failed at 14:32. It touched eleven services. How do you find out what happened, and what has to have been built beforehand?

Before running a chaos experiment, you are asked to write down the steady-state hypothesis. What is it, and why does the experiment mean nothing without it?

Walk through how you would structure the response to a distributed incident, and say why containment should usually come before diagnosis.

The failure · An incident is resolved in 20 minutes. Nobody reconciles; 1,800 duplicate charges made during the retry storm are discovered by customers over the following week.

Make the case for and against injecting faults in production, and say what must be true before you do it.

What are the signs that a set of microservices is actually a distributed monolith, and why is that the worst of both worlds?

Two services both need customer address data. Describe the options for giving them both access and the consequences of each.

A service maintains a local read model built from another service's events. What are the operational obligations that come with it?

The failure · A projection silently stops after an unhandled event type is introduced. Reads keep succeeding against a read model frozen at last Tuesday, and nothing alerts because there are no errors — only an absence of updates.

You are cutting a system into services. What are the criteria for a good boundary, and how would you test a proposed one before committing?

Three systems each hold a customer's subscription status and they disagree. Design the resolution — both the immediate one and the structural one.

The failure · A customer cancels. Billing stops charging, the entitlement service still grants access, and the CRM shows active. Support, reading the CRM, tells the customer they are still subscribed; the customer is not being charged and keeps full access for four months.

An LLM agent calls a tool that performs a write, and the call times out. Why is this the same problem as any other remote call, and what is different?

The failure · An agent times out calling a refund tool, decides the refund did not go through, calls it again with a slightly reworded reason, and the customer is refunded twice.

A long-running agent workflow crashes at step 7 of 12. What has to have been recorded for it to resume correctly, and what makes this harder than resuming a conventional job?

The failure · A workflow crashes after calling a provisioning API but before recording it. On resume, the agent provisions a second environment, and the first one is orphaned and billed for months.

Three agents work on the same task in parallel and each can write to shared state. What distributed-systems problems have you just recreated, and how do you contain them?

The failure · Two agents editing the same configuration file each read it, make their change, and write it back. The second write lands last and silently discards the first agent's change; both report success.

What failure modes does an agentic system have that a conventional distributed system does not, and how would you detect them?

The failure · An agent reports "cleaned up 14 stale resources". The audit log shows 14 delete calls, 11 of which returned errors the agent summarised as successes. Three resources were deleted; eleven remain and are believed gone.

You are given a greenfield system with a two-year horizon and a team of eight. What would you deliberately not distribute, and how would you defend that to a stakeholder who wants microservices?