Distributed Systems Interview

You move a function from inside your service into a new service and call it over HTTP. The code at the call site looks almost identical. What has actually changed about that call?

A team says "we are not distributed, we run a single application server". That server talks to Postgres, Redis, S3 and Stripe. Are they distributed?

Two servers in the same rack log an event each. Server A logs 10:00:00.120, server B logs 10:00:00.090. Can you conclude B's event happened first?

A team wants to split a module out of the monolith into its own service because "it will scale better and deploy independently". What do you want to know before agreeing?

Your system must never let a user hold more than one active subscription. Users live in the Accounts service, subscriptions in the Billing service. Where does that invariant live, and what does that decide?

A service keeps a per-user rate-limit counter in process memory. It works perfectly on one instance. What breaks when you run twelve instances behind a load balancer, and what are the honest options?

A batch endpoint accepts 500 records. Halfway through, the downstream store starts rejecting writes. What is the shape of the problem, and what must the API tell the caller?

Your health check marks a node unhealthy after it misses three 1-second heartbeats. The node is actually alive but stuck in a 6-second garbage-collection pause. What did your system just conclude, and was it entitled to?

Someone proposes adding checksums and signature verification between your own internal services "for correctness". What failure model are they assuming, and is it the right one here?

A database is configured with three replicas and the team reports "we can survive two failures". You discover all three run on hosts in the same rack, backed by the same storage array. What is your assessment?

Walk through a node that has just been partitioned from the rest of the cluster. Enumerate what it knows and what it merely believes, and explain why the distinction changes what it is allowed to do.

One node in a ten-node cluster is not down, but it responds to 4% of requests with 30-second latency. Dashboards are green, the node passes health checks, and overall error rate is 0.1%. Why is this harder than an outright crash, and how do you contain it?

Service A calls Service B and the call times out. Did B execute the request?

A job measures its own duration by taking the wall-clock time before and after. Occasionally it logs a negative duration. What happened, and what should it have used?

On a social feed, a user posts "I lost my keys" and then a reply "found them!". Some readers see the reply before the post. What ordering property is missing, and is a total order needed to fix it?

You have added Lamport timestamps to every event so you can order them. A colleague says this lets you detect concurrent updates. Are they right?

A lease-based leader holds a 10-second lease and serves reads locally without contacting peers, arguing that the lease guarantees it is still leader. Under what assumptions is that sound, and how would you strengthen it?

A team proposes putting every state change through a single totally-ordered log so that "all replicas stay identical". What have they actually built, what does it buy, and what does it cost?

Name the reasons to keep more than one copy of your data, and say which of them a read replica actually delivers.

Your database can acknowledge a write after the leader has it, or after one follower has confirmed it. Describe the trade you are making in each direction.

A team configures N=3, W=2, R=2 and concludes "R+W>N, so reads always see the latest write". What assumptions are hiding in that conclusion?

A user refreshes a page twice. The first refresh shows a comment; the second shows it gone; the third shows it again. Nothing was deleted. Explain what the load balancer has to do with it.

A user updates their profile, gets a 200, and immediately reloads the page — which shows the old value. Nothing is broken. Explain why this happens and how you would fix it.

To cut write latency you enable multi-leader replication across three regions. What class of problem have you accepted, and what must the application now provide that it did not before?

A vendor describes their datastore as "strongly consistent". What do you need them to say before that sentence means anything?

A database is serializable. A colleague concludes that a read issued after a completed write will definitely see it. Are they right?

Your product manager hears "eventually consistent" and asks whether that means the data might be wrong forever. What is the honest answer, and what should you actually promise them?

Explain the CAP theorem.

Take a ride-hailing app: driver location updates, ride assignment, surge pricing, ride history, and payment. Assign a consistency requirement to each and justify the differences.

Your system has had no partition in two years. A colleague argues CAP is therefore irrelevant to your design. Respond.

You must decide whether a store really is linearizable for single-key operations. You cannot read its source. How do you find out?

Two users edit the same record at nearly the same moment from different replicas. The system keeps the one with the later timestamp. What just happened to the other edit, and when is that acceptable?

How does a replica tell the difference between "this write supersedes what I have" and "this write is concurrent with what I have"?

A team wants to use CRDTs so that "conflicts resolve themselves". For which parts of a collaborative task app does that work, and where does it break down?

Your replicas have converged: every one holds byte-identical state. A customer reports that their withdrawal is missing. Both statements are true. How?

Why do consensus systems require a majority rather than, say, any two nodes out of five?

A Raft cluster elects a new leader while the old one is partitioned away. The old leader still has clients pointed at it. What stops it from causing damage, and what does not?

A three-node etcd cluster is re-electing a leader every few seconds. Writes intermittently fail. Where do you look, and what is the usual cause?

What problem does consensus solve?

For each of these, say whether consensus is required: assigning unique order ids; deciding which node runs a cron job; keeping a counter of page views; deciding whether a customer can spend from their balance; publishing a feature-flag change.

A correctly implemented consensus system elects exactly one leader per term. Explain why the storage system it writes to still needs to check something.

Under what conditions does a Raft cluster fail to make progress even though every node is running and no node has crashed?

An order must reserve inventory and charge a card. They live in two different services. Why can you not just wrap this in a transaction, and what do you do instead?

In two-phase commit, a participant votes yes and then loses contact with the coordinator. What is it allowed to do, and why is that the worst part of the protocol?

A saga compensates a failed booking by issuing a refund. A colleague calls this "rolling back". What is wrong with that description, and what does it change about the design?

Your saga has six steps across five services. Would you orchestrate it centrally or let each service react to events, and what would change your mind?

Sagas give up isolation as well as atomicity. Name the anomalies that creates and how you would contain them.

Which of these are idempotent, and which only look it: DELETE /orders/5; POST /orders; balance = 100; balance += 10; "send this email"?

You are adding idempotency keys to a payments API. Who generates the key, what is stored against it, how long is it kept, and what happens on a concurrent duplicate?

A broker offers at-most-once and at-least-once delivery. Why are those the only two on offer, and how do you choose?

A consumer deduplicates by keeping message ids in Redis with a 5-minute TTL. Under what circumstances does this fail, and what would you build instead?

A handler is described as idempotent. It writes a row, publishes an event, and increments a metric. Is the handler idempotent?

How do you guarantee exactly-once processing?

A synchronous HTTP call from the checkout service to the email service occasionally makes checkout slow and sometimes fails it. Someone proposes a queue. What does the queue actually change?

When would you use a work queue rather than publish/subscribe, and what goes wrong if you pick the wrong one?

A queue has a 30-second visibility timeout. Processing usually takes 5 seconds but occasionally 90. What is happening to those slow messages, and how do you fix it properly?

One message in a queue causes the consumer to crash on every attempt. Describe what happens without a dead-letter queue, and what a good DLQ policy looks like.

A team relies on message order to apply account updates. What order does their broker actually guarantee, and what would you check before trusting it?

Is Kafka just a message queue?

A dashboard shows "orders in the last hour" and the number is wrong after a network hiccup upstream. What are the two different clocks in play, and which one should the dashboard use?

Your stream job emits an hourly aggregate when the watermark passes the end of the hour. What is the watermark actually asserting, and what happens to data that arrives afterwards?

A Kafka consumer group is rebalancing every few minutes and lag is climbing. What causes a rebalance, and why does it make things worse rather than better?

Where exactly should a stream consumer commit its offset relative to doing its work, and what guarantee does each choice give?

A stateful stream job maintains a per-user running total. The job is restarted after a crash. What has to be true for the totals to be correct afterwards?

You are sharding a table of events by a timestamp key. Hash or range? Argue both sides.

A cache uses hash(key) % N to pick a node. What happens when you add the eleventh node to a ten-node cluster, and what would you use instead?

One shard out of 32 is at 90% CPU while the others sit at 15%. Walk through your diagnosis and your options.

After sharding by user id, a feature needs "all orders over £1000 across all users, sorted by date". What does that cost now, and what are the options?

You must move a partition from node A to node B in a live system without losing writes or serving stale reads. Describe the handover.

Why is "coordination is slow" an incomplete description of what coordination costs?

Give three concrete techniques for enforcing a business rule without coordinating on every operation, and say what each gives up.

What is the difference between a lock and a lease, and why does a distributed system almost always want the second?

Your services use ZooKeeper for leader election, configuration and service discovery. ZooKeeper becomes unavailable for ten minutes. What should happen, and what usually does?

You need globally unique identifiers across many nodes. Compare the options, and say which requires coordination.

Why can a distributed lock still be unsafe, even when the lock service is correct?

A service instance is terminated. For how long can clients still try to send it traffic, and what should happen to those requests?

A 500-node cluster tracks membership by gossip rather than a central registry. What did that buy, and what did it cost?

Two replicas have been partitioned for an hour and hold millions of keys. How do you find and repair the differences without transferring everything?

Design the failure detector for a cluster where evicting a healthy node is expensive and keeping a dead one is also expensive. What do you build?

A service adds automatic retries — three attempts on any failure — and reliability gets worse during the next incident. Explain.

Demand exceeds capacity. Your service can queue the excess or reject it. Which is better, and how do you decide the threshold?

A fast producer feeds a slow consumer through a queue. The queue depth grows for hours. What is missing, and where should the signal go?

A single dependency slows from 20ms to 900ms. Twenty minutes later the whole platform is down, including services that never call it. Trace the mechanism.

A system was healthy at 60% load. A brief spike pushed it to 100%. The spike ended ten minutes ago, load is back to 60%, and the system is still down. Why does removing the cause not fix it?

What is wrong with a 30-second default timeout on an internal service call, and how would you choose a better one?

A user request has a 2-second budget and passes through four services, each with its own 5-second timeout. What goes wrong, and what should the design be?

A request fans out to 50 shards and waits for all of them. Each shard has a p99 of 100ms and a median of 10ms. What is the request latency, and what do you do about it?

A client cancels a request. What can the server actually guarantee about the work, and what should the API contract say?

Adding a cache is usually described as a performance change. What consistency question does it introduce, and who has to answer it?

A popular key expires. Within 50ms, 4,000 requests miss and all query the database. Describe the fixes and their trade-offs.

You must invalidate a cached value across 200 application instances, each holding a local in-process copy. How, and what remains broken?

Your cache hit rate is 97% and the cache cluster fails. Is the system fine on the remaining 3%, and what should have been designed in?

A write returns success. What has actually been guaranteed, and what questions would you ask to find out?

A team uses an object store as a database: they list a prefix, read the objects, and write updates back. What properties are they assuming that an object store may not provide?

Explain why a system that keeps a durable log still needs checkpoints, and what determines how often to take one.

You need a backup of a system whose state is spread across twelve shards. Taking a snapshot of each at slightly different moments — what could go wrong?

Why do distributed compute frameworks try to run the computation where the data already is, rather than fetching the data to the computation?

In a map-reduce style job, why is the shuffle usually the expensive part?

A job with 10,000 tasks completes 9,997 in four minutes and the last three take an hour. What causes that, and what does speculative execution actually fix?

A cluster shows 60% CPU utilisation but jobs are queued and cannot start. What is going on, and what would you change?

A stakeholder asks why a write from Sydney to a database in Frankfurt cannot be made to feel instant with better engineering. What do you tell them?

Your DR plan is an active-passive standby in a second region with asynchronous replication. What are you actually promising, and what will go wrong on the day?

EU customer data must not leave the EU. How does that constraint propagate through a system that was designed as a single global deployment?

You run active-active in three regions. A partial failure in one region — not a clean outage, but 30% error rates — begins affecting the other two. How is that possible?

You need low-latency writes in both Europe and the US, and a strict global uniqueness constraint on a user-chosen handle. What trade-off appears, and how do you resolve it?

A product page shows inventory, reviews, recommendations and price. The reviews service is down. What should the page do, and what decision had to be made in advance?

A user reports that one request failed at 14:32. It touched eleven services. How do you find out what happened, and what has to have been built beforehand?

Before running a chaos experiment, you are asked to write down the steady-state hypothesis. What is it, and why does the experiment mean nothing without it?

Walk through how you would structure the response to a distributed incident, and say why containment should usually come before diagnosis.

Make the case for and against injecting faults in production, and say what must be true before you do it.

What are the signs that a set of microservices is actually a distributed monolith, and why is that the worst of both worlds?

Two services both need customer address data. Describe the options for giving them both access and the consequences of each.

A service maintains a local read model built from another service's events. What are the operational obligations that come with it?

You are cutting a system into services. What are the criteria for a good boundary, and how would you test a proposed one before committing?

Three systems each hold a customer's subscription status and they disagree. Design the resolution — both the immediate one and the structural one.

An LLM agent calls a tool that performs a write, and the call times out. Why is this the same problem as any other remote call, and what is different?

A long-running agent workflow crashes at step 7 of 12. What has to have been recorded for it to resume correctly, and what makes this harder than resuming a conventional job?

Three agents work on the same task in parallel and each can write to shared state. What distributed-systems problems have you just recreated, and how do you contain them?

What failure modes does an agentic system have that a conventional distributed system does not, and how would you detect them?

You are given a greenfield system with a two-year horizon and a team of eight. What would you deliberately not distribute, and how would you defend that to a stakeholder who wants microservices?