Learn Distributed Systems

Not the number of machines, and not the word "microservices". A system is distributed the moment its correctness depends on a component that can fail independently and whose state you can only learn about through messages that may be lost, delayed or reordered.

Q · When does my system stop being one program and start being a distributed system?

The syntax is identical and the semantics are not. A local call either returns or throws; a remote call has a third outcome — no answer — and that outcome carries no information about whether the work was done. Everything else follows from that.

Q · My RPC client makes a remote call look exactly like a local one. What is that hiding?

A network can lose a message, delay it arbitrarily, deliver it out of order, deliver it twice, or partition the cluster into groups that each think the other side is gone. Those five behaviours are not edge cases to handle later — they are the design input.

Q · What exactly does the network do to my messages, and which of it can I stop worrying about?

Two threads can read the same bytes. Two nodes never can. Each holds a copy whose age is the time since the last message, which means "the current value" is not a thing either node can observe — only a thing they can agree to pretend about, at a price.

Q · Why can I not just read the other service’s state the way I read a variable?

Two machines cannot agree on what time it is closely enough to order events by timestamp. Which means a comparison of two timestamps from two hosts is not an ordering — it is a guess, and it is wrong in exactly the cases you built it to handle.

Q · Why can I not just compare timestamps to work out which event happened first?

There are exactly four reasons that survive scrutiny: the work does not fit on one machine, one machine failing is unacceptable, users are far away, or components must be isolated from each other. Everything else on the usual list is one of these four wearing a costume.

Q · What problem am I actually solving by putting this on more than one machine?

Most systems that adopt distribution do not need it. The costs — partial failure, ambiguity, operational surface, debugging across boundaries — are paid on day one and every day after. The benefits are conditional on assumptions that frequently turn out to be false.

Q · What is the honest case for keeping this on one machine?

A boundary is not a line on a diagram. It is the place where shared memory ends, where partial failure enters, and where an invariant stops being enforceable by the compiler. Choose it by asking which state must never disagree — not by which nouns look separate.

Q · If I am going to split this, where exactly should the cut go?

Every coordination decision in a distributed system is downstream of a single question: what must never be false? Teams that cannot state their invariants precisely end up coordinating everywhere, which is slow, or nowhere, which is wrong.

Q · How do I decide which parts of my system actually need agreement?

A single machine fails all at once — the process is running or it is not. A distributed system fails in pieces, at different times, and each surviving piece has a different and incomplete picture of which pieces those were. Every other idea in this domain is a response to that.

Q · Why is "some of it is broken" so much harder to handle than "all of it is broken"?

Service A calls Service B and the call times out. The single most common mistake in distributed systems is treating that as "it failed". It is not a failure result — it is the absence of a result, and five different realities produce it.

Q · My request to another service timed out. Did the work happen?

Before you can say a protocol is correct you have to say what it is correct against. Crash-stop, crash-recovery, omission, timing and Byzantine are the standard ladder — each admits more behaviours, and each costs more to tolerate. Choosing one is a design decision that most teams make implicitly.

Q · What kinds of failure is my design actually built to survive?

A failure detector converts silence into a suspicion. It is worth being precise about how weak that conversion is: five different situations produce the same silence, and one of them is that the observer, not the target, is the isolated one.

Q · My monitoring says the node stopped sending heartbeats. What have I actually learned?

This is the hardest distinction in practice and, in an asynchronous network, it is formally undecidable. Not difficult — undecidable. Every design that acts on "the node is dead" is acting on a guess, and the useful question is what happens when the guess is wrong.

Q · How do I tell whether that node has crashed or is merely slow?

A Byzantine fault is a component that does something arbitrary rather than simply stopping: answering differently to different peers, returning well-formed nonsense, or behaving correctly until it matters. Tolerating it costs 3f+1 nodes, signatures and an extra round — which is why almost nobody does, and why that choice deserves to be explicit.

Q · What if a node does not just fail, but lies — and should my system care?

A fault domain is the set of things that go away at the same time for the same reason. Process, machine, rack, zone, region — and also the ones that are not on the diagram: a shared config store, a certificate authority, a control plane, a deploy pipeline.

Q · When this thing fails, what else goes with it?

Availability arithmetic — three replicas at 99.9% gives nine nines — assumes failures are independent. They are not. The interesting failures have a common cause, and a common cause takes all the replicas at once regardless of how many there are.

Q · Why did all three replicas fail at the same time?

Three replicas on one machine look replicated on the diagram and are not. Redundancy is a count of copies; resilience is whether the system survives a specific failure. Only placement, failure independence and an exercised recovery path turn the first into the second.

Q · I have three copies of everything. Why did the outage still happen?

Machine A logs an event at 12:00:01. Machine B logs one at 12:00:03. Which happened first? You cannot tell, and no amount of clock synchronisation makes the comparison sound. A wall-clock timestamp is a reading of a local oscillator, not a position in a shared timeline.

Q · Two machines timestamped two events. Can I use those timestamps to order them?

Skew is the difference between two clocks at the same instant. It is small most of the time, unbounded some of the time, and — the part that matters — invisible to the machines involved. A node cannot detect that it is the fast one.

Q · How far apart can two clocks in my fleet be, and what breaks at that distance?

Two clocks live in every machine and they answer different questions. The wall clock says what date it is and may jump; the monotonic clock only counts forward and has no idea what date it is. Using the wrong one is a real, shipped bug that surfaces during NTP steps and leap seconds.

Q · Which clock do I read to measure how long something took?

If clocks cannot order events across machines, something must. Lamport's happens-before relation orders exactly the pairs of events that could have influenced each other — and deliberately leaves everything else unordered. It is a partial order, and that is the point, not a limitation.

Q · If I cannot use clocks, what does it even mean for one event to come before another?

A single integer per node, incremented on every event and carried on every message, produces a numbering consistent with causality. The implication runs one way only, and almost every description of Lamport clocks gets that backwards.

Q · Can one counter per node give me a usable ordering without any clock?

Replace the single integer with one counter per node and compare component-wise. Now "neither is greater" is representable, which means concurrency becomes detectable — and conflicts become visible instead of silently resolved. The bill arrives as metadata that scales with the number of writers.

Q · How do I tell whether two versions conflict, rather than one superseding the other?

None, FIFO, causal, total. Each is strictly stronger than the last, each rules out anomalies the last permitted, and each costs more coordination — with the last one costing consensus. Choosing an ordering model is choosing how much availability you are willing to spend.

Q · How much ordering does my system actually need, and what does each level cost?

Delivering the same messages in the same order to every node sounds like a messaging feature. It is not: it is exactly as hard as consensus, and each can be built from the other in a few lines. That equivalence is the reason total ordering is expensive, and it is not going to get cheaper.

Q · Why does making every node see the same order cost so much more than making them see a causally correct order?

Replication is usually introduced as "for availability", as though availability were one thing. There are four distinct motives, they want four different protocols, and choosing the protocol before naming the motive is how teams end up paying multi-region coordination costs for a read-scaling problem.

Q · I have one copy of the data and it works. What does a second copy actually buy me?

One node accepts all writes and everyone else copies it. This is the default arrangement in almost every database you will meet, and its real product is not the copies — it is a single agreed order of writes, obtained without any agreement protocol on the write path.

Q · Why do most replicated systems funnel every write through one node, when that node is obviously a bottleneck and a single point of failure?

The leader waits until a follower confirms the write before telling the client it succeeded. This is the only way to make an acknowledgement mean something across machines — and it makes the write path only as available as the followers it waits for.

Q · What does it take for "your write succeeded" to survive the immediate death of the node that said it?

The leader acknowledges the client and replicates afterwards. This is the default in most deployments, it is often the right choice, and its defining property is a window of acknowledged-but-unreplicated writes whose size you can measure and should know.

Q · If my replica is always a little behind, exactly how much data am I agreeing to lose, and when do I find out?

Two or more nodes accept writes for the same data and replicate to each other. It buys local write latency and write availability during a partition, and it buys them by making write conflicts a structural certainty rather than a rare accident.

Q · What actually changes when a second node is allowed to accept writes for the same key?

The weakest guarantee that keeps a user interface honest: whatever else a reader may miss, they must see the effects of their own writes. It is a per-session property, it is far cheaper than global consistency, and it is the fix for the most common replication bug in production.

Q · A user saves a change, the page reloads, and the change is gone. Nothing failed — so what guarantee was missing?

A reader who has seen a value must never subsequently see an older one. Without this, a page refresh can un-post a comment and a polling client can watch a counter oscillate — and unlike stale data, moving backwards is something users interpret as the system being broken.

Q · Why does hitting refresh sometimes show older data than the previous load, and why is that so much worse than merely being stale?

Write to W replicas, read from R, and if R + W > N the read set must intersect the write set. The arithmetic is trivially true. The conclusion people draw from it — "so I read the latest value" — holds only under assumptions that real systems routinely violate, and the assumptions are the lesson.

Q · If R + W > N guarantees my read set overlaps my write set, why can I still read a stale value?

No leader, no failover, no election. A coordinator writes to N replicas and waits for W, reads from R, and repairs what it finds out of date. It removes an entire class of operational problems and replaces them with a permanent, low-grade requirement to reconcile versions.

Q · What does a system look like when no node is special, and what does that cost?

A consistency model is a contract about which histories the system is allowed to produce — that is, which observations you will never make. "Strong consistency" names no such contract, and the first skill this module teaches is refusing the phrase until someone says which model they mean.

Q · Someone says the database is "strongly consistent". What have they actually told me?

The strongest single-object guarantee: the system behaves as if every operation took effect instantaneously at some moment between its invocation and its response, and that moment respects real time. The whole subtlety lives in the word "interval" — an operation is a span, and linearizability asks whether some placement of effect points inside those spans explains what you saw.

Q · What does it mean for a distributed system to behave "as if there were only one copy"?

These are confused constantly, including in vendor documentation. Serializability is about transactions being equivalent to some serial order. Linearizability is about single operations respecting real time. Neither implies the other, and the conjunction has its own name — strict serializability — which is what most people mean when they say either word.

Q · My database says it is serializable. Does that mean a read always sees the latest committed write?

The most maligned and most misdescribed model in the field. Its actual claim is narrow and precise: if no new updates are made to an item, all replicas eventually agree on its value. It is a liveness property with no safety content, which means it forbids nothing you can observe in a finite run — and that, rather than "stale data", is the honest criticism.

Q · What exactly is an eventually consistent system promising me?

Preserve the order of operations that are causally related; allow any order for operations that are genuinely concurrent. It removes the anomalies that make users think a system is broken, it remains available during a partition, and there is a theorem saying nothing stronger can do both.

Q · How much ordering can I keep while still accepting writes on both sides of a partition?

Four properties — read your writes, monotonic reads, monotonic writes, writes follow reads — scoped to a single session. They cost a small amount of client-carried state, they require no agreement between nodes, and together they eliminate nearly every consistency complaint a user actually files.

Q · What is the cheapest set of guarantees that makes a replicated system stop feeling broken to its users?

Not "pick any two of consistency, availability and partition tolerance" — that framing is wrong and has misled a generation of design discussions. The theorem says that during a network partition, a system cannot be both linearizable and available for every request at every non-failing node. Partitions are not a choice you make; they happen to you.

Q · What does CAP actually constrain, and why is "pick two" the wrong way to say it?

CAP describes one failure. PACELC adds the case you actually live in: if there is a Partition, trade Availability against Consistency — Else, trade Latency against Consistency. The second half is where nearly all your engineering time goes, and CAP is silent about it.

Q · The network is fine, which is almost always. What is the trade-off then?

The decision is not made by comparing models. It is made by naming the invariant that must not be violated, checking whether one node can verify it alone, and then buying the weakest guarantee that protects it — per operation, not per system.

Q · I have to pick a consistency level for this operation. How do I decide without guessing?

A partition heals and two replicas hold different values for the same key. Neither happened first in any meaningful sense. The system must return something, so something decides — and if you did not choose what, a library default did.

Q · Two replicas accepted different writes to the same key. What decides the outcome?

"Keep the one with the later timestamp" is the most widely deployed conflict resolution rule and the least defensible. "Later" depends on a notion of time that does not exist across machines, and the rule silently discards the write that loses — with no error, no log line, and no way to recover it.

Q · Is it safe to resolve conflicts by keeping the write with the later timestamp?

A vector clock scoped to one object, tracked at the replicas rather than at every client. It answers one question exactly — does this version supersede that one, or are they concurrent — and answering it is what turns silent data loss into an explicit decision.

Q · Given two versions of an object, how do I know whether one supersedes the other or they conflict?

The store can tell you two versions conflict. It cannot tell you that two cart additions should union, two balance updates should compose, and two document titles need a human. The merge rule is domain knowledge, and it has to satisfy three algebraic properties or your replicas will never agree.

Q · The system detected a conflict. How do I decide what the value should be?

Design the data type so that merging is a mathematical join — order-independent, duplicate-safe, always convergent. It genuinely removes conflict resolution from the write path. It does not make the result the answer your business wanted, and it cannot enforce a global invariant.

Q · Can I make merge automatic and provably convergent — and what does that not give me?

"Eventually consistent" is a promise with preconditions, and most systems that claim it satisfy them only most of the time. Convergence needs delivery to every replica, a merge that is order-independent, and a quiet period in which to finish. Remove any one and replicas diverge permanently.

Q · My system says it is eventually consistent. What has to be true for that to actually happen, and how would I know if it stopped?

Consensus is not "making the cluster consistent". It is a much narrower thing: getting a set of nodes to agree on one value, once, in a way that survives some of them failing — and every other agreement problem you meet is this one wearing a costume.

Q · What problem does consensus actually solve, and what does it leave untouched?

Every guarantee in the previous lesson is conditional. Consensus needs a reachable majority, a network that eventually behaves, and a failure model that excludes lying nodes. Take any one away and the protocol does not degrade gracefully — it stops, or it stops being correct.

Q · What does a consensus protocol assume, and what happens when the assumption is false?

Many designs need exactly one node to act — one writer, one scheduler, one owner of a shard. Electing that node is easy. The hard part is that the node it elected cannot tell the difference between "I am still the leader" and "I was replaced eleven seconds ago and nobody could reach me to say so".

Q · How does a cluster choose exactly one node to act — and how does that node know it still may?

You cannot stop an old leader from believing it leads. What you can do is number leadership generations and have everyone reject anything stamped with an old number. A monotonically increasing counter is the whole mechanism — and it is why split-brain is survivable rather than catastrophic.

Q · If a deposed leader keeps acting, why does the system not corrupt itself?

A partition does not produce confusion. It produces two internally consistent, entirely reasonable worlds, each of which believes it is the whole world. Neither node is malfunctioning. The danger is not the belief — it is what the system lets a believer do.

Q · A partition splits my cluster and both halves think they are in charge. What actually goes wrong, and what stops it?

Every timeout-based scheme leaves a window where an old holder still believes it holds authority. You cannot close the window. A fencing token makes the window harmless: the resource itself remembers the highest token it has accepted and refuses anything lower, so a stale actor’s writes bounce off.

Q · How do I stop a node that lost its lock — but does not know it — from corrupting the resource it was protecting?

Raft was designed to be understandable, and its elections are the reason. Three states, one timeout, one rule about who may vote, and one rule about who may win — from which the guarantee "a new leader already holds every committed entry" falls out without any case analysis.

Q · How does Raft choose a leader, and why can the winner never be missing committed data?

The leader writing an entry to its own log means nothing. An entry is committed when it is replicated to a majority — and until then it is a proposal that a future leader is free to delete. Understanding that one distinction explains every log divergence you will ever debug.

Q · When is an entry actually committed, and what happens to entries a deposed leader wrote but never committed?

Paxos came first and proved the problem was solvable; Raft came later and proved it could be explainable. They reach the same guarantees by different decompositions. Knowing the shape of the family — and the one property they all share — matters far more than being able to derive any of them.

Q · How do Paxos, Multi-Paxos, Zab and Viewstamped Replication differ from Raft — and does the difference change anything I do?

Consensus is the right tool for a small number of facts and the wrong tool for almost everything else. The failure mode is not choosing it when you should not — it is putting it in the path of every business operation, and discovering that your throughput ceiling and your availability floor are now the same number.

Q · This decision feels like it needs agreement. Does it actually need consensus?

One database gives you all-or-nothing for free. Two services do not, because there is no component that can see both uncommitted states at once. Before reaching for a protocol, look hard at whether the invariant has to span the boundary at all.

Q · Two services must both change state, or neither. Neither can see the other’s uncommitted work. What are my actual options?

2PC makes several independent stores commit or abort together. It works by making each participant give up its right to decide alone — and that surrendered right is both the source of the guarantee and the source of every problem the protocol has.

Q · How can three independent databases be made to commit or abort as one, and what exactly does that cost?

The real objection to two-phase commit is one specific gap: the coordinator dies after collecting YES votes, and every participant sits holding locks with no legal way to decide. Understanding that window precisely tells you both why 2PC gets a bad name and how modern systems remove the problem.

Q · The coordinator crashed after everyone voted yes. Why can the participants not simply decide for themselves?

A saga replaces one atomic transaction with a sequence of local ones, each committed immediately and each paired with a compensating action. It never blocks. In exchange it gives up the "I" in ACID entirely — the half-finished state is visible to everyone, and your business logic now has to cope with it.

Q · If I cannot hold a transaction open across services, what does the alternative actually guarantee — and what does it stop guaranteeing?

A rollback erases history: the old value returns and nobody can prove the new one ever existed. A compensation adds history: the money moved, then moved back, and the statement shows both lines. Time did not reverse. Everything hard about sagas follows from that one difference.

Q · My saga failed at step 3, so I will undo steps 1 and 2. Why is "undo" the wrong word, and what does it cost me to use it?

An orchestrator holds the saga as an explicit state machine, issues commands and records replies. You can point at one place and ask "where is order 4821?" — and you have created a component that every workflow now depends on, and whose deployments must cope with thousands of in-flight sagas running the previous version.

Q · Who owns the knowledge of what step a saga is on — and what changes when the answer is "one component"?

Each service reacts to events and publishes its own. No component owns the flow, so adding a participant requires changing nothing upstream. The cost is that the workflow exists only as an emergent property of a subscription graph — and nobody can read it, test it end to end, or say where a given order is.

Q · If services just react to each other’s events, what happens to the workflow — and to my ability to reason about it?

A timeout leaves the outcome unknown. Retrying and not retrying are both guesses, with different costs — and the moment you retry, the receiver has no way to tell your second attempt from a genuine second request unless you gave it one before the first attempt left.

Q · The outcome is unknown. Should I retry — and what have I actually done to the receiver when I do?

An operation is idempotent when applying it twice leaves the world in the same state as applying it once. The trap is scope: the database write may be idempotent while the handler around it — which also emits an event, increments a counter and sends an email — is not. Idempotence is only as strong as the least idempotent thing the handler does.

Q · What makes an operation actually safe to repeat — and why do handlers that "are idempotent" still produce duplicate effects?

Deduplication rests on a claim that two requests are "the same". That claim is made across a machine boundary, using only what the caller sent, by a store that has its own partitioning, retention and failure domain. Every one of those is a scope boundary, and a duplicate slips through wherever two of them disagree.

Q · Two requests arrive carrying the same key. Under what conditions is that enough to conclude they are the same operation?

At-most-once and at-least-once are not two configuration options with a third, better one hiding behind a paywall. They are the two possible positions of the acknowledgement relative to the processing — and since ack and processing are separate actions that a crash can land between, the choice is forced by physics, not preference.

Q · Why are there exactly two delivery semantics, and what decides which one I have?

Three different things get called exactly-once: message delivery, processing attempts, and business effect. Delivery cannot be exactly-once over an unreliable network — that is a proof. What real systems provide is exactly-once *effect* for outputs inside one transactional boundary, and at-least-once for everything outside it.

Q · Systems advertise exactly-once. What are they actually offering, and where does it stop?

Detecting a repeat is a set-membership test over a stream that never ends, using memory that does. Every design choice — where the check happens, how exact it is, how long it remembers, how it is partitioned — is a way of trading one of those against the others, and each has a failure that looks like nothing at all.

Q · Given that duplicates will arrive, where and how do I detect them without storing every identifier forever?

Putting a broker between two services is usually sold as "decoupling". The precise trade is narrower and more useful: you convert a live availability dependency on the consumer into a durability dependency on the broker, and you pay for it with unbounded staleness and a new operational surface.

Q · Both services are running. Why not just call the other one directly?

A work queue distributes tasks across a pool of interchangeable workers. Each message is meant to be handled once, by whichever worker grabbed it. That single design choice — a claim, then a destructive completion — determines the scaling behaviour, the ordering behaviour, and every failure mode in the module.

Q · I have more work than one worker can do. What does a queue actually give me, and what does it take away?

In pub/sub the publisher announces that something happened and does not know or care who listens. Each subscriber gets its own copy and its own fate. The value is that adding a consumer requires no change to the producer; the cost is that the producer can no longer tell you whether anything downstream worked.

Q · Three teams need to know when an order is placed. How do I tell all of them without the order service knowing they exist?

The choice is usually decided by a single question: should one worker handle this, or does every interested party need to see it? That rule gets you the right answer nearly always — and then consumer groups over a log give you both at once, which is why the distinction has been quietly collapsing for a decade.

Q · Do I need a queue or a topic — and why does the answer keep being "a log"?

Receive, process, acknowledge. Where you put the acknowledgement relative to the processing is not a detail — it is the entire choice between at-most-once and at-least-once, and there is no arrangement of those three steps that gives you exactly-once.

Q · When exactly should my consumer tell the broker it is done — and what does the answer cost me?

When a consumer receives a message, the broker does not give it away — it hides it for a bounded period. If processing outlives that period, the message reappears and a second worker starts on it while the first is still running. This is not a rare edge case; it is a concurrency bug the broker will manufacture for you on a timer.

Q · My handler sometimes takes longer than expected. What does the broker do about it, and what does my code have to survive?

Retrying a transient failure is correct. Retrying a message that can never succeed is an infinite loop that consumes your consumers, your logs and your budget, and in an ordered stream it stops everything behind it. Telling the two apart is the whole problem, and you cannot do it perfectly.

Q · This message fails every single time. How long should I keep trying, and what stops it from taking the pipeline down?

Routing failed messages to a DLQ protects the pipeline, and that is the easy half. The DLQ is only a safety mechanism if someone is alerted, can inspect what failed, can fix the cause, and can make an explicit decision to replay or discard. A DLQ nobody reads is not a safety net — it is a silent data-loss mechanism with a reassuring name.

Q · The message failed five times and went to the DLQ. Now what — and who finds out?

An append-only log is an immutable, ordered sequence that consumers read by advancing a position. Nothing is removed when it is read; retention is governed by time or size, not by consumption. That single structural difference — a cursor over immutable data instead of a claim over a mutable set — is why a log supports replay, multiple independent readers, and per-key ordering, and why a queue never can.

Q · Everyone calls Kafka a message queue. What is actually different, and why does it change my design?

A topic is split into partitions, each an independent append-only log with its own offsets. Order is total inside a partition and undefined across them. Choosing a partition key is therefore choosing which ordering guarantee you get — and simultaneously choosing where your hot spots will be.

Q · My topic promises ordering. Ordering of what, exactly?

A consumer group is a set of processes that share the partitions of a topic between them, each partition assigned to exactly one member. Inside a group that behaves like a work queue; across groups, every group reads everything independently. One stored copy, two models, and a parallelism ceiling set by partition count rather than by how many machines you own.

Q · How do several consumer processes share a topic without duplicating work — and how does another team read the same data without affecting mine?

When a member joins or leaves a consumer group, partitions are redistributed. In the classic protocol every member gives up everything and waits for a new assignment, so the whole group stops processing. Worse, a member that is merely slow is declared dead, which triggers a rebalance, which makes everyone slower — a loop that produces the rebalance storm.

Q · A consumer restarted and my whole group stopped for 20 seconds. Why does one member leaving affect everyone?

A consumer must record how far it has got. Commit the offset before processing and a crash loses records. Commit after and a crash reprocesses them. There is no arrangement that does neither, because the offset store and the output store are different systems — and that, precisely, is where exactly-once actually lives or dies.

Q · When should my consumer commit its position, and why can I not have both no loss and no duplicates?

Every record carries two timestamps, whether or not you record both: the moment the thing occurred, and the moment your system observed it. They are never equal and sometimes differ by days. Every windowed aggregation is computed against one of them, and choosing without noticing is how a dashboard becomes confidently wrong.

Q · My hourly counts are wrong after an outage. Which clock was I counting by?

An event occurred at 10:00 and arrives at 10:05. The 10:00 window closed four minutes ago and something downstream has already acted on its result. You have exactly three options — drop it, restate the result, or route it aside — and each one hands a different problem to somebody else.

Q · The window already emitted its answer and a record for it just showed up. Now what?

A watermark is the processor saying "I believe I have now seen everything that happened up to time T". It is an estimate, not a fact — nothing can tell it that a source is finished. It is nonetheless the mechanism that lets an event-time window ever close, and understanding it as a heuristic rather than a guarantee is the difference between a pipeline you can debug and one you cannot.

Q · If I can never know an event-time window is complete, what makes it fire?

Sharding is not one decision. Storage, write throughput, working-set memory and recovery time are four separate ceilings, and which one you hit determines the partition key. Picking the key before naming the ceiling is how teams end up with a split that solves nothing.

Q · One machine is no longer enough. What does splitting the data actually buy me, and what does it silently take away?

hash(key) % N is the obvious way to spread keys across N nodes, and it works beautifully until N changes. Then roughly 1 − 1/max(N, N′) of all keys move — about 80% when going from four nodes to five. That single fact is the entire reason the next two lessons exist.

Q · If I route with `hash(key) % N`, what exactly happens when N changes?

Keep keys in sorted order and a range query touches only the partitions covering that range. The price is that load now follows the shape of your data and the shape of your traffic — and the single most common key in software, a timestamp, sends every write to exactly one partition.

Q · I need ordered scans across a partitioned dataset. What does keeping keys in order cost me?

Place nodes and keys on the same circular hash space and let each key belong to the next node clockwise. Adding or removing a node then disturbs only its neighbours — about K/N keys move instead of nearly all of them. The distributed-systems question the pattern write-ups skip: what happens to the requests already in flight while ownership changes hands.

Q · When a node joins or leaves, how much data actually moves — and what happens to requests issued during the change?

A ring with one position per node distributes badly — with ten nodes, the largest share is typically three times the smallest. Giving each machine many logical positions turns a very lumpy random partition into a nearly even one, spreads recovery load across the whole cluster, and lets a bigger machine simply take more positions. It is not free: the metadata, the streaming and the failure probabilities all change.

Q · Why does a consistent hash ring need many positions per machine instead of one?

A hash spreads keys. It does not spread requests. When one key — a celebrity account, a viral post, a global counter, a status row every worker polls — takes a large share of the traffic, it lands on exactly one partition no matter how good the hash is. A single key is the atomic unit of partitioning, and you cannot split below it without changing the data model.

Q · One key takes a huge share of my traffic. Why does hashing not help, and what actually does?

Moving partitions between nodes is not a background chore. It is a sustained, self-inflicted load spike: terabytes across the network, doubled disk I/O at both ends, and a destination whose caches are empty for every key that arrives. Rebalancing has to run while the system keeps serving — and it competes with that serving for exactly the same resources.

Q · Data has to move between nodes while the system stays up. What does that actually cost, and what breaks at the moment ownership changes?

Joins, transactions, aggregations, secondary indexes and unique constraints were all cheap when the data sat on one machine, because one storage engine could see all of it. Across partitions each becomes a distributed protocol with its own latency, its own failure modes and its own consistency story. The lever that matters most is chosen long before any of them: the partition key.

Q · Which operations get harder once the data is no longer co-located, and what does each one cost now?

The price of agreement is usually quoted in milliseconds. That is the smaller half. The real cost is structural: a node that must confirm with its peers before acting cannot act at all when those peers are unreachable — so every coordination point you add multiplies your availability by someone else’s.

Q · What does it actually cost me to make two nodes agree before either acts?

Before buying agreement, ask three questions: can a violation be repaired afterwards, do the operations commute, and can ownership be partitioned so one node decides alone? A surprising number of "we need a distributed lock" problems dissolve under one of them — and a stubborn residue genuinely does not.

Q · Can I restructure this so nodes act independently instead of agreeing first?

Every coordination mechanism in this domain exists to protect some statement that must never become false. Name that statement first — precisely, with its scope — and the architecture follows from it. Skip that step and you will choose a mechanism, then reverse-engineer a justification.

Q · What must never become false in my system, and what does that requirement force me to build?

A distributed lock is not a mutex that happens to be over a network. A mutex is a correctness mechanism backed by hardware; a distributed lock is a hint backed by a timeout, and the difference decides whether you may use it for efficiency or for correctness.

Q · When can I trust a distributed lock, and what is it actually protecting?

A lock that never expires deadlocks the first time a holder dies. A lease fixes that by making authority time-bounded — and in doing so it converts a distributed-systems problem into a clock problem, which is a trade worth understanding before you make it.

Q · How does authority get revoked from a node that has stopped responding — and what does the expiry actually assume?

The canonical distributed-systems accident: A takes a lock, A pauses, the lease expires, B takes the lock, A resumes believing nothing happened, and both act. Nobody is at fault, no error is logged, and the resolution is not a longer timeout.

Q · A process holding a lock stalls for a minute and then wakes up. What does it believe, and what does it do?

etcd, ZooKeeper and Consul are marketed as different products and are, underneath, the same four primitives over a consensus-backed key-value store. Learn the primitives — compare-and-swap, ephemeral keys, watches, leases — and every one of them becomes a configuration detail.

Q · What does a coordination service actually give me that a database does not?

A unique constraint in one database is a solved problem. Spread the data across shards and it becomes the hardest kind of invariant there is — a negation over global state, which cannot be checked locally by anyone. Four designs exist, and each fails differently.

Q · How do I guarantee a username is unique when no single node holds all the usernames?

A service registry answers "where is payments?" — and the answer is never a fact. It is a belief, formed from heartbeats that may be late, replicated through a store with its own consistency model, and cached at every hop on the way to the caller. The staleness of the four layers adds up, and the sum is how long you keep sending traffic to a machine that is gone.

Q · When my client asks a registry where a service is, what is that answer actually worth?

"Which nodes are in the cluster right now?" looks like a question with an answer. It is not. Every node holds a local view assembled from the absence of messages, views disagree, and no node can distinguish a crashed peer from a slow one. The engineering question is not how to get the right answer — it is which decisions are allowed to depend on a belief that may be wrong.

Q · Which nodes are in my cluster right now, and what am I allowed to do with that answer?

Each node picks a few random peers each round and exchanges what it knows. Information spreads like an infection: exponentially at first, reaching everyone in O(log N) rounds, at a per-node cost that does not grow with the cluster. The honest price is that it is probabilistic and eventual — no node can ever know that a fact has reached everyone, or when it will.

Q · How does a fact reach every node in a large cluster without anyone talking to everyone?

Replicas drift apart — a write that reached two of three, a node that was down for an hour, a hint that expired, a bit that rotted. Read repair fixes what gets read and abandons everything else. Anti-entropy is the background process that compares replicas systematically and repairs what nobody asked for, and it is the only reason your replication factor still means something a year in.

Q · My replicas have drifted apart and nobody noticed. What brings them back together?

Two replicas hold a billion keys and want to know which ones differ. Comparing them key by key costs a billion comparisons and a terabyte of transfer. A hash tree over key ranges answers the same question with one 32-byte exchange when they match, and about fifteen exchanges to locate a single difference when they do not. It is one of the genuinely beautiful ideas in distributed systems, and the details of getting it wrong are where all the operational pain is.

Q · How do two replicas find which of a billion keys differ, without sending each other a billion keys?

A binary failure detector forces you to pick one timeout, and that timeout is simultaneously too twitchy for the cheap decisions and too slow for the expensive ones. A graded detector — one that reports how suspicious a node is rather than whether it is dead — lets each consumer choose its own threshold where it actually knows the cost of being wrong.

Q · How long should I wait before deciding a node is dead — and why is that the wrong question?

Inside one process, backpressure is a blocking call: the producer stops because the consumer will not take the item. Across machines, "stop" is a message. It takes time to arrive, it can be ignored, and it usually reaches a queue rather than a producer — which is not backpressure, it is buffering.

Q · My service is overloaded. How does the pressure actually get back to whoever is generating the work?

Above capacity you will not serve every request. The only question is whether the system chooses which ones to drop, or lets timeouts choose at random. Shedding is that choice made deliberately — and it only works if a rejection costs far less than a success.

Q · I cannot serve everything. Which requests do I drop, and how do I drop them without spending the capacity I am trying to save?

An unbounded queue does not absorb overload; it postpones it and makes it worse. First latency climbs past every deadline, so the work you finally do is worthless. Then memory runs out. Admission control is the decision to find out at the door instead.

Q · Should this request be let in at all, given what I already owe?

A single user request becomes 20 service calls becomes 200 database calls. Now let each tier retry up to three times. The database does not see 3× the load, it sees 27× — because retries at independent tiers multiply rather than add, and every tier thinks it is being modest.

Q · Every tier retries three times, which seems reasonable. Why is the bottom of my stack seeing thirty times the traffic?

"Retry up to three times" sounds like a limit. It is not: it bounds one request while leaving the aggregate unbounded, because the number of requests is not something you control. A budget bounds the thing that actually hurts — total retry traffic as a share of original traffic.

Q · What is the right limit on retries, if "three per request" does not bound anything that matters?

Exponential backoff spaces out one client’s attempts. It does nothing about the fact that ten thousand clients all failed at the same instant and are all now counting down the same interval. Backoff without jitter reproduces the spike; it just reproduces it one second later.

Q · I added exponential backoff and the load spikes are still there, just further apart. Why?

You split the monolith into forty services, so a failure in one should stay in one. It does not, because the blast radius follows shared resources — a thread pool, a connection pool, a node pool, a database — and service boundaries drawn on a diagram do not cut any of those.

Q · One service is failing. Why is the failure spreading, and what actually stops it?

One shared pool gives the best utilisation and the worst isolation: whoever misbehaves takes everyone with them. Separate pools give the opposite. The engineering question is not which is better, it is how finely to cut — and the arithmetic of that cut is unforgiving in both directions.

Q · How do I stop one tenant, one dependency or one bad query from consuming the capacity everyone else needs?

The client waits 2 seconds. A waits 2 seconds for B, B waits 2 seconds for C. Every value is defensible on its own and the composition is nonsense: by the time C is still working, the client left long ago and A and B are doing paid work for nobody.

Q · The client gives me 2 seconds. How much of it may each hop below me spend?

A budget nobody can see is not a budget. Unless the remaining time travels with the request, every service independently waits its own configured default — and work continues for a long time after the last person who cared has left.

Q · How does a service four hops down learn that it has 300ms left rather than its configured 30 seconds?

When nobody is waiting for a result, computing it is pure waste, and under load that waste is most of your capacity. But cancellation is a message that may not arrive, and stopping a write halfway through leaves state that matches no intention at all.

Q · My caller disconnected. Should I stop the work I am doing on their behalf — and can I safely?

Most slow responses are not slow because the work is hard — they are slow because that particular replica hit a garbage collection, a cold cache, or a noisy neighbour. Asking a second replica after the p95 has elapsed converts a tail problem into a small, bounded amount of extra load.

Q · One replica is slow for reasons that have nothing to do with my request. Can I just ask someone else?

Each shard is slow only 1% of the time, which sounds excellent. Fan a request out to 100 of them and the chance that at least one is slow is 63%. The aggregate does not inherit the component’s median — it inherits the component’s tail, amplified by the width of the fan-out.

Q · Every shard has a good p99. Why is the p50 of my fan-out request terrible?

A local cache is a variable. The moment the cache is shared between machines, or duplicated on each of them, you have built a replicated store — with copies, staleness, a new network hop, and a new dependency — and you have built it without any of the machinery a replicated store normally comes with.

Q · Where should the cached copy live, and what did putting it there cost me?

The source changed. Which caches are now wrong, and how would they find out? Every answer is a message with a delivery semantics — and the ordering between that message and an in-flight cache fill is where the permanent-staleness bugs live.

Q · A value changed at the source. Which copies are stale, and what actually tells them?

In one process, a stampede is a few threads racing to recompute the same value and a mutex solves it. Across five hundred instances there is no mutex to take, so the database receives every one of those misses at once — for a single key.

Q · A hot key just expired and every instance missed simultaneously. What stops all of them hitting the database?

One product changes and ten thousand derived keys are wrong. A bulk import changes five million rows. And some of the copies are in browsers and CDN nodes you will never reach. At scale, invalidation stops being a correctness mechanism and becomes a best-effort speed-up over a TTL you must be willing to live with.

Q · One source record changed. How many cached things are now wrong, where are they, and can I actually reach them all?

A key lives on one shard. Add a hundred nodes and it still lives on one. When one celebrity, one flash-sale product or one feature flag receives more traffic than a single node can serve, the only options are to replicate the key, move it closer, or stop asking for it.

Q · One key is taking more traffic than any single node can serve. What can I actually do about it?

Open any distributed store and you find the same six layers: an API, a partitioner, a replicator, an agreement or conflict-resolution layer, a single-node storage engine, and a disk. Nothing in that stack is new to you — you have already met every layer separately. What is new is that the guarantees compose, and mostly compose downward.

Q · What is actually inside a "distributed database", and which layer owns which guarantee?

GFS and HDFS answered one question well: how do you store a file bigger than any single disk, on machines that fail weekly, and still read it at the speed of many disks at once? The answer — split the file into large chunks, replicate each chunk, and keep the map in one small service — is thirty years old and still the shape of the thing.

Q · How do you store a file that is larger than any one machine, on machines that keep dying?

No directories, no partial updates, no file handles. An object store takes a key and a blob of bytes, hashes the key to decide where the bytes live, and gives you back a guarantee about durability that is far stronger than anything you would build. The price is that everything you liked about a file system is gone.

Q · How does a key turn into bytes on a specific set of disks, and what does the flat namespace cost you?

A write returned 200. That fact alone tells you almost nothing. It may be in a memory buffer on one machine, on one machine’s platter, or on three machines in three zones — and the difference is invisible to the client, invisible on the dashboard, and decisive the moment a machine is replaced.

Q · The write returned 200 and then the machine died. Is the data still there?

There are exactly two ways to remember what a system knows: write down the current state, or write down every change. Every recovery scheme worth using does both — a periodic snapshot of state, plus the ordered log of everything since. The same pair, unchanged, is how a replica catches up.

Q · After a crash, how does a node rebuild what it knew — and why does the same structure make replication work?

You want a picture of what the whole system knew at one moment. There is no one moment — every node has its own clock and its own present, and messages are in flight between them while you look. Chandy and Lamport showed how to take a picture that is *causally* consistent anyway, without pausing anything.

Q · How do you capture a consistent global state of a running distributed system without stopping it?

Parallelism on one machine is about using more cores. Distributed compute is about using more machines, and the difference is not a matter of degree: the moment work crosses a machine boundary, communication becomes the dominant term and every task acquires an independent way to fail.

Q · When does splitting a computation across machines actually make it faster, and what does the split cost?

Two functions and one sort. MapReduce is not how anyone should write a new batch job today, and it remains the clearest way to see what distributed computation actually costs — because it puts the expensive part, the shuffle, right in the middle of the picture where you cannot ignore it.

Q · How does a computation expressed as two simple functions become a fault-tolerant job across a thousand machines?

Map is cheap. Reduce is cheap. The step nobody writes — moving intermediate data from every producer to every consumer — is where distributed compute spends its time, its money and its incidents. N mappers times M reducers is N×M transfers, and that product grows faster than your cluster does.

Q · Why is the job network-bound when the computation is trivial?

The intuition every programmer starts with is that you fetch the data and then work on it. At cluster scale that reverses: the code is kilobytes and the data is terabytes, so you ship the code to whichever machine already holds the bytes. Except when you should not — and the exceptions are more common every year.

Q · Is it cheaper to move the data to the computation, or the computation to the data?

A scheduler places tasks, tracks capacity, and re-runs work it believes was lost. That last word is the whole problem: "believes". A scheduler cannot know a worker died — it only knows the worker stopped talking, and re-running a task that is still executing is how a batch job charges a customer twice.

Q · The worker stopped responding. Do I re-run its task?

Behind a barrier, a job finishes when its slowest task finishes. Nine hundred and ninety-nine tasks completing in four seconds buys you nothing if the thousandth takes an hour. The counter-intuitive fix is to do the work twice on purpose — and it is only safe under conditions worth stating carefully.

Q · Why is my job as slow as its worst task, and what can I do about it?

Crossing a region boundary multiplies the cost of agreement by roughly a hundred. Nothing else about the system changes — the same consistency models are available, the same protocols work. What changes is that every one of them now has a price you can feel, and a failure domain you cannot hide.

Q · What actually changes when my system spans more than one region?

London to New York is about 5,600 km. Light in fibre covers that in ~28 ms one way, ~56 ms there and back, before a single packet is routed, queued or processed. No framework, no protocol and no amount of tuning moves that number. It is the floor under every cross-region design decision.

Q · How much of my cross-region latency is a physical constant, and how much can I actually engineer away?

Single-writer region, multi-writer, or partitioned ownership where each region is authoritative for its own slice of the keys. The first is simple and slow for distant users; the second is fast and makes every conflict real; the third is the one people forget, and is very often the right answer.

Q · Which region is allowed to accept a write, and what does each answer cost me?

One region serves; another stands ready. The model is easy to explain and easy to be correct about — there is exactly one writer, so there are no conflicts, ever. Its two weaknesses are not conceptual: the failover is slow and multi-step, and it is almost never exercised, which means its probability of working the first time is far below what the runbook implies.

Q · A standby region exists. What actually happens when I have to use it, and how long has it been since anyone checked?

Both regions serve, both accept writes, there is no failover step to get wrong. What you buy is local latency and continuously-proven capacity. What you pay is that every concurrent-write scenario you could previously wave away is now a thing that happens, in production, at a rate you do not control — and that a bad write propagates to both regions at wire speed.

Q · Both regions are serving live traffic. Which problems did that solve, and which did it create?

The link between two regions fails. Both are healthy, both are serving users, neither can reach the other, and neither can tell whether the other is dead or merely unreachable. Whether both may keep accepting writes has an answer — but it is a property of the invariant, not of your preference, and it is different for different data in the same system.

Q · The regions cannot see each other and both are up. Which of them is allowed to say yes?

Some data must legally remain inside a jurisdiction. That is not a deployment preference — it is a hard constraint that propagates upward into your partitioning key, your replication topology, your quorum placement, your indexes, your caches, your backups and your logs. It can forbid outright the design you would otherwise have chosen, and it is best discovered before you build it.

Q · A class of my data may not leave its jurisdiction. What does that forbid, and what does it force?

The five-step spine of every distributed incident. Most teams execute three of them, declare victory when the error rate returns to baseline, and leave derived state permanently wrong — because nothing in the dashboard was ever measuring it.

Q · The errors have stopped and the graphs are green. Is the system actually correct again?

The recommendation service is down and checkout still works. That outcome is not a virtue of the code — it is the result of somebody having decided, in advance and in writing, which dependencies are on the critical path for which feature. Nobody makes that decision well during an incident.

Q · When a dependency fails, which parts of my product should keep working — and did anyone decide that before today?

An experiment on a live system has six parts: a hypothesis, a measured steady state, an injected failure, an observation, an abort condition and a conclusion. The abort condition is not optional paperwork — it is the entire difference between an experiment and an outage you caused on purpose.

Q · How do I test a failure assumption on a real system without the test becoming the incident?

B slows down. A’s threads sit waiting on B. A saturates and starts timing out. A’s clients retry. Load on A rises. Services that depend on A begin to fail. The defining property is the positive feedback: every step of the failure response increases the load that caused the failure.

Q · Why did one slow dependency take down five services that were all healthy a minute ago?

Draw the dependency graph and, for each node, ask what stops working if it disappears. The answer is almost always worse than the team expects — because the graph everyone reasons from shows logical calls, and the failures propagate through shared infrastructure that appears on no diagram.

Q · If this component disappears right now, what exactly stops working — and what have I forgotten?

The discipline is deliberately introducing controlled failures to test assumptions you already hold about how the system behaves. Every word carries weight: deliberate, controlled, to test an assumption. Remove any one of them and what remains is an outage with a fashionable name.

Q · What is chaos engineering actually for, and what separates it from causing an outage on purpose?

Latency, packet loss, node crash, dependency error, disk full, network partition, clock skew. The first four are easy to inject and mostly confirm what you expect; the last three are hard to inject and are where the assumptions actually break. Difficulty and value point the same way, which is why most programmes only ever test the easy half.

Q · Which faults can I actually inject, and which of them will tell me something I do not already know?

Which request? Which service? Which region? Which version? Which dependency? Which state transition? Six questions, in order. If your system cannot answer them for a single failing request, every incident is solved by guessing, and the guesses are shaped by whoever spoke first.

Q · One request in ten thousand fails. How do I find out why, instead of guessing?

The obvious way to reconstruct what happened is to merge the logs and sort by time. It does not work: clocks on different machines disagree by more than the intervals you are trying to order, so the merged view can show an effect before its cause. Identifiers that carry causality are the answer, and this is where the time module pays off.

Q · I have logs from three services. How do I reconstruct what actually happened, in order?

Splitting a system into services does not make it faster or more scalable. It converts function calls into network calls, and in doing so hands you network failure, independent failure, distributed state, versioning, observability cost and deployment coordination — every subject in this domain, per boundary.

Q · What do I actually get, and what do I actually pay, when I split a system into services?

Many services, tightly coupled synchronously, sharing deployment assumptions and often a database schema. You pay every distributed-systems cost — network failure, ambiguous outcomes, versioning, tracing, operational overhead — and receive none of the independence those costs were supposed to buy.

Q · We have twelve services. Why does everything still have to ship together?

Does this boundary reduce coupling? Does it align with ownership and data? Can the two sides fail independently? Does it require constant synchronous chatter? Four questions, asked before the cut, that separate a boundary which buys independence from one that only buys network calls.

Q · I am about to draw a line between two parts of my system. How do I know it is in the right place?

Two services reading and writing the same schema keep joins and transactions — which are genuinely valuable and expensive to replace. They also make the schema a public interface, blur ownership, and turn independent deployment into a coordinated one. Both halves are true, and which dominates depends on facts you can check.

Q · Two services need the same data. Is sharing a database a shortcut or a mistake?

For every piece of state in the system, exactly one component decides what it is. Everything else — caches, search indexes, read models, replicas, downstream copies — is derived. Most incidents filed as "data inconsistency" are really an unanswered question about which component was authoritative.

Q · Two services disagree about a customer’s address. Which one is right?

The source of truth is the store whose value is correct by definition, and from which every other copy can be reconstructed. Naming it converts an argument into a repair procedure — and the reason so many incidents drag on is that nobody can name it.

Q · If everything except one store were deleted, which one would let me rebuild the rest?

Events flow into a projection, which produces a read model shaped for one query. The read model is fast, purpose-built and derived — and it is always behind the source by an amount you should be measuring, because everything downstream of that lag is a design decision you either made or inherited.

Q · I need a query the write model cannot serve efficiently. What does building a read model actually cost?

Derived state drifts. Not might — will. Compare the source of truth against each derivation, find the delta, repair it, and alert on the size of the delta. This is a designed part of the system with an owner and a schedule, and building it after the first incident is building it a year late.

Q · My search index and my database disagree. What repairs that, and who runs it?

User to orchestrator to model provider, tool service, memory store, retrieval service and worker agents. Every one of those arrows is a network call between machines that fail independently. Nothing in this domain stops applying because the caller is a model — and one property gets worse, because the caller is non-deterministic and can decide to retry on its own.

Q · What do I already know about distributed systems that applies, unchanged, to an agent architecture?

An agent calls send_email. The result never reaches the model — the call timed out, the stream broke, the process restarted. On the next turn the model sees a tool call with no result and does the natural thing: it calls it again. This is exactly the timeout-ambiguity case, with one addition that makes it worse: the party deciding to retry is non-deterministic and may retry with different arguments.

Q · A side-effecting tool may have executed. How do I make the retry safe when the thing retrying is a model?

Who owns task state, who decides completion, can two agents execute the same task, how are conflicts resolved, and how is progress persisted. Five questions, and every one of them is an ordinary distributed coordination problem with a decades-old answer. Describing them as agents "negotiating" or "collaborating" is not a harmless simplification — it hides the engineering that has to happen.

Q · Two or more agents are working on the same task. Which process is allowed to act, and what happens when it stops responding?

The process holding a twelve-step workflow is evicted at step seven. The question is not whether it crashed — it will — but what the smallest durable record is from which the work can continue without repeating a side effect. An agent workflow with side effects is a saga: there is no rollback, only compensation, and the recovery design has to be built on that.

Q · The process died mid-workflow. What do I need to have written down to continue safely?

Non-determinism makes replay unreliable, so the debugging technique this domain leans on hardest is weakened. A retried model call may take a different path, so a retry is not a repeat. Context is lost between steps and the agent forgets what it did. And a tool error can be read as content, so a failure becomes an answer. None of these raise an exception.

Q · What goes wrong in an agent system that no error rate, latency graph or health check will show me?