The question this answers
When does my system stop being one program and start being a distributed system?
Distribution adds no guarantee. Crossing a machine boundary only removes four: shared memory, a single clock, instant communication, and reliable failure detection. Any guarantee you want back has to be rebuilt in protocol, and paid for in latency, availability or both.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows its own local state and the contents of messages that have already arrived. Everything else — whether a peer is alive, what the global state is now, what time it is elsewhere, whether its own last message was delivered — is inference drawn from evidence that was already stale when it arrived.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The test that is actually useful
Counting machines is the wrong test. A single-process web app that talks to a managed database on another host is already distributed; a forty-service deployment that shares one database and dies as a unit is often distributed in the worst way and coordinated in none of the good ways.
The useful test has two clauses. Does a component your correctness depends on fail independently of you? And is the only way you learn about its state a message that can be lost, delayed, reordered or duplicated? If both are yes, you are in this domain, and everything this domain says applies whether you have two nodes or two thousand.
Note what this test excludes. Two threads in one process share memory and share fate: when the process dies, both die, and neither is left holding half of a decision. That is Concurrency’s subject and it is genuinely different — the same reasoning about interleavings, but with a reliable oracle for "is the other party still there" that we do not have.
| Shared memory | One clock | Communication cost | Failure detection | |
|---|---|---|---|---|
| Two threads, one processtypical | Yes | Yes | Nanoseconds | Reliable — shared fate |
| Two processes, one machinetypical | No (unless mapped) | Yes | Microseconds | Reliable — the OS tells you |
| Two machines, one rackprotocol | No | Approximately | Hundreds of microseconds | Unreliable |
| Two regionsprotocol | No | No | Tens of milliseconds, floor set by physics | Unreliable, and slower to suspect |
The four things you lose
Everything else in this domain is a consequence of exactly four losses. They are not four independent problems; they interact, and most real bugs are two of them at once.
No shared memory. State exists as copies. Two nodes never look at the same bytes, only at their own picture of them, which is as old as the last message. No Shared Memory: Every Node Sees a Copy takes this apart.
No perfect global clock. Two nodes cannot agree what "now" is, so a timestamp comparison is not an ordering. There Is No Global Clock takes this apart, and the time module rebuilds ordering from causality instead.
No instant communication. Every fact you have about another node describes its past. Correctness arguments that assume you are reading current state are wrong by construction. The Network Changes Everything takes this apart.
No perfect failure detector. You cannot tell a crashed node from a slow one, or from a healthy one behind a broken link. This is the least known of the four and it causes the most expensive incidents.
- Loss 1 forces replication, and replication forces you to decide what "the value" even means.
- Loss 2 forces you to order events by causality rather than by wall clock.
- Loss 3 turns every read of remote state into a read of remote *history*.
- Loss 4 is why split brain, double execution and stuck workflows exist at all.
The fourth loss is the one that gets you
Most engineers arrive already knowing about latency and about stale replicas. Very few arrive knowing that "is that node dead?" has no answer. In an asynchronous network — one where messages can be delayed by an unbounded amount — a crashed node and a slow node produce byte-for-byte identical evidence at every observer. There is no measurement that separates them, because the only thing you can measure is the absence of a message, and absence has two causes.
This is not a limitation of your monitoring. It is a property of the model, and it is why systems that must act on "the leader is gone" cannot simply detect it — they have to *decide* it, get a majority to agree on the decision, and then fence off the old leader so that being wrong is survivable rather than catastrophic. That chain — suspect, agree, fence — is most of the consensus module.
The practical consequence is a habit: whenever you write "if the node is down", replace it with "if we have decided to treat the node as down" and then ask what happens when that decision is wrong. Usually you find an assumption that only one process is doing something, held together by nothing.
What distribution does not buy you
It is worth being blunt: splitting a system across machines does not make it faster, more reliable, or more scalable. It makes those things *possible* and it makes them *conditional*. Faster only if the work parallelises and the added round trips cost less than the parallelism saves. More reliable only if the failures are independent — see Correlated Failure: The Independence Assumption Is Usually False and Redundancy Is Not Resilience, where they usually are not. More scalable only if the state can be partitioned without needing cross-partition agreement.
What it buys immediately, with no conditions attached, is partial failure, ambiguity, and an operational surface that has to be debugged across process boundaries. That asymmetry — costs certain, benefits conditional — is the argument in When Not to Distribute, and it is the single most useful thing in this module for a working engineer.
Key points
- A system is distributed when correctness depends on a component that fails independently and is only observable through fallible messages.
- Four losses: no shared memory, no global clock, no instant communication, no perfect failure detector.
- The fourth is the least known: a crashed node and a slow node are indistinguishable from outside.
- You never *detect* that a node is down; you *decide* to treat it as down, and must survive that decision being wrong.
- Distribution removes guarantees. Everything you get back is rebuilt in protocol and paid for.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Draw the boundary: which components can fail without taking the others with them.
- • For each boundary, list what crosses it — every crossing is a message that can be lost, delayed, reordered or duplicated.
- • For each piece of state, ask which node owns it and how every other node learns about it.
- • For each decision the system makes, ask which node makes it and on what evidence.
- • For each such piece of evidence, mark whether it is local knowledge or remote inference. The inferences are where the bugs live.
- • A message is lost and the sender cannot tell whether it arrived.
- • A message is delayed past the point where its content is still true.
- • Messages arrive in a different order than they were sent.
- • A message is delivered twice because a retry raced a slow original.
- • A link fails while both endpoints stay healthy, so each believes the other died.
- • Split brain: two nodes each conclude the other is dead and both act as the single writer. The operator sees two nodes reporting
role=leaderin the same metric series, and a data set with two disjoint suffixes. - • Stale-read bug: a service reads a replica, decides on data that was correct three seconds ago, and writes a conclusion that is now wrong. The operator sees no errors at all — only a support ticket describing an impossible state.
- • Zombie worker: a node judged dead resumes after a pause and completes work that has already been reassigned. The operator sees duplicate side effects with no corresponding duplicate request in the entry-point logs.
- • Cross-boundary debugging failure: an incident that is visible only as elevated latency at the edge and normal metrics on every individual service, because the problem is the interaction and no single node observes it.
- • None of the four losses can be removed. Coordination lets you *agree on a shared fiction* — a single leader, a single order, a single decision about who is alive — and pay for it in round trips and in availability during partitions.
- • Every coordination point converts an availability problem into a latency problem when the network is healthy, and into an availability problem again when it is not. Coordination Couples Availability is the whole subject.
- • The engineering skill is finding the smallest set of decisions that genuinely need agreement, and letting everything else proceed without it — Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- • Local state on each node remains exactly as durable as that node made it; distribution does not weaken a local commit.
- • Anything defined across nodes — an invariant, an ordering, a count — is unenforced for the duration of the failure unless a protocol was explicitly built to hold it.
- • The system continues to serve requests with each node acting on its own last-known picture, which is precisely the behaviour that produces divergence.
- • Detect: the absence of a signal is a signal — alert on silence, not only on errors.
- • Contain: bound the damage a wrongly-suspected node can do, using fences and leases rather than trust.
- • Recover: bring the node back and reconcile its state against the authority, rather than assuming its state is fine because the process started.
- • Reconcile: run a comparison that would notice divergence even when nothing errored — this class of bug is silent by construction.
- • Verify: check the invariant itself, not the health of the components that were supposed to maintain it.
- • Count of nodes claiming an exclusive role, as a gauge. It should be exactly one, and an alert on
!= 1catches split brain in seconds rather than days. - • Time since last successful message from each peer, per observer — not a global "healthy" boolean, because observers disagree and the disagreement is the data.
- • Age of the data behind every cross-service read, propagated as a field, so staleness is visible rather than assumed away.
- • Rate of decisions taken on inferred state (leader elections, failovers, evictions), which should be low and boring.
- • When the workload genuinely exceeds one machine — data too large, throughput too high, or a single machine’s failure being unacceptable.
- • When independent failure is the goal rather than the cost: isolating a risky component so its collapse cannot take the core down.
- • When users are geographically spread and the speed of light, not your code, is the latency budget.
- • When the motivation is organisational fashion rather than a constraint you can name and measure.
- • When the system is small enough that a single machine with a replica would meet every requirement, and the distribution only adds partial failure.
- • When the team cannot yet debug across a process boundary — the failure modes arrive before the tooling does.
- • One process, vertically scaled, with a hot standby: covers a surprising fraction of real systems and has no partial failure between components.
- • A modular monolith with strict internal boundaries: gets the design discipline of services with none of the network in the middle, and can be split later at a boundary you have already validated.
- • A managed service that hides the distribution behind an interface with a stated guarantee — you still inherit its failure modes, but not its protocol design.
The four things a machine boundary takes away
What people believe, and what is true
A distributed system is one with many servers.
It is one where correctness depends on independently-failing components observed only through fallible messages. One app plus one remote database qualifies.
Adding machines makes the system more reliable.
It adds more things that can fail and more ways they can fail *partially*. Reliability comes from independence and from protocol, and both must be engineered.
Good monitoring can tell you whether a node is down.
Monitoring can tell you a node has stopped responding to you. The distinction between dead and slow is not observable — that is the point.
Distributed systems are just concurrency with more latency.
Concurrency has shared memory and shared fate. Here you have neither, plus an unreliable failure detector — the reasoning changes shape, not just scale.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
You are distributed once something you depend on can fail without you, and you can only learn about it by message. That removes four things: shared memory, a single clock, instant communication, and any reliable way to know who is alive.
Practical
Go through your design and mark every statement of the form "if X is up" or "the current value of Y is". Each one is an inference from a stale message. Rewrite them as "we have decided to treat X as down" and "Y was Z as of N milliseconds ago", then check whether the logic still holds. The ones that stop holding are your incidents.
Advanced
The formal frame is the system model: synchronous (known bounds on delay and clock drift), asynchronous (no bounds at all), and partially synchronous (bounds hold eventually, but you never know when). Almost every impossibility result in this field — FLP, the CAP argument, the unattainability of a perfect failure detector — is a statement about the asynchronous model, and almost every working system is engineered for the partially synchronous one. Knowing which model a claim assumes is how you tell a real guarantee from a marketing one.
Apply it
- ⚡ A team splits a monolith into six services and keeps the single shared database. Which of the four losses have they taken on, and which have they avoided? What have they gained?
- ⚡ A batch job runs on one machine and writes to network storage. Argue both sides of whether this is a distributed system, then say which answer changes your design.
- 💬 Is an application server plus a managed Postgres a distributed system? Defend your answer.
- 💬 Name the four things you lose when a call crosses a machine boundary. Which one causes the most expensive bugs, and why?
- 💬 Your health check says a node is down. What do you actually know?