The question this answers
Two or more agents are working on the same task. Which process is allowed to act, and what happens when it stops responding?
With fenced ownership over a durable task record: at most one process may advance a given task at a time, and a process whose lease has expired cannot commit even if it has not noticed. Without it: none. Concurrent orchestrators will interleave arbitrarily on shared state, and no amount of instruction or protocol at the model layer changes that.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
An agent process knows what it read from the task store and when, and whether its own lease was valid at that moment. It does not know whether another process is also working the task right now, whether its own lease has since expired, or whether a peer that has gone quiet has crashed or is merely slow — [[crash-vs-slow]], unchanged. A supervisor knows only what workers reported, which is a claim, not an observation.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The anthropomorphism is the bug
Multi-agent designs are routinely described in social language. The agents *negotiate*. The supervisor *delegates*. The workers *collaborate* and *reach consensus*. Each of those phrases replaces an engineering question with a metaphor that sounds like it has already been answered, and the effect is that the question never gets asked.
Translate them and the missing work becomes visible. "The agents negotiate" means two processes are writing shared state with no lock. "The supervisor delegates" means an unfenced lease has been handed out. "The workers collaborate" means an unsynchronised fan-out with partial failure. "The agents reach consensus" means two model outputs agreed, which is not agreement about anything — consensus is a durable decision recorded once, and two components producing the same string have not recorded anything.
This matters practically because the metaphor suggests the wrong fixes. If agents negotiate, you improve the negotiation — better prompts, a protocol for messages, a richer conversation between them. If two processes contend for state, you assign an owner. The second fix works and the first does not, and teams spend months on the first because the vocabulary pointed there.
So the discipline for this lesson: whenever a multi-agent design is explained to you in social terms, ask which process holds the write token for this record, and what happens if it stops responding mid-task. A design that cannot answer that has not been designed yet, and adding another agent will not help.
The five questions, and their existing answers
These are the questions that decide whether a multi-agent system works. Every answer on the right is older than the agent framework you are using.
| Question | The mechanism | What goes wrong without it |
|---|---|---|
| Who owns task state?protocol | A durable task record with a single owner field, held under a lease that expires — `[[leases]]`, `[[distributed-locks]]`. | Two processes advance the same task; steps interleave; effects duplicate. |
| Who decides completion?protocol | One authoritative writer performs a state transition to `complete`, based on evidence in the log. Completion is a write, not an opinion. | A task is marked done because a model said so, while its effects never happened. |
| Can two agents execute the same task?protocol | Yes — assume it. At-least-once dispatch plus idempotent steps, per `[[agent-idempotency]]`. Do not attempt exactly-once dispatch: `[[exactly-once]]`. | You build a scheme that "should not" double-dispatch, and the first redelivery duplicates every side effect. |
| How are conflicts resolved?protocol | The same options as any distributed write: single ownership (best), version vectors, or an application merge — `[[application-merge]]`. Last-write-wins is usually wrong here because the losing write may have had side effects. | Two agents write the shared plan; one silently overwrites the other; work is repeated or dropped. |
| How is progress persisted?protocol | An append-only step log outside any agent’s context — `[[checkpoint-and-log]]`, `[[append-only-logs]]`. | Progress lives in a context window, so a crash loses it and a second replica never had it. |
A supervisor is a leader, with all that implies
The supervisor pattern — one agent plans and dispatches, several workers execute — is a leader-follower architecture. Naming it that immediately imports a set of questions the pattern’s usual description omits.
How was this leader chosen, and can there be two? If the supervisor is a process that can be started twice — by a queue redelivery, by an autoscaler, by a retry — then two supervisors may plan and dispatch concurrently for the same task. That is [[split-brain]], and the resolution is the same as everywhere else: a lease with a term, and workers that reject instructions carrying a stale term. [[terms-and-epochs]] and [[fencing-tokens]] are the mechanisms.
What happens if the supervisor dies mid-task? Workers keep working, results arrive with nobody to receive them, and the task sits in running forever. Unless the workers hold leases too, and unless something scans for tasks whose owner’s lease expired, the failure is silent. The observable is a growing population of tasks that are neither failed nor complete, which no error-rate dashboard shows.
And the twist that is genuinely agent-specific: a restarted supervisor has no memory of its previous invocation beyond what was persisted. An ordinary leader recovers its state from a replicated log. A re-invoked supervisor recovers its state from a context window that is rebuilt from whatever the step log holds — so if the log is incomplete, the new supervisor will re-plan, and re-planning is not idempotent. It may produce a different plan, dispatching work that overlaps the work already in flight. This is why the step log must record dispatched work, not just completed work.
- Supervisor A (term 4) — GC pause; lease expired 12 s ago
- Supervisor B (term 5) — acquired the lease, re-planned from the step log
- Worker 1 — executing a task dispatched by A
- Worker 2 — executing the same task, dispatched by B
- Task store — holds owner, term, and the step log
- s1believes “I am the supervisor and may dispatch”✕ and it is false
- w1believes “my instructions are current”✕ and it is false
- s2believes “no other supervisor is dispatching”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Completion is a write, not a claim
The single most damaging shortcut in multi-agent systems is letting the model decide that the task is finished. It is seductive because the model is the component with the best view of the work — and it is wrong because the model’s view is its context, which is a lossy replica.
A model outputs "the task is complete" when the task looks complete *in the text it can see*. That text may omit a step whose result never arrived. It may include a tool error the model interpreted as a satisfactory outcome. It may be a summarised context in which three steps were compressed into a sentence. In each case the model is reporting faithfully on incomplete information, and the resulting complete status is false.
The correct construction: completion is a transition in the task store, written by deterministic code, conditioned on evidence in the step log — every required step in succeeded, every effect confirmed, every invariant checked. The model’s "done" is one input to that check, on par with a timer or a budget. deterministic-evaluators in Agentic Engineering is the tooling; the distributed-systems point is simply that an untrusted component may not perform the state transition that ends the workflow.
The same reasoning kills "the agents agree that it is done". Two model outputs matching is a correlation between two calls to a similar function on similar input, not an agreement protocol. [[what-consensus-solves]] is agreement about a value, made durable, tolerant of failure, and impossible to un-decide. Nothing about two agents concurring provides any of that.
When not to add another agent
Every additional agent adds a process that can hold stale state, act on it, and produce a side effect. The costs are the ordinary distributed ones and they arrive immediately; the benefits are usually about parallelism or specialisation and are frequently obtainable without a second process at all.
Parallelism within one orchestrator — several tool calls in flight from a single loop — gives you concurrency with none of the ownership problem, because there is still exactly one process advancing the task. That is the version to exhaust first. A second *process* is warranted when the work must survive the first process dying, when it must run somewhere else, or when it genuinely proceeds independently for a long time. Specialisation is a prompt-and-tool-scope decision and rarely needs a separate process.
And when you do add one, the useful test is whether the work partitions cleanly. Agents working on disjoint state — separate files, separate records, separate tickets — need almost no coordination, and multi-agent works beautifully there. Agents working on shared state need ownership, fencing and merge policies, and it works badly. This is [[data-ownership]] and [[service-boundaries]] applied to agents: cut so the pieces do not touch, and the coordination problem disappears rather than being managed.
- Disjoint work, no shared writes → multi-agent is nearly free. Do it.
- Shared writes, one owner per record → workable, needs leases and fencing.
- Shared writes, no owner → this is the design that fails, and no prompt fixes it.
- Parallelism only → use concurrent tool calls inside one loop, not more processes.
- Specialisation only → use different prompts and tool scopes, not more processes.
Key points
- Social vocabulary — negotiate, delegate, collaborate, agree — hides ordinary coordination problems and points at the wrong fixes.
- Five questions decide a multi-agent design: who owns state, who decides completion, can a task run twice, how are conflicts resolved, how is progress persisted.
- Every answer is an existing mechanism: leases, fenced ownership, at-least-once plus idempotence, a merge policy, an append-only log.
- A supervisor is a leader; two supervisors is split-brain, and the fix is a term plus fencing, not better instructions.
- A restarted supervisor re-plans from the step log — so the log must record dispatched work, not only completed work.
- Completion is a state transition written by deterministic code from evidence; a model saying "done" is an input, not an authority.
- Two agents agreeing is not consensus; consensus is a durable, single, unrevokable decision.
- Agents on disjoint state need almost no coordination. Cut the work so it does not overlap, rather than managing the overlap.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A task exists as a durable record with a status, an owner, a term and an append-only step log.
- • A process claims the task by acquiring a lease with a bounded expiry, incrementing the term.
- • Every write the owner makes carries its term; the store rejects writes carrying a stale term, which fences a slow or partitioned owner.
- • The owner dispatches work to workers, recording the dispatch in the log before the work begins.
- • Workers hold their own leases on their subtasks and heartbeat; an expired worker lease makes the subtask reclaimable.
- • Results are written to the log by the worker, keyed idempotently, so a redelivered subtask collapses into the original.
- • Deterministic code evaluates the log against the completion criteria and performs the transition to
complete. - • A sweeper scans for tasks whose owner lease expired and returns them to the queue for reclaim.
- • The owner pauses — GC, a slow model call, a throttled container — long enough for its lease to expire while it believes it still holds it.
- • A queue redelivers a task whose first execution is still running, producing two owners.
- • A worker completes its subtask and the result is lost, so the supervisor waits forever on a step that is done.
- • A supervisor crashes and restarts, re-plans from an incomplete log, and dispatches overlapping work.
- • Two agents write the same shared artefact — a plan, a document, a summary — and one overwrite discards work that had side effects.
- • The task store is unavailable, so no process can safely claim or fence anything, and the correct behaviour is to stop rather than proceed.
- • Duplicated work with duplicated effects: two agents process the same ticket and the customer gets two replies. Both traces look correct in isolation; only a cross-trace view by task id reveals it.
- • Task stuck in
runningforever: the owning process died and nothing had a lease to expire. The observable is a rising count of tasks older than any plausible duration, with a zero error rate — nothing alerts unless age is monitored explicitly. - • Falsely completed task: the record says
completeand the work never happened, because a model’s claim was written straight through. Discovered by the customer, or by a reconciliation job, never by monitoring. - • Lost plan update: a supervisor overwrites a worker’s progress note on the shared task record. The observable is a worker repeating a step it already finished, with no error in either process.
- • Zombie supervisor: an old-term supervisor keeps dispatching after being replaced, so worker count doubles for a task. The operator sees unexplained load and duplicate results, and the old supervisor’s logs show it behaving perfectly.
- • Reclaim storm: a store slowdown causes many leases to expire at once, every task is reclaimed simultaneously, and the reclaim traffic keeps the store slow —
[[retry-amplification]]in the coordination layer rather than the request path.
- • Ownership is the primary coordination cost, and it is paid on claim and on heartbeat rather than on every action — which is why leases are affordable where a lock per step would not be.
- • The term or fencing token is what converts "I believe I am the owner" into a checkable claim at the store, and it is the only part that provides safety rather than liveness.
- • Completion is a second coordination point, and it must have exactly one writer.
- • Communication *between* agents — passing results, sharing context — is not coordination and does not need any of this machinery. Conflating the two is what makes agent-to-agent messaging protocols look more load-bearing than they are.
- • A fenced owner that loses its lease can no longer commit, so its continued execution is wasteful but not unsafe.
- • Work already dispatched continues; results arriving after a reclaim must be accepted idempotently rather than rejected, or the reclaim throws away completed work.
- • The step log remains the truth throughout, so a new owner can always reconstruct what happened — provided dispatches were logged, not just completions.
- • With the task store unavailable, no task can be safely advanced; the correct posture is to stop, since proceeding without fencing is how duplicate effects are created.
- • Detect: monitor task age, lease expiry rate, and the count of tasks with no live owner. None of these appear in error-rate monitoring.
- • Contain: expire leases and stop the old owner from committing before assigning a new one — fence first, reassign second, exactly as in
[[region-active-passive]]. - • Recover: reclaim the task, rebuild context from the step log, and resume at the first unresolved step rather than re-planning from scratch.
- • Reconcile: for steps in an unknown state, query the tool by idempotency key; for shared artefacts overwritten in a conflict, apply the merge policy rather than accepting the last write.
- • Verify: assert that each task has at most one live owner and that every
completetask has evidence for each required step — the two invariants that this whole design exists to protect.
- • Live owners per task — the direct measure of whether the ownership scheme is working, and it should be exactly one or zero.
- • Lease expiries and reclaims per minute, which spike before duplicate work appears and are the earliest available warning.
- • Fencing rejections at the store, which prove the fence is engaged; a permanently zero rate usually means it is not wired up.
- • Task age distribution with a threshold alert, since a stuck task produces no other signal.
- • Duplicate subtask executions per task id, measured across traces rather than within one — the only place duplication is visible.
- • Any system where more than one process can act on the same task — including a single agent running on two replicas, which is the case teams most often overlook.
- • Long-running workflows where a process will certainly be restarted before the work completes.
- • Queue-driven agents, where at-least-once delivery makes concurrent execution a certainty rather than a risk.
- • Designs under review, where the five questions quickly separate a workable architecture from a diagram.
- • A single-process, short-lived agent with no shared state: leases and terms are pure overhead there.
- • Work that is genuinely disjoint, where the right move is to rely on the disjointness rather than to add coordination that does nothing.
- • When the machinery is added but the effects are still not idempotent — ownership reduces the frequency of duplicates without removing them, and treating it as sufficient is how the remaining ones become surprising.
- • One agent with concurrent tool calls: parallelism without a second process, and therefore without an ownership problem at all.
- • A deterministic workflow engine that supplies task ownership, timers, retries and the step log as infrastructure — the least glamorous and most reliable option.
- • Partition the work so agents touch disjoint state, which removes the coordination problem instead of managing it.
- • A single writer with a queue: agents propose actions, one ordinary deterministic process applies them, and all the contention collapses into that process’s serial execution.
- • Fewer agents. Agentic Engineering’s
when-not-multi-agentmakes this case directly, and it is usually right.
Two processes, one task record
What people believe, and what is true
Multi-agent systems coordinate through conversation.
They coordinate through shared mutable state, exactly like any set of processes. The conversation is how information moves; it is not how contention is resolved, and no message protocol substitutes for an owner.
Two agents agreeing is a form of consensus.
Consensus is one durable decision, made once, that cannot be un-decided, and that tolerates failures. Two similar functions producing the same string satisfies none of those conditions.
The supervisor guarantees only one worker does each task.
Only if the supervisor is itself unique, which requires a lease and a term. A supervisor that can be started twice guarantees the opposite.
If the agent says the task is done, the task is done.
The model reports on its context, which may be missing steps, may contain an error it read as success, and may have been summarised. Completion must be derived from the log by code.
Adding agents adds redundancy.
It adds concurrency. Redundancy requires that a failure of one is *detected* and its work reassigned, which is leases and sweepers. Without those, more agents means more ways to leave a task half-done.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Two agents on one task is two processes writing shared state. Give the task one owner under an expiring lease, make every write carry a term the store checks, put progress in an append-only log, and let deterministic code decide completion.
Practical
Answer the five questions in writing before building: owner, completion authority, double-execution safety, conflict policy, persistence. Then wire the mechanics — lease with expiry, term checked at the store, dispatch logged before work starts, sweeper for expired owners, idempotent result writes so a reclaim does not discard finished work. Monitor live-owners-per-task, lease expiry rate, and task age, because none of the failures here produce errors.
Advanced
The reason multi-agent designs fail more often than their single-agent equivalents is not that coordination is hard in general — it is that these systems reintroduce shared mutable state *and* keep a lossy replica of it inside each participant’s context. Each agent decides what to do next from a stale, compressed view, so the usual mitigation of "read before you write" is unavailable: the read that mattered happened several steps ago and has since been summarised. The only constructions that hold are the ones that do not depend on any participant having a current view — exclusive fenced ownership so that a stale actor cannot commit, and idempotent effects so that a stale actor’s duplicate collapses. Both are properties of the store, not of the agents, which is the general lesson: in an agent system, correctness lives in the deterministic components, and the model is a planner whose output must survive being wrong.
Apply it
- 🔧 Answer the five questions for an existing multi-agent design, and mark which have no implemented mechanism.
- 🔧 Add a term to your task store writes and demonstrate that a paused owner is rejected after its lease expires.
- ⚡ A queue redelivers a task after a visibility timeout while the first orchestrator is still calling the model. Describe every place the two executions can collide and which mechanism prevents each.
- ⚡ A datastore slowdown causes 4,000 leases to expire within a minute. Describe what happens next and how you would bound it.
- 💬 Two agents are working the same task. Which one is allowed to write, and how does the store know?
- 💬 A supervisor pauses for 30 seconds and its lease expires. It wakes up and dispatches more work. What stops that work from landing?
- 💬 Why is "the agents agreed the task is complete" not consensus?
- 💬 When would you add a second agent process, and when would you use concurrent tool calls instead?