The question this answers
What do I already know about distributed systems that applies, unchanged, to an agent architecture?
None inherent. An agent system inherits exactly the guarantees of its weakest component and its state store, and adds none of its own. Specifically: the orchestrator provides no atomicity across steps, the model provider provides no exactly-once semantics, and the agent loop provides no durability unless something outside it writes to disk.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The orchestrator knows what it has persisted and what it received. It does not know whether a tool call it did not hear back from executed, whether the memory it just read reflects a write from a concurrent session, or whether a worker agent it dispatched is running, finished or dead. The model knows strictly less: its entire knowledge is the context it was handed, so anything absent from that context is, to it, indistinguishable from something that never happened.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Draw the arrows and the whole domain reappears
Here is the architecture at its most ordinary. A user sends a request. An orchestrator — a loop, in a process, on a machine — assembles context and calls a model provider over the internet. The model asks for a tool. The orchestrator calls a tool service. It reads and writes a memory store. It queries a retrieval service. It may dispatch worker agents, which do the same thing recursively. Results come back, the loop runs again, and eventually something is returned.
Now notice: the orchestrator is a client of five independent remote services, none of which it controls, all of which can be slow, unavailable, or ambiguous. That is not an agent-specific problem, it is the founding condition of this domain, and every technique the earlier modules built applies without modification. The value of this module is not new theory. It is the mapping.
The mapping, arrow by arrow
Read this table as a claim that the agent-specific failure on the left has already been solved — or proven unsolvable — on the right. That is the useful thing to know, because it means the answer to "how do we handle duplicate tool calls?" is not a novel research problem, it is [[idempotent-operations]] with a different vocabulary.
| Arrow | What goes wrong | What it actually is |
|---|---|---|
| Orchestrator → model providertypical | Request times out, or the stream breaks halfway through a tool-call block. Did the provider generate it? Were you billed? Unknown. | Timeout ambiguity, plus a partial response — see `[[timeout-ambiguity]]` and `[[partial-failure]]`. |
| Orchestrator → tool serviceprotocol | A call is retried and the side effect happens twice: two emails, two charges, two tickets. | At-least-once delivery without idempotence — `[[agent-idempotency]]`. |
| Orchestrator → memory storetypical | The agent writes a fact, then reads it back on the next step and gets the old value. | A read-your-writes violation against a replicated store — `[[read-after-write]]`. |
| Orchestrator → retrievaltypical | The document was updated but the index has not caught up, so the agent grounds an answer in stale content and states it confidently. | Eventual consistency of a derived view — `[[materialized-views]]`, `[[eventual-consistency]]`. |
| Orchestrator → worker agentsprotocol | Six subtasks dispatched, four return, one errors, one never answers. What is the state of the task? | Partial failure in a fan-out, with the tail setting the latency — `[[partial-failure]]`, `[[fan-out-tail-latency]]`. |
| Step queueprotocol | A step is redelivered because the visibility timeout expired while the model was still thinking. | Visibility timeout shorter than processing time — `[[visibility-timeout]]`, `[[delivery-semantics]]`. |
| Provider outagetypical | The model provider is down or rate-limiting; every agent in the fleet retries simultaneously. | A hard dependency with correlated failure and retry amplification — `[[correlated-failure]]`, `[[retry-amplification]]`. |
The one property that is genuinely different
If it were only the mapping above, this module would be a footnote. It is not, because one component in the diagram behaves unlike any component the rest of the domain assumes.
Every distributed systems result in this domain assumes crash-stop or crash-recovery components: a node either does what it was programmed to do, or it stops. [[byzantine-failures]] — components that produce arbitrary output — are treated as an exotic case requiring special protocols, because ordinary software does not behave that way.
A model does. Not maliciously, but the failure mode is structurally the same: given the same input it may produce different output; given an error it may produce plausible-looking content; given a truncated context it may confidently assert something that is not there. It is a component that fails by producing a wrong answer rather than by stopping, which is precisely the class the standard protocols exclude.
The practical consequence is one design rule that runs through the rest of this module: the model is not a trusted node in the protocol. It does not hold state, it does not decide completion, it does not own a lock, and its output is an input to be validated rather than a result to be applied. Everything that must be correct lives in ordinary, deterministic code around it. Agentic Engineering’s structured-outputs and argument-validation are the mechanisms; the distributed-systems reason for them is this paragraph.
Where the durability boundary is
Ask of any agent system: if this process dies right now, what survives? The answer is almost always "whatever was written to a store", and the answer is almost never "the conversation".
The context window is in memory, in one process, on one machine. It is not a checkpoint, it is not replicated, and it is not the source of truth — it is a cache of a state that ought to exist elsewhere. Systems that treat the context as the state have chosen a single-node, non-durable architecture, and they behave exactly like one: a restart loses the work, a second replica does not know what the first was doing, and there is no way to answer "what has this task already done?" after a crash.
Drawing the boundary explicitly is most of the design. On the durable side: the task record, the step log with its statuses, tool call ids and results, the memory writes. On the ephemeral side: the assembled prompt, the in-flight stream, the model’s reasoning. [[checkpoint-and-log]] is the underlying idea and [[agent-workflow-recovery]] is what it looks like here.
design A — context is the state
orchestrator holds messages[] in memory
process dies at step 7 of 12
survives: nothing. the task is lost, or restarted from zero.
restarted from zero: steps 1-7 side effects happen AGAIN.
design B — step log is the state
every step: write intent → call → write result
process dies at step 7 of 12
survives: 6 completed steps with results, 1 step in
state "intended, outcome unknown"
resume: rebuild context from the log, resolve step 7 by
querying the tool with its idempotency key, continue at 8.
the difference is not sophistication. it is whether a durable
write happens on the same side of the network call as the effect.What this reframing buys you
Three things, immediately, and they are why the framing is worth the effort.
The failure list is already written. You do not have to imagine how an agent system fails. Loss, delay, reordering, duplication, partition, partial failure — the same six, in the same places, with the same mitigations. Reviewing an agent design with the same checklist you would use for any service call graph finds real bugs on the first pass.
The vocabulary is shared with the people who will operate it. "The tool call is at-least-once and the handler is idempotent on a caller-supplied key" is a sentence an SRE understands completely. "The agent sometimes does things twice" is not a design statement, and it does not get budget.
It tells you what not to build. Several problems that feel like they need agent-specific machinery — coordinating parallel workers, deciding when a task is done, resuming after a crash — have thirty-year-old solutions. Reaching for leader election, leases and a step log is unglamorous and correct, and [[multi-agent-coordination]] is that argument in full.
Key points
- An orchestrator is a client of several independent remote services; every arrow in the architecture is a network call with all the properties this domain teaches.
- Duplicate tool calls, stale memory, stale retrieval, partial worker completion, queue redelivery and provider outages are existing distributed-systems problems with existing answers.
- One property is genuinely different: the model fails by producing a wrong answer rather than by stopping, which is closer to a Byzantine fault than a crash fault.
- Therefore the model is not a trusted node: it holds no state, decides no completion, owns no lock, and its output is validated input rather than an applied result.
- The context window is a cache, not a checkpoint. If the process dies, only what was written to a store survives.
- The reframing buys a ready-made failure list, a vocabulary operators already speak, and a clear signal about which problems are already solved.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The caller sends a request with a deadline; the orchestrator adopts that deadline as a budget to spend across every downstream call.
- • The orchestrator assembles context from the memory store and the retrieval service — two remote reads, each possibly stale.
- • It calls the model provider, typically streamed, over a connection that may break mid-response.
- • The model returns content or a request to invoke a tool; the orchestrator validates the request rather than trusting it.
- • The orchestrator records an intent to call, invokes the tool, and records the outcome — the durable pair that makes recovery possible.
- • It appends the result to context and loops, or dispatches worker agents and waits for a fan-out that may be partial.
- • A termination condition — a completion state written by ordinary code, a budget cap, or a deadline — ends the loop.
- • The model call times out with the request possibly served and certainly billed.
- • The stream breaks partway through a tool-call block, leaving a syntactically incomplete instruction.
- • A tool executes and its response is lost, so the orchestrator cannot tell whether the effect happened.
- • The memory store serves a stale replica and the agent proceeds on outdated facts.
- • The retrieval index lags behind the source of truth and grounds the answer in superseded content.
- • A worker agent dies mid-task and nothing observes it, because nothing was waiting on a lease.
- • The queue redelivers a step whose model call was simply slow, so two orchestrators run the same step concurrently.
- • The provider rate-limits the whole fleet at once — a correlated failure that no per-agent retry policy improves.
- • Duplicate side effect: a customer receives the same email twice, or is charged twice. The operator sees two successful tool invocations with different call ids and no error anywhere in the trace.
- • Stuck task: a workflow sits in
runningforever because the process that owned it died and nothing had a lease to expire. The observable is a queue of tasks whose age exceeds any plausible completion time, with zero error rate. - • Confidently stale answer: the agent cites a document that was updated an hour ago. There is no error, no exception and no failed request — only a wrong answer, discoverable through evaluation rather than monitoring.
- • Fleet-wide stall on provider degradation: model p99 rises from 4 s to 40 s, every orchestrator holds its worker and its connection for the full duration, and the pools saturate. The operator sees the *orchestrator* failing, and the actual cause is one hop away.
- • Retry amplification into a rate-limited provider: each agent retries three times, subagents retry too, and the effective request rate multiplies by an order of magnitude nobody configured. The observable is a 429 rate that will not come down after the trigger has passed.
- • Cost spike with no completions: token spend per task doubles while the completion rate stays flat — the signature of a loop retrying something it cannot see the result of.
- • The agent loop itself requires none — a single orchestrator running a single task is not a coordination problem, which is why simple agents work fine and why complexity should be earned.
- • Coordination appears at exactly three points: two orchestrators may run the same task (needs ownership), two agents may write the same state (needs a single writer or a merge), and a task must be declared complete (needs one authoritative writer, never an opinion).
- • Each coordination point costs a round trip to a durable store and a decision about what happens when that store is unavailable.
- • The model participates in none of it. Coordination is between processes; the model is a function those processes call.
- • Anything written to the durable store before the failure remains; anything held only in context is gone.
- • Side effects already executed by tools remain executed — there is no rollback across a service boundary, which makes an agent workflow a saga, see
[[sagas]]. - • The task’s state is whatever the step log says, which may include steps whose outcome is genuinely unknown; that ambiguity has to be represented rather than collapsed to success or failure.
- • Read paths generally continue to work, so an agent can appear healthy while operating on stale grounding.
- • Detect: alert on task age and on steps stuck in an unresolved state, not only on error rates. The characteristic agent failure produces no errors.
- • Contain: cap budgets — tokens, tool calls, wall-clock, subagent depth — so a failure that produces no error still terminates. This is
[[load-shedding]]for a workload that cannot otherwise stop itself. - • Recover: resume from the step log, resolving unresolved steps by querying the tool rather than by re-executing blindly.
- • Reconcile: compare the orchestrator’s record of side effects against the tool services’ own records; the gap is the duplicate-or-missing set.
- • Verify: check task outcomes against expectations, which for this workload means evaluation rather than health checks — a structurally correct task can still be wrong.
- • One correlation id spanning the entire workflow, propagated into every model call, tool call, memory write and subagent — without it,
[[correlating-distributed-logs]]is impossible and every incident is archaeology. - • Per-step latency broken down by hop: model, tool, memory, retrieval. Most "the agent is slow" reports resolve to one hop.
- • Duplicate tool invocations per idempotency key, which should be non-zero and stable; a rising rate means retries are increasing upstream.
- • Task age distribution and the count of tasks in unresolved steps — the leading indicator for stuck work.
- • Tokens and cost per completed task, which is the amplification signal: it rises before the failure becomes visible any other way.
- • Any agent system with side effects, where a duplicate or a lost step has a real-world cost.
- • Any system with more than one process able to act on the same task — which includes a single agent deployed on two replicas.
- • Reviews and incident analysis, where the mapping turns vague symptoms into named, already-solved problems.
- • Deciding what to build: the framing consistently shows that the needed mechanism is ordinary and already exists.
- • A single-process, read-only agent — a summariser, a classifier — carries none of this weight, and imposing step logs and idempotency keys on it is pure overhead.
- • Early prototypes, where durability machinery slows the loop that is teaching you what the product should be. Add it when side effects appear, which is the real threshold.
- • When the framing is used to justify infrastructure the workload does not need: not every agent needs a queue, a lease manager and a saga coordinator, and
[[when-not-to-distribute]]applies here as strongly as anywhere.
- • A deterministic pipeline with the model used only for the steps that need judgement — fewer arrows, fewer failure modes, and far easier to reason about. Agentic Engineering’s
pipeline-patternis the shape. - • A single process with no external tools and no persistence, for tasks short and safe enough that a crash simply means retry the whole thing.
- • A durable workflow engine, which supplies the step log, retries and timers as infrastructure rather than as code you maintain.
- • Synchronous request-response with a human in the loop for the effectful step, which replaces a hard correctness problem with an approval — often the right trade early on.
An agent system is a distributed system. Label every arrow.
- orchestrator process — holds the loop, the plan and the step log. Everything it has not written down dies with it.
- model provider — a remote service, reached over a network, with its own capacity limits
- tool service — where the side effects actually happen
- memory store — shared mutable state, usually with no concurrency control
- worker process — a second orchestrator with its own lossy copy of the same task state
What people believe, and what is true
Agent reliability is a prompting problem.
Duplicate charges, lost tasks and stale grounding are caused by network calls failing, not by wording. No prompt makes a tool call idempotent or a crashed process resume.
The framework handles retries, so we are covered.
A framework retrying a call it did not hear back from is precisely how a side effect happens twice. Retry without idempotence is a bug generator, and the framework cannot know which of your tools are safe to repeat.
The conversation history is the state.
It is an in-memory cache on one machine. If the process dies it is gone, and a second replica never had it. State is what you wrote down.
Multi-agent systems are more robust because work is spread out.
Spreading work adds coordination, partial failure and ownership questions. Robustness comes from durable state and idempotent effects, and a single agent with both beats five agents with neither.
The model can decide whether the task is finished.
Completion is a state transition in a durable store, written by ordinary code from evidence. A model saying "done" is a claim to be checked, and treating it as authoritative is how tasks are marked complete without their effects having happened.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
An orchestrator calling a model, tools, memory and retrieval is a client of four remote services. Everything this domain teaches about remote calls applies unchanged — plus one twist: the model can fail by being wrong rather than by stopping.
Practical
Draw your own architecture and label each arrow with its failure mode and the deadline it consumes. Then answer two questions: what survives if this process dies right now, and what happens if this tool call is executed twice. Those two answers determine almost everything else — the step log, the idempotency keys, and whether the loop needs a queue behind it at all.
Advanced
The sharpest way to state the architecture is that an agent loop is an *unreliable orchestrator of non-transactional side effects*, which makes it a saga executor whose plan is generated at runtime rather than written in advance. That single sentence predicts most of the design: it needs a durable step log because the plan is not known up front, it needs compensations rather than rollback because effects cross service boundaries, it needs idempotency keys because retries are unavoidable, and it needs a termination condition external to the plan because the planner cannot be trusted to stop. What it does *not* need is anything invented for agents specifically — [[saga-orchestration]] describes the same machine.
Apply it
- 🔧 Take one agent workflow and write, for each step, what happens if the call times out and whether the effect is safe to repeat.
- 🔧 Identify which parts of your agent’s state are in the context window and which are durable, then move one thing across the line.
- ⚡ An agent completes eight of twelve steps, the pod is evicted, and the task is retried by the queue. Describe every side effect that happens twice and what would have prevented it.
- 💬 Draw an agent architecture and label every network call with what happens when it times out.
- 💬 If the orchestrator process dies mid-task, what survives? Walk me through your durability boundary.
- 💬 Why is a model a different kind of component from a database, in failure terms?
- 💬 The model provider starts returning 429s. Trace what happens across a fleet of a thousand concurrent agents.