Agentic Distributed Systems
5 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
User to orchestrator to model provider, tool service, memory store, retrieval service and worker agents. Every one of those arrows is a network call between machines that fail independently. Nothing in this domain stops applying because the caller is a model — and one property gets worse, because the caller is non-deterministic and can decide to retry on its own.
Q · What do I already know about distributed systems that applies, unchanged, to an agent architecture?
An agent calls send_email. The result never reaches the model — the call timed out, the stream broke, the process restarted. On the next turn the model sees a tool call with no result and does the natural thing: it calls it again. This is exactly the timeout-ambiguity case, with one addition that makes it worse: the party deciding to retry is non-deterministic and may retry with different arguments.
Q · A side-effecting tool may have executed. How do I make the retry safe when the thing retrying is a model?
Who owns task state, who decides completion, can two agents execute the same task, how are conflicts resolved, and how is progress persisted. Five questions, and every one of them is an ordinary distributed coordination problem with a decades-old answer. Describing them as agents "negotiating" or "collaborating" is not a harmless simplification — it hides the engineering that has to happen.
Q · Two or more agents are working on the same task. Which process is allowed to act, and what happens when it stops responding?
The process holding a twelve-step workflow is evicted at step seven. The question is not whether it crashed — it will — but what the smallest durable record is from which the work can continue without repeating a side effect. An agent workflow with side effects is a saga: there is no rollback, only compensation, and the recovery design has to be built on that.
Q · The process died mid-workflow. What do I need to have written down to continue safely?
Non-determinism makes replay unreliable, so the debugging technique this domain leans on hardest is weakened. A retried model call may take a different path, so a retry is not a repeat. Context is lost between steps and the agent forgets what it did. And a tool error can be read as content, so a failure becomes an answer. None of these raise an exception.
Q · What goes wrong in an agent system that no error rate, latency graph or health check will show me?