Deploying an Agent
Agent code, prompts, models and tool definitions all go through CI and an evaluation gate before they become an artifact, a deployment and then traffic.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does a delivery pipeline look like when the behaviour you are shipping is probabilistic?
The pipeline you already have gates on tests that are deterministic: the same input gives the same output, and a pass means a pass. An agent gives different outputs for the same input, and the thing that most often changes its behaviour — a prompt, a model version, a tool description — is not code and does not go through the pipeline at all.
Ship the agent like any other service, and treat the prompt as configuration. When behaviour needs adjusting, edit the prompt in a dashboard or a config store and it takes effect immediately — which is exactly what you want when the agent is saying something wrong to customers.
A prompt edit is a production behaviour change with no test, no review, no artifact and no record. The next incident starts with "when did the prompt change?" and nobody can answer.
- A prompt edit is a production behaviour change with no test, no review, no artifact and no record. The next incident starts with "when did the prompt change?" and nobody can answer.
- Fixing one bad case by adding a line to the prompt routinely breaks three cases nobody re-checked, because there is no suite that would have noticed.
- A model version change is treated as a dependency bump when it is the largest single behaviour change you can make. The same prompt against a different model is a different system.
- Tool descriptions are load-bearing: they determine when the model calls a tool and with what arguments. Editing one changes tool-selection behaviour across every request, with no test coverage at all (Tool Schemas owns why).
- Without a gate you learn about a regression from users, and users report the confident, plausible, wrong answers last.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The pipeline is the same shape as any other — code, CI, artifact, deploy, traffic — with one stage inserted that has no equivalent in deterministic delivery: evaluation.
- Evaluation is not a substitute for tests; it sits alongside them. Unit and integration tests still cover the deterministic parts — parsing, validation, retries, tool plumbing — and the eval stage covers behaviour (A Tool Call Is a Backend Call).
- Because outputs vary, an eval gate is statistical: run a fixed dataset, score it, compare aggregate scores against the current production version, and gate on the delta rather than on an absolute pass. The Agentic domain owns how those scores are constructed (Evaluating Agents: Testing Probabilistic Systems, Golden Datasets).
- That comparison only means something if the artifact is fully specified. The deployable unit is code plus prompt versions, model identifier, tool definitions and policies — the whole set, pinned together (Prompts and Models Are Deployables).
- Deterministic evaluators — schema validity, required tool called, forbidden tool not called, output parses, citation present — are cheap, stable and gate well. Model-graded quality scores are noisier and are better read as a trend across runs than as a pass/fail line (Deterministic Evaluators, LLM-as-Judge).
- The gate is a filter, not a proof. Real traffic contains inputs your dataset does not, which is why the stage after it is a canary rather than a full rollout (Canarying a Model or Prompt Change).
The path from a prompt edit to production traffic
Everything that determines behaviour enters on the left and cannot reach traffic except through the gate. That is the whole design: there is no side door from a prompt editor straight to production.
The evaluation stage is the only structural addition to an ordinary pipeline. Its output is a comparison against what is currently deployed, which is why the currently deployed version has to be evaluated too.
What each stage owes you
The stages are ordinary until the fourth. Read the failsBy column as the list of ways teams end up with an agent pipeline that runs without gating anything.
- 1Change
A pull request touching any behaviour input: code, prompt, model identifier, tool definition or policy.
fails by Prompts and model identifiers live outside version control, so most behaviour changes never become a pull request.
evidence A diff exists for the last behaviour change in production, with an author and a reviewer.
- 2CI checks
Deterministic tests over the plumbing — schema validation, tool argument construction, retry and timeout handling, error paths.
fails by Everything is tested through the model, so failures are slow, expensive and ambiguous.
evidence Tool-calling paths covered by tests that do not call a model at all.
- 3Eval run
Executes a versioned dataset against the candidate and against the deployed version, and scores both.
fails by Only the candidate is scored, so there is no baseline and the result is an absolute number nobody can interpret.
evidence Paired scores with a delta, and the dataset version recorded.
- 4Gate
Blocks on deterministic and safety criteria; reports graded quality as a delta for human judgement.
fails by Gating on a noisy aggregate score, which flaps and gets re-run to green.
evidence A blocked change in recent history, and a documented rule for what blocks versus what informs.
- 5Artifact
Produces one immutable, addressable unit containing code, prompts, model identifier, tool definitions and policies (Tags Versus Digests).
fails by Prompts are fetched at runtime from a mutable store, so the artifact does not determine behaviour.
evidence A running instance reports the same versions the artifact declares.
- 6Canary
Routes a small share of real traffic to the candidate and evaluates quality, cost, latency and safety (Canarying a Model or Prompt Change).
fails by Only error rate and latency are watched, so a quality regression passes untouched.
evidence A canary report naming all four signals, not two.
- 7Rollout and observe
Expands to full traffic, with online evaluation on sampled requests and a kill switch available.
fails by Observation stops at rollout, so drift after deployment is invisible (Regression Gates and Online Evaluation).
evidence A quality signal on the operational dashboard alongside error rate and latency (Dashboards an Operator Can Act On).
The stage most often missing is not the eval run. It is the baseline: without scoring the deployed version at the same time, the gate produces a number rather than a decision.
The one-line prompt fix
This is where the discipline is actually decided, because the pressure is real and the change genuinely is small. The agent said something wrong to a customer an hour ago and someone has a fix ready.
The sentence is added and takes effect on the next request. The reported case is fixed and everyone moves on. There is no diff, no review and no eval run. Three weeks later the agent is answering a different class of question worse, nobody connects it to this edit, and the prompt in version control no longer matches the one in production.
The sentence becomes a pull request. The reported case is added to the dataset as a new eval case. The gate runs candidate and production side by side: the new case passes and the regression set is unchanged. The artifact ships to a canary, then to full traffic, in the time the pipeline takes.
Both fix the reported case. Only one tells you what the sentence did to every other case, and only one leaves a record that lets the next person understand why that sentence is there. If the pipeline is too slow to use during an incident, the pipeline is the thing to fix — a fast path with the safety subset still gating is the right compromise, not a console edit.
How to do it properly
Most important first.
- Make prompts, model identifiers, tool definitions and policies part of the built artifact, so they cannot reach production except through the pipeline (Build Once, Deploy Many).
- Run evals in CI on every change to any of those inputs — not only on application code. A prompt-only pull request must run the same gate as a code change.
- Gate on comparison with the currently deployed version, not on an absolute score. "No worse than production on the regression set, and better on the cases this change targets" is a claim you can test.
- Put deterministic evaluators on the blocking path and treat model-graded scores as advisory unless you have calibrated them against human judgement.
- Record the eval run — dataset version, scores, model identifier, prompt versions — as an attestation attached to the artifact digest (Build Provenance).
- Keep an explicit safety subset of the dataset that gates independently: cases where the wrong behaviour is harmful rather than merely worse. A quality improvement does not buy a safety regression.
- Continue evaluating after deployment on sampled production traffic, because the offline dataset ages (Regression Gates and Online Evaluation).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
An unevaluated prompt, model or tool change reaches every request the agent handles at once. What contains it is the eval gate before rollout, a canary during it, and a kill switch after (The Agent Kill Switch).
What can go wrong
- The eval dataset overfits: it accumulates the cases the team already fixed, so it passes everything and catches nothing new.
- Score variance mistaken for regression, producing a flapping gate that people learn to re-run until it goes green — the same dynamic as flaky tests, with the same cost (Flaky Tests).
- Evals gated on the agent code path only, while prompts ship through a separate config route that bypasses the pipeline entirely.
- A judge model that is itself changed without notice, moving every score at once and making historical comparison meaningless.
- Eval runs slow enough that people skip them for "small" prompt edits — which are precisely the edits that change behaviour most per character (CI Is a Feedback System).
- A gate on aggregate score that hides a large regression in a small, important segment.
- "Evals replace tests." They cover different things. Deterministic plumbing still needs deterministic tests, and an eval suite over an agent whose tool calls are untested is testing the wrong layer.
- "The eval suite passed, so the change is safe." It is safe on the distribution you assembled. Real traffic is a different distribution, which is why a canary follows (Canarying a Model or Prompt Change).
- "Prompts are configuration, so they can be edited live." Configuration is a production input with a blast radius too; that argument proves less than it appears to (A Config Change Is a Production Change).
- "A newer model is an upgrade." It is a behaviour change. Better on public benchmarks says little about better on your dataset, your prompts and your tools.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A prompt change and a code change follow the same path, and you can point at the pull request for the last prompt edit in production.
- Every deployed artifact has an eval run attached, with dataset version and scores.
- The deployed model identifier, prompt versions and tool definition versions can be read from a running instance, not inferred.
- A regression caught by the gate exists in recent history — a gate that has never blocked anything is not yet evidence of anything.
- Rollback is the previous artifact, which restores code, prompts, model identifier and tool definitions together. Reverting only the prompt leaves an untested combination running.
- Where the model identifier refers to something the provider updates underneath a stable name, rollback does not restore the previous behaviour. Pin to the most specific identifier available and treat the provider-versioned name as an input you do not control (Dependency Pinning).
- Rolling back an agent does not undo the actions it already took. Side effects through tools — messages sent, records written, refunds issued — need their own compensation path, which is a design problem the Agentic domain owns (Idempotency).
- Automate the eval run, the comparison against production and the attestation. None of that should depend on someone remembering.
- Automate blocking on deterministic and safety evaluators, where the criterion is unambiguous.
- Keep the judgement human where the score is a model's opinion: a 3% drop on a graded quality metric is a conversation, not a build failure.
- Keep dataset curation human. A dataset assembled automatically from production failures drifts toward whatever failed most recently, which is not the same as what matters most.
- The eval stage adds latency and cost to every change, including one-line prompt fixes, and it is at its most annoying during an incident when someone wants to patch a prompt immediately.
- Good datasets are expensive to build and to maintain, and they decay: an unmaintained dataset gives false confidence, which is worse than none.
- Gating on comparison means you need to keep evaluating the production version too, which roughly doubles eval cost.
- A strict gate slows iteration on exactly the fast-moving part of the system, which is a real cost to weigh against the blast radius of the change (Blast Radius: If This Is Wrong, How Much Does It Affect?).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe stage order — code, checks, evals, artifact, canary, traffic — holds for any system whose behaviour depends on a model, regardless of provider or framework. What varies is how scores are computed and how much variance they carry.
- SIMPLIFIEDPresented as one linear gate. Mature setups run several: fast deterministic checks per commit, an expensive graded suite before promotion, and continuous online evaluation on sampled production traffic.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — how a statistical gate is calibrated, and what a confidence interval on a pass rate is worth as a release criterion.