Regression Gates and Online Evaluation
Offline evals gate changes in CI; online evaluation samples production traces, runs A/B or shadow comparisons and closes the feedback loop — and both need enough samples to separate signal from noise.
CI gating with offline evals
Treat the eval suite as a test suite in CI. On every change to a prompt, tool schema, retrieval config or model pin, run the deterministic evals on the full dataset and the judge-based evals on a fixed subset, then compare to the baseline stored for the main branch. Gate on no regression beyond noise on the headline metrics and on hard limits for safety and budgets: zero safety violations, p95 steps under the cap, cost per task under the cap.
Keep the CI run fast enough to be run on every pull request — a few minutes. Put the slow, expensive, full end-to-end suite on a nightly schedule and on release branches. Store every run's metrics with the dataset, prompt, model and evaluator versions so the trend chart is honest.
- Per PR: deterministic evals on all cases, judge evals on a fixed subset, compare to baseline.
- Nightly / release: full end-to-end suite with N repetitions per case.
- Block on: regression beyond the confidence interval, any severe safety violation, budget breaches.
Is B better than A? A small worked example
Version A passes 82 of 100 cases; version B passes 86 of 100. Is B better? The standard error of a proportion is sqrt(p(1−p)/n); for p≈0.84 and n=100 that is about 0.037, so each rate has a 95% interval of roughly ±7 points. The difference of 4 points is well inside that. A two-proportion z-test gives z ≈ 0.78, p ≈ 0.44 — no evidence B is better. You would need roughly n ≈ 800 cases per arm to detect a 4-point difference at 80% power, or a much larger effect at n=100.
Two things help more than raw size. First, pair the comparison: both versions run the same cases, so look at the cases where they disagree. If B passes 12 cases A failed, and A passes 8 cases B failed, a sign test on 20 discordant pairs gives p ≈ 0.50 — still nothing. If it were 18 versus 2, p ≈ 0.0004 — real. Paired analysis removes the variance from case difficulty. Second, repeat runs: 5 runs per case turns a single bit into a pass fraction and shrinks the per-case noise.
The practical rules: report confidence intervals, not point estimates; use paired analysis when both versions ran the same cases; and when the interval includes zero, say "no detectable difference" rather than "B is better".
1from math import comb2 3def sign_test(b_wins: int, a_wins: int) -> float:4 """Two-sided p-value that B and A are equally likely to win a discordant case."""5 n, k = b_wins + a_wins, max(b_wins, a_wins)6 tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n7 return min(1.0, 2 * tail)8 9print(sign_test(12, 8)) # ~0.50 -> no evidence10print(sign_test(18, 2)) # ~0.0004 -> B is betterOnline evaluation: A/B, shadow and sampling
Offline evals only cover cases you thought of. Online evaluation measures the system on real traffic. A shadow run executes version B on a copy of production requests without showing its output to users (and with side-effecting tools stubbed), then compares B's traces to A's. It is safe, catches distribution shift, and is the right first step for agent changes with side effects. An A/B test routes a fraction of users to B for real and compares outcome metrics — task completion, escalation rate, thumbs-down rate — with the same statistics as above, plus guardrail metrics that abort the test on a safety or cost breach.
Independently of experiments, sample production traces continuously: a random 1–5% plus everything that hit an error, a step cap, a budget limit or negative feedback. Score the sample with the same evaluators used offline (deterministic ones on all, judge on a subset) and push the scores to the dashboard described in Logging, Metrics and Alerts. This is how you see quality drift from model updates, new user behaviour or data changes before it becomes an incident.
Closing the feedback loop
Feedback signals — thumbs up/down, "escalated to human", user rephrased the same question, task abandoned — are noisy individually but strong in aggregate, and they point to exactly the cases the golden dataset is missing. Route every negative-feedback trace into a review queue; the reviewer labels the correct outcome and it becomes a golden case. Within a few weeks the dataset reflects real failure modes rather than the author's imagination.
The loop only works if it is cheap to promote a trace to a case, so build that path early: one click from trace to draft case, with the input, context and trajectory pre-filled.
Key points
- CI runs offline evals on every change and blocks on regression beyond noise, safety violations and budget breaches.
- Point estimates lie: 82% vs 86% on 100 cases is not evidence. Report confidence intervals.
- Paired analysis on discordant cases and repeated runs give far more power than adding raw cases.
- Shadow runs test agent changes on real traffic without user exposure; A/B tests measure real outcomes with guardrails.
- Continuously sample production traces and score them with the offline evaluators to detect drift.
- Negative feedback traces become golden cases; make that path one click.
When to use — and when not to
- Every change to prompts, tools, retrieval, or model version.
- Before exposing a new agent version to users, especially with side-effecting tools (shadow first).
- When the vendor releases a new model and you need to decide whether to migrate.
- Do not A/B an agent with unstubbed side effects before a shadow run has shown it is safe.
- Do not declare a winner from a 4-point difference on 100 cases.
- Do not run the full end-to-end suite on every commit if it takes 40 minutes; split it.
Failure modes
- Gate compares point estimates; a noisy 2-point drop blocks a good change or a real 5-point drop passes.
- Shadow run skipped; B issues real refunds during the "test".
- A/B guardrails missing; a cost regression runs for two weeks (
cost-explosion-after-launch). - Production sampling only takes random traces and never the failed ones; drift is invisible.
- Feedback collected but never promoted to the dataset; the same failure recurs across versions.