Evalsevalsassertionsschematrajectorybudgets

Deterministic Evaluators

Most of what matters about an agent run can be checked by code — schemas, tool calls, trajectory shape, budgets — and those checks are exact, free and never flaky; use them before reaching for a judge.

Interview question
Progress

Why code first

A deterministic evaluator is a function from (trace, case) to a score with no model in the loop. It runs in microseconds, costs nothing, gives the same answer every time and fails with a message that says exactly what went wrong. Every criterion you can move from a judge into code makes the eval suite cheaper, faster and more trustworthy.

Agents are unusually amenable to this because so much of their behaviour is structured: tool names, JSON arguments, step counts, end state in a database. The final prose is the only part that is hard to check, and even there, references, numbers and required phrases are checkable.

A catalogue of checks

The checks below cover most agent evals. Compose them per case; each returns pass/fail plus a reason.

  • Exact / normalised match: final answer equals reference after trimming, casing and whitespace normalisation. For numbers, parse and compare with tolerance. Use for closed-form answers only.
  • Schema validation: final output and every tool call's arguments validate against their JSON Schema (see Tool Schemas, Structured Outputs). Catches the most common regression after a model change.
  • Tool-call assertions: required tool was called; forbidden tool was not; a specific argument equals an expected value; a tool was called at most once (idempotency, see Idempotency).
  • Trajectory checks: ordering constraints (lookup_order before issue_refund), no repeated identical calls (a loop signature), termination via the finish action rather than the step cap.
  • Regex / JSON checks: output contains an order id matching ORD-\d{6}, contains no email address or card-number pattern, parses as JSON with required keys.
  • Cost and step budgets as assertions: steps <= 8, tokens <= 20_000, cost_usd <= 0.05, latency_ms <= 6_000. Budgets in the eval suite catch efficiency regressions before they reach the invoice (see Budgets, Limits and Termination).
  • End-state checks: query the sandboxed database or filesystem after the run; the refund row exists with the right amount, the file was written, nothing else changed.

A small evaluator harness

A harness needs three things: a registry of named checks, a way for each case to say which checks apply with what parameters, and a runner that collects results per case and aggregates. Sixty lines of Python is enough to start; frameworks can come later.

Composable checks over a trace. Each check returns (passed, reason); the runner aggregates.
1import json, re
2from dataclasses import dataclass
3 
4@dataclass
5class Result:
6    name: str
7    passed: bool
8    reason: str = ""
9 
10def tool_calls(trace):
11    return [s for s in trace.spans if s.kind == "tool"]
12 
13def check_called(trace, case, tool, **_):
14    ok = any(s.name == tool for s in tool_calls(trace))
15    return Result("called:" + tool, ok, "" if ok else f"{tool} never called")
16 
17def check_not_called(trace, case, tool, **_):
18    ok = all(s.name != tool for s in tool_calls(trace))
19    return Result("not_called:" + tool, ok, "" if ok else f"forbidden {tool} was called")
20 
21def check_arg(trace, case, tool, key, expected, **_):
22    for s in tool_calls(trace):
23        if s.name == tool:
24            got = s.input.get(key)
25            return Result(f"arg:{tool}.{key}", got == expected, f"got {got!r}, want {expected!r}")
26    return Result(f"arg:{tool}.{key}", False, f"{tool} not called")
27 
28def check_order(trace, case, before, after, **_):
29    names = [s.name for s in tool_calls(trace)]
30    ok = before in names and after in names and names.index(before) < names.index(after)
31    return Result(f"order:{before}<{after}", ok, "" if ok else f"sequence {names}")
32 
33def check_no_repeat(trace, case, **_):
34    seen, dup = set(), None
35    for s in tool_calls(trace):
36        key = (s.name, json.dumps(s.input, sort_keys=True))
37        if key in seen: dup = key; break
38        seen.add(key)
39    return Result("no_repeat", dup is None, f"repeated {dup}" if dup else "")
40 
41def check_regex(trace, case, pattern, present=True, **_):
42    found = re.search(pattern, trace.final_output or "") is not None
43    return Result(f"regex:{pattern}", found == present, f"found={found}")
44 
45def check_budget(trace, case, steps=None, cost_usd=None, **_):
46    n = len([s for s in trace.spans if s.kind == "llm"])
47    c = sum(s.cost_usd for s in trace.spans)
48    ok = (steps is None or n <= steps) and (cost_usd is None or c <= cost_usd)
49    return Result("budget", ok, f"steps={n} cost={c:.4f}")
50 
51CHECKS = {f.__name__.removeprefix("check_"): f for f in
52          [check_called, check_not_called, check_arg, check_order, check_no_repeat, check_regex, check_budget]}
53 
54def evaluate(trace, case):
55    # case.checks: [{"type": "called", "tool": "lookup_order"}, {"type": "budget", "steps": 8}, ...]
56    results = [CHECKS[c["type"]](trace, case, **{k: v for k, v in c.items() if k != "type"})
57               for c in case.checks]
58    return {"case": case.id, "passed": all(r.passed for r in results), "results": results}
59 
60def summarize(evaluations):
61    n = len(evaluations)
62    return {"pass_rate": sum(e["passed"] for e in evaluations) / n, "n": n}

Handling non-determinism in the run itself

Deterministic evaluators do not make the agent deterministic. The same case can pass in one run and fail in the next. Run each case N times (3–5 is typical) and report the pass fraction per case rather than a single bit; a case at 3/5 is a real signal about instability that a single run would hide. Set temperature=0 where the provider honours it, fix the model version, and mock external tools with recorded responses so the only variance left is the model's.

  • Record tool responses once; replay them in evals so a flaky search API does not fail your suite.
  • Pin model identifiers; "latest" aliases silently change the distribution.
  • Report per-case pass fraction across N runs; alert on cases that flip between versions.

Key points

  • Deterministic checks are exact, free and never flaky — exhaust them before adding a judge.
  • Agent behaviour is mostly structured: tool names, arguments, order, step counts, end state.
  • Budgets (steps, tokens, cost, latency) belong in the eval suite as assertions.
  • A harness is a registry of named checks plus per-case configuration; start tiny.
  • Run cases N times and report pass fraction; mock tools with recorded responses.
  • Pin model versions so the only variance is the model's sampling.

When to use — and when not to

Use it when
  • Any criterion expressible as a predicate on the trace or the end state.
  • CI gates where speed and zero variance matter (Regression Gates and Online Evaluation).
  • Unit-level evals for routers, tool selection and argument construction.
Avoid it when
  • Open-ended quality criteria (faithfulness, tone) — use LLM-as-Judge with calibration.
  • Exact match on free text; normalise or use a reference-based judge instead.
  • As a substitute for end-state checks when the tool has side effects — check the sandbox, not just the call.

Failure modes

  • Exact string match on a free-text field fails on harmless formatting differences.
  • Tool-call assertion checks the name but not the arguments; a wrong refund amount passes.
  • Live external tools in the eval loop make the suite flaky for reasons unrelated to the agent.
  • No step budget in the suite, so a regression to 20-step runs is discovered on the invoice.
  • Checks written against the current output instead of the requirement, freezing bugs as expected behaviour.
← LLM-as-JudgeRegression Gates and Online Evaluation →