Evalsevalsjudgerubricsbiascalibration

LLM-as-Judge

Using a model to score outputs scales evaluation to criteria code cannot express, but the judge has biases of its own and is only trustworthy after calibration against human labels.

Interview question
Progress

When a judge is the right evaluator

Some criteria have no deterministic check: is this summary faithful to the source, is the tone appropriate for a grieving customer, does the explanation actually answer the question asked. For these, an LLM given the output, the context and a rubric can produce a score at a fraction of human cost. That is the whole value proposition — scale on criteria that need judgment.

A judge is not the right evaluator when a deterministic check exists. Whether the JSON validates, whether the refund tool was called with the right amount, whether the answer contains the reference number — code answers these exactly, for free, with zero variance. Use the judge only for the residual (see Deterministic Evaluators).

  • Acceptable: faithfulness to context, rubric-based quality, instruction following on open-ended tasks, comparing two responses.
  • Not acceptable: anything schema-checkable, numeric correctness, tool-call assertions, safety-critical decisions without human review.

Rubrics, absolute and pairwise scoring

A judge without a rubric is a vibe check. Write the rubric as a small set of binary criteria with examples of pass and fail ("Cites the order number from context: yes/no"; "Contains no claim absent from the retrieved documents: yes/no"). Binary criteria are far more consistent than a 1–10 scale, and a score becomes the count of criteria met. Ask the judge for a short justification before the verdict — it improves consistency and gives you something to inspect when the score looks wrong.

Absolute scoring grades one output against the rubric. It is what you need for a metric over time, but judges are poorly calibrated on absolute scales: the same output might get 7 today and 8 tomorrow. Pairwise scoring shows the judge two outputs (A from the old version, B from the new) and asks which better satisfies the rubric. Judges are much more reliable at relative comparison, which makes pairwise the better choice for "is B better than A" decisions. Its cost is that it produces a win rate, not a standalone quality number.

Pairwise judge with position swap. Both orderings are scored; disagreement becomes a tie.
1RUBRIC = """Criteria: (1) answers the question asked, (2) every claim is supported
2by CONTEXT, (3) states the order number, (4) no unsupported promises.
3Output JSON: {"reasoning": "...", "winner": "A" | "B" | "tie"}"""
4 
5def pairwise(judge, question, context, a, b):
6    def ask(x, y):
7        r = judge(f"{RUBRIC}\n\nQUESTION: {question}\nCONTEXT: {context}\n\nA: {x}\n\nB: {y}")
8        return json.loads(r)["winner"]
9    first = ask(a, b)                       # A first
10    second = ask(b, a)                      # B first; 'A' here means b
11    second = {"A": "B", "B": "A", "tie": "tie"}[second]
12    return first if first == second else "tie"

Judge biases

Judges have systematic biases that will show up in your metrics if you do not control for them.

  • Position bias: in pairwise mode, the first (or last) option wins more often regardless of content. Mitigate by scoring both orderings and treating disagreement as a tie, as above.
  • Self-preference: a model rates outputs from itself or its own family higher. Use a different model family as judge than the one being evaluated, or at least validate the judge on a sample where the source is hidden.
  • Length bias: longer, more elaborate answers score higher even when they add nothing. Put "concise; extra content is not rewarded" in the rubric and check whether score correlates with length on your data.
  • Verbosity of confidence: assertive wrong answers beat hedged right ones. Give the judge the reference or context and make faithfulness a criterion.
  • Rubric drift: editing the rubric changes every score. Version the judge prompt and model; re-score the baseline when either changes.

Calibration against humans

Before a judge metric enters a release gate, measure how well it agrees with people. Take 100–200 cases, have one or two humans label them with the same rubric, run the judge, and compute agreement — percent agreement for binary criteria, Cohen's kappa if you want to correct for chance, or a correlation for continuous scores. Agreement in the range of human–human agreement is the target; substantially below it means the rubric is ambiguous or the judge is not up to the task.

Re-calibrate whenever the judge model or rubric changes, and spot-check a random 5% of judge decisions on every eval run. A judge whose agreement with humans is unknown is producing numbers, not measurements.

Key points

  • Use a judge only for criteria code cannot check; deterministic evaluators come first.
  • Rubrics of binary criteria with justification-then-verdict beat 1–10 scales.
  • Pairwise comparison is more reliable than absolute scoring for A-vs-B decisions.
  • Control position bias by swapping order; control self-preference by using a different model family.
  • Calibrate against human labels (100–200 cases) before trusting a judge in a gate; re-calibrate when the judge changes.
  • Version the judge prompt and model with every metric.

When to use — and when not to

Use it when
  • Scoring faithfulness, tone or helpfulness on open-ended outputs at scale.
  • A/B comparisons between prompt or model versions via pairwise win rate.
  • Triage of production samples to find candidates for human review.
Avoid it when
  • Anything a schema, regex, exact match or tool-call assertion can check.
  • Safety-critical or compliance decisions without a human in the loop.
  • When you have not calibrated the judge and cannot say how often it agrees with people.

Failure modes

  • Judge from the same model family rewards its own style; the "improvement" is self-preference.
  • Position bias makes B win 65% of pairs with the content swapped; no order swap was done.
  • A 1–10 scale drifts between runs, producing phantom regressions (flaky-evals).
  • Rubric rewrites mid-quarter; the trend chart compares incompatible scores.
  • Judge scores confident hallucinations highly because the context was not provided to it.
← Golden DatasetsDeterministic Evaluators →