Most teams building on language models measure quality the way a benchmark paper does. They assemble a few hundred examples, score them, and get a number. The number goes in a spreadsheet next to last week’s number. If it went up, ship.

This is a strange way to run an engineering process, and it becomes obvious if you apply it to anything else. Imagine a test suite that returned “94% of your code is working” and no indication of which part. Nobody would accept that from a unit test framework. Yet it is the standard reporting format for model quality, and it produces exactly the outcome you would expect: teams ship regressions they cannot see, because the aggregate moved in the right direction while a specific capability quietly broke.

The reframe that fixes most of this is small. An eval is not a score. It is a test suite. Everything useful follows from taking that seriously.

A case that fails specifically

The single highest-value change is to make every case carry a label describing what it is testing, so a failure names a capability rather than moving a decimal. “Accuracy fell from 0.94 to 0.91” prompts a shrug. “All six multi-currency refund cases now fail” prompts a fix, and it prompts it in the right file.

# Every case is a named behavior, not a row in a scorecard.
CASES = [
  dict(id="refund.multi_currency",       tags=["refund","money","i18n"]),
  dict(id="refund.partial_after_30d",    tags=["refund","policy"]),
  dict(id="escalate.abusive_language",   tags=["safety","routing"]),
  dict(id="refuse.no_order_ownership",   tags=["authz","refusal"]),
  dict(id="tone.terse_when_asked",       tags=["style"]),
]

# Report by tag, not just in total. This is the whole trick.
#
#   refund   12/12   =
#   authz     8/8    =
#   safety   14/15   -1   <- regression, and you know where
#   style    19/22   +2
#
# The aggregate here went UP. Ignore the aggregate.

Note what the sample output shows: the total improved while a safety capability regressed. That is the ordinary case, not an edge case. Aggregates hide exactly the failures you most need to see, because the categories you care about most are usually the smallest.

Where the cases come from

Not from a brainstorm. A set written in a meeting encodes what the team imagined users would do, which is reliably not what they do. Three sources produce cases that earn their place.

• Every incident becomes a case, permanently. This is the discipline that compounds. Something went wrong in production, someone fixed it, and the fixed behavior is now pinned. Without this step the same failure returns two model versions later and nobody recognizes it.
• Sampled real traffic, stratified. Random sampling gives you the head of the distribution, which is where the system already works. Stratify: by intent, by length, by language, by whether a tool was called, by whether the user came back and rephrased. That last signal is the cheapest quality metric most teams already have and never look at.
• Adversarial cases from someone who is not on the team. Whoever wrote the prompt cannot see its blind spots, because the blind spots are made of their own assumptions.

Size matters less than most people assume. Two hundred well-stratified cases with named capabilities will tell you more than five thousand scraped ones, and you can actually read the failures, which is the only way the suite improves.

Grading, in descending order of trustworthiness

Use the cheapest reliable grader for each case, and be honest about which tier you are in. Teams routinely reach for a model judge on cases that a string comparison would settle exactly.

• Programmatic assertions. Did it call the right tool with the right arguments? Is the JSON valid against the schema? Is the total correct? Deterministic, free, and unambiguous. Push as many cases as possible into this tier: often it is far more than expected, because most real requirements are structural rather than stylistic.
• Reference-based checks. A known-good answer with a tolerance. Works for extraction, classification, and summarization against a rubric of required facts. Cheap, and it catches most silent breakage.
• Model as judge. Necessary for genuinely open-ended output. Also the tier with real failure modes, covered below.
• Human review. The ground truth, and the scarce resource. Spend it calibrating the judge and reading failures, not on grading volume.

The judge has biases, and they are measurable

A model grading model output is a useful instrument with known distortions. They are not mysterious, and each has a countermeasure.

• Position bias. Shown two answers, judges favor one slot. Run every pair in both orders and keep only agreements; disagreement is itself the signal that the two answers are equivalent.
• Length bias. Longer answers score higher, roughly regardless of content. If your change made outputs more verbose, expect a free and entirely fake improvement.
• Self-preference. A judge tends to prefer text from its own family. If the judge and the system under test are the same model, the eval is partly measuring family resemblance.
• Absolute scores drift. Asking for a 1-to-5 rating gives numbers that are not stable across judge versions, so this quarter’s 4.2 is not comparable to last quarter’s 4.2.

The single change that removes most of this: stop asking for a score and ask for a comparison against the current production output. Relative judgements are far more stable than absolute ones, and “is the new one better than what we ship today” is the actual question anyway.

def pairwise(case, baseline_out, candidate_out, judge):
    # Both orders, then require agreement. Disagreement means "tie",
    # which is information, not noise.
    a = judge(case.prompt, first=baseline_out,  second=candidate_out)
    b = judge(case.prompt, first=candidate_out, second=baseline_out)
    if a == "first"  and b == "second": return "baseline"
    if a == "second" and b == "first":  return "candidate"
    return "tie"

# Calibrate before trusting it: have a human label 60 pairs, then measure
# how often the judge agrees. Below about 80% agreement the judge is not
# an instrument, it is a random number with a rationale attached.

That calibration step is the one almost everyone skips, and it is what separates an eval you can act on from a number that merely exists. It costs an afternoon and it tells you whether the rest of the harness means anything.

Wiring it into the gate

An eval that runs when someone remembers is not a test suite, it is a research activity. The value appears when it blocks a merge. That requires deciding, in advance, what counts as a failure, and the useful rule is not a threshold on the average.

# Two gates, different severities. Both run on every change to prompts,
# retrieval config, tool schemas, or model version. Yes, prompts are code.

# 1. Hard gate: no named capability may regress at all.
for tag, (was, now) in by_tag.items():
    assert now >= was, f"regression in {tag}: {was} -> {now}"

# 2. Soft gate: incident-derived cases never regress, ever.
for c in cases_from_incidents:
    assert results[c.id].passed, f"previously fixed bug is back: {c.id}"

# Temperature 0 is not determinism. Run the suite 3x and treat any case
# that flips as flaky: quarantine it, do not let it gate, and go find out
# why the behavior is unstable. Flaky evals train people to ignore red.

Treating prompt files as code, with review and a gate, is the change most teams resist and most benefit from. A prompt edit is a production behavior change with no type system and no compiler. It deserves more scrutiny than a refactor, not less.

What the harness will not tell you

Worth stating so the suite does not become falsely reassuring. It measures the behaviors you thought to write down. It says nothing about the ones you have not imagined, which is where the next incident lives. And a suite that has passed unchanged for three months is not evidence of stability, it is evidence that it stopped tracking what your users are doing.

The health metric for an eval suite is not its pass rate. It is how many cases were added last month, and where they came from. A suite fed by production incidents and sampled traffic gets sharper as the system ages. One written once and frozen becomes decoration, and it will keep reporting green right up to the point someone reads a support queue and discovers otherwise.