How to Build LLM Evals That Survive Model Changes

A new model ships, the launch chart shows a taller bar, and someone asks whether you should switch. Without your own tests the honest answer is "we don't know". LLM evals are the fix: a small, repeatable suite built from your own tasks that turns "it feels better" into a number you can compare across models, prompts and price tiers. This guide covers how to define what good means, how to choose a grader, how to check an LLM judge before trusting it, and how to keep the suite useful as models change. For the public-benchmark side of the story, read how to read an AI benchmark claim.

What an eval is, and what it is not

An eval has two parts: inputs that represent the work, and a grader that scores each output. Anthropic's documentation frames the starting point as success criteria, and says good criteria are specific, measurable, achievable and relevant.^[1] Its own example of "specific" is replacing "good performance" with "accurate sentiment classification". Its example of "measurable" is a threshold such as "Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter."^[1]

OpenAI's guide describes the same idea as a workflow of three steps: define the task as an eval, run it with test inputs, then analyse the results and iterate on the prompt.^[2] It compares the approach to behavior-driven development, in which you specify how the system should behave before you build and test it.^[2]

An eval is not a leaderboard. The Holistic Evaluation of Language Models (HELM) project shows what a broad public evaluation looks like: seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) across 16 core scenarios, applied to 30 models.^[4] That is valuable for comparing models in general. It cannot tell you how a model handles your support tickets, your contract format or your SQL schema, which is why a small private suite sits next to public numbers instead of being replaced by them.

Step 1: write criteria you can measure

Pick the two or three dimensions that decide whether your feature works. Anthropic lists common ones: task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency and price.^[1] Then attach a method and a test set to each. The Anthropic page gives a useful template, reproduced here with the figures it uses as examples:

Source: Anthropic, "Define success criteria and build evaluations", read 2026-10-11.^[1] The sizes are the page's illustrations, not recommendations for your project.

Our interpretation: notice that latency and price are criteria too. A model swap that keeps quality equal and halves cost is a result worth detecting, and you can only detect it if cost per run is recorded next to the quality score. The arithmetic of that cost is covered in tokens, context windows and what a request costs.

Step 2: build the test set from real work

Anthropic's first design principle is to be task-specific: "Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!"^[1] Its examples of edge cases are irrelevant or nonexistent input, overly long input, and ambiguous cases where humans would disagree.^[1]

A practical way to start, which is our suggestion rather than a documented requirement:

1. Pull 30 to 50 real inputs from logs or tickets, with personal data removed.
2. Add the failures you already know about. Every bug report becomes a test case.
3. Add a few deliberately hostile or malformed inputs.
4. Write the expected answer or the rubric for each before you look at any model output.
5. Grow the set over time. The Anthropic page also suggests asking Claude to generate more cases from a baseline set, which is reasonable as long as a person reviews what it produces.^[1]

On volume, the same page says: "More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals."^[1] That is a trade-off statement, not a law. For a high-stakes domain you may still want expert-reviewed cases, but a large automated set is what lets you run the suite on every change.

Step 3: choose the cheapest reliable grader

Anthropic ranks grading methods by speed, reliability and scalability and advises choosing "the fastest, most reliable, most scalable method".^[1] In its summary:

• Code-based grading is the fastest and most reliable and scales extremely well, but lacks nuance for complex judgments. Exact match and "key phrase in output" checks belong here.^[1]
• Human grading is the most flexible and highest quality, but slow and expensive, and the page says to avoid it if possible.^[1]
• LLM-based grading is fast and flexible and suits complex judgment, with the caveat to test reliability first, then scale.^[1]

OpenAI's evals guide shows the code-graded end of this range with a string_check grader that compares the model output to a reference label using operations such as equality or contains.^[2] That is the right tool for classification, extraction into fixed fields, routing decisions and anything else with a single correct answer.

A minimal exact-match grader in Python looks like the example in Anthropic's documentation:

def evaluate_exact_match(model_output: str, correct_answer: str) -> bool:
    return model_output.strip().lower() == correct_answer.lower()

Use it wherever you can force the model to answer in a constrained format. If you cannot, the next step is a judge.

Step 4: use an LLM judge, but verify it first

For tone, helpfulness, faithfulness to a source or other judgments that code cannot express, Anthropic's guidance for LLM grading is concrete:^[1]

• Write detailed, clear rubrics. Its example: "The answer should always mention 'Acme Inc.' in the first sentence. If it does not, the answer is automatically graded as 'incorrect.'"
• Make the output empirical: ask for 'correct' or 'incorrect', or a 1 to 5 scale, because purely qualitative output is hard to assess at scale.
• Encourage reasoning by using a grader with thinking turned on so it reasons before it scores.
• Use a different model to grade than the one that generated the output. The page labels this "generally best practice".

The research literature supports the idea and also warns about it. The MT-Bench and Chatbot Arena paper reports that strong judges such as GPT-4 can match human preferences with "over 80% agreement, the same level of agreement between humans".^[3] The same abstract names the known weaknesses of LLM judges: position bias, verbosity bias, self-enhancement bias and limited reasoning ability.^[3] The abstract names them without detailing them, so the following reading is ours: a judge may favour whichever answer appears first, prefer longer answers, or prefer text that resembles its own, and each of those can quietly distort a comparison between two models.

Practical checks that follow from that list:

1. Label a sample by hand. Take 30 to 50 outputs and grade them yourself. Compare with the judge. If agreement is poor, fix the rubric before running anything at scale.
2. Swap positions in any pairwise comparison and count a win only if it holds both ways.
3. Watch length. If the winning model is also the wordier one, look at a few cases manually.
4. Keep the judge fixed. Change the judge and you change the yardstick, so store the judge model and rubric with every result.

Here is a compact judge in the pattern of Anthropic's own example. It uses the anthropic Python SDK; the model name is the one used on the documentation page, so check the current models list before you copy it.

import anthropic

client = anthropic.Anthropic()


def build_grader_prompt(answer: str, rubric: str) -> str:
    return f"""Grade this answer based on the rubric:
<rubric>{rubric}</rubric>
<answer>{answer}</answer>
Output 'correct' or 'incorrect' in <result> tags."""


def grade(answer: str, rubric: str) -> str:
    message = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=2048,
        messages=[{"role": "user", "content": build_grader_prompt(answer, rubric)}],
    )
    text = next(block.text for block in message.content if block.type == "text")
    return "correct" if "<result>correct</result>" in text.lower() else "incorrect"

The parsing is deliberately strict: anything that does not contain the expected tag counts as incorrect, so a malformed judge reply lowers your score instead of silently passing.

Step 5: make the suite survive model changes

The point of the suite is to be rerun. Habits that keep it useful:

• Version everything. Store, with each run, the model identifier, the prompt, the judge model, the rubric and the date. A score without these cannot be compared later, the same lesson as with public benchmarks.
• Change one thing at a time. Swap the model or the prompt, not both.
• Report cost and latency with quality. A cheaper tier that scores the same on your set is a finding. See Claude Haiku 5.5: specs, pricing and when to use it for an example of a small tier worth testing this way.
• Track failures, not only the average. An average hides a new class of error. Keep a list of cases that flipped from pass to fail.
• Refresh the set. If production traffic drifts, your test set should drift with it. Public benchmarks can be contaminated by training data; a private set avoids that only while it stays private.
• Test the hard parts. Unfaithful or invented answers are a failure mode in their own right; why models hallucinate and what reduces it lists the mitigations your cases should probe.

Keep it portable

Hosted tooling can disappear. OpenAI's evals guide states that the company is deprecating its Evals platform: content becomes read-only for existing users on October 31, 2026, with shutdown scheduled for November 30, 2026, and the page points new users to Datasets as an alternative.^[2] Whatever tool you use, keep your test cases in plain JSONL or CSV in version control and keep graders as ordinary functions. Then moving between tools, or between a vendor's console and your CI pipeline, is a matter of rewriting a thin wrapper.

What this approach does not solve

• Small sets are noisy. With 50 cases, one flipped answer moves the score two points. Treat small differences as unproven.
• Judges drift and bias. Verification against human labels reduces the risk but does not remove it.
• Evals measure what you wrote down. Misjudged criteria produce a confident answer to the wrong question.
• We did not run an experiment for this article. The code and numbers above come from the cited documentation and papers; we have not benchmarked any model for this piece.

A one-page checklist

1. Two or three measurable success criteria, including cost and latency.
2. At least 30 inputs from real work plus known failures and edge cases.
3. Expected answers or rubrics written before you see outputs.
4. Code grading wherever the answer is constrained; an LLM judge elsewhere.
5. The judge checked against a hand-labelled sample, positions swapped, a different model from the generator.
6. Every run stored with model, prompt, judge, rubric and date.
7. Data and graders kept in portable files.

With that in place, the next model launch becomes an afternoon of reruns instead of a debate. Read the claims in the benchmark guide to decide which models earn a place on your shortlist, then let your own suite decide.