An eval is a test suite for a system whose output varies. It is a set of inputs, a way of grading what comes back, and a number you can compare before and after a change. LLM evaluation is what lets you swap a model, edit a prompt or retune retrieval and know whether the product got better, instead of reading five outputs and hoping. This guide covers how to define success, which grading method to use, where LLM judges go wrong, and how to keep an eval useful for longer than a week. It pairs with how to read an AI benchmark claim, which covers other people's evals, and with why AI models hallucinate, which is one of the failures an eval should catch.
Step 1: write success criteria you can measure
Anthropic's documentation lists four properties of good criteria: specific, measurable, achievable and relevant.[2] Its own contrast is useful: instead of "good performance", specify "accurate sentiment classification".[2] A worked example from the same page reads: an F1 score of at least 0.85 on a held-out test set of 10,000 diverse Twitter posts, a 5% improvement over the current baseline.[2]
The page also stresses that "most use cases need multidimensional evaluation along several success criteria."[2] It lists common dimensions:
| Dimension | The question it asks |
|---|---|
| Task fidelity | How well must the model perform, including on rare or hard inputs? |
| Consistency | Should similar inputs get semantically similar answers? |
| Relevance and coherence | Does the answer address the question in a logical order? |
| Tone and style | Is the language right for the audience? |
| Privacy preservation | Does it follow instructions not to use or share certain details? |
| Context utilization | Does it use and build on the provided context and history? |
| Latency | What response time does the application tolerate? |
| Price | What is the budget per call and in aggregate? |
The last two are easy to forget because they are not about answer quality, but they decide which model you can ship. If cost is a criterion, the arithmetic in tokens, context windows and what an AI request really costs belongs in your eval report. HELM makes the same argument at research scale: it measures seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) on shared scenarios so that trade-offs between metrics are visible rather than hidden behind one accuracy figure.[4]
Step 2: choose the cheapest grader that is reliable
Anthropic's guidance is to "choose the fastest, most reliable, most scalable method" and describes three options:[1]
| Method | Strength | Weakness | Typical use |
|---|---|---|---|
| Code-based grading | Fastest and most reliable, extremely scalable | Lacks nuance for complex judgements | Exact match, key-phrase match, format checks |
| Human grading | Most flexible and highest quality | Slow and expensive | Calibrating the other two, final spot checks |
| LLM-based grading | Fast, flexible, scalable, suited to complex judgement | Must be tested for reliability first | Tone, relevance, open-ended answers |
Our interpretation: treat these as a ladder. Use code wherever a rule can express the criterion, because a failing string match tells you exactly what broke. Use an LLM grader only for what rules cannot capture. Use humans to check that the LLM grader agrees with them, not to grade everything.
The documentation's own code-based example is a one-line exact match, normalising case and whitespace:[1]
def evaluate_exact_match(model_output, correct_answer):
return model_output.strip().lower() == correct_answer.lower()Step 3: use LLM judges, and distrust them a little
For subjective criteria the documentation gives three tips for LLM-based grading: write detailed, clear rubrics; make the output empirical or specific (for example only "correct" or "incorrect", or a 1 to 5 scale); and encourage reasoning by using a grader with thinking on so it reasons before it scores.[1] Its examples also carry the comment that it is generally best practice to use a different model to evaluate than the one that generated the output.[1]
The research case for judges is strong but conditional. Zheng et al. report that strong judges such as GPT-4 reach over 80% agreement with both controlled and crowdsourced human preferences, the same level at which humans agree with each other.[3] The same paper identifies biases and limits: position bias, verbosity bias, self-enhancement bias and limited reasoning ability.[3] In practice that suggests these checks, which are our recommendations rather than steps taken from the paper:
- Swap the order of candidate answers and check the verdict does not flip.
- Compare scores against answer length, in case longer answers win regardless of quality.
- Do not let a model grade its own family's outputs without a human sample to check against.
- Keep a small human-graded subset and report judge agreement with it alongside the score.
Step 4: build the test set from your real task
Anthropic's design principles are to be task-specific, automate where possible, and prioritise volume over quality: "More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals."[1] Mirror the real task distribution and include edge cases: irrelevant or nonexistent input, overly long input, poor or harmful user input, and ambiguous cases where even humans would struggle to agree.[1]
The documentation's example sets give a sense of scale and method pairing, for example 1,000 labelled tweets graded by exact match for sentiment, and 100 customer inquiries scored by an LLM on a 1 to 5 Likert scale for tone.[1] These are illustrations of structure, not recommended sizes.
A minimal harness
This sketch is ours, not from the sources. It keeps generation and grading as plug-in functions, so the same loop works for code graders and LLM graders and reports a score per criterion rather than one blended number.
from collections import defaultdict
from typing import Callable
Case = dict # {"input": str, "expected": str, "criterion": str}
def run_eval(
cases: list[Case],
generate: Callable[[str], str],
grade: Callable[[Case, str], bool],
) -> dict[str, float]:
passed: dict[str, int] = defaultdict(int)
total: dict[str, int] = defaultdict(int)
for case in cases:
output = generate(case["input"])
total[case["criterion"]] += 1
if grade(case, output):
passed[case["criterion"]] += 1
return {c: passed[c] / total[c] for c in total}Save the per-case outputs and verdicts, not just the scores. When a number moves, you will want to read the cases that flipped.
Keeping an eval alive
An eval set rots like any test suite. The practical habits are the unglamorous ones:
- Add every production failure as a case. A bug that reached users is a free, perfectly realistic test.
- Version the set, the rubric and the grader model together. A score is only comparable to another score produced the same way.
- Re-check the judge whenever you change the grader model or rubric, against the human-graded subset.
- Track cost and latency next to quality, since the dimension list above includes both.
- Distrust a perfect score. If everything passes, the set is probably too easy; add harder cases.
What the sources do not tell you
None of the sources gives a minimum number of test cases, a threshold for acceptable judge agreement, or evidence about how a particular current model behaves as a grader. The Anthropic pages are product documentation, not independent research, and the judge-bias paper dates from 2023 using models of that period, so measure bias on your own judge instead of assuming the published figures transfer.
