How to Build LLM Evals for Your Own Application

An eval is a repeatable test for an LLM application: inputs, a way to grade the outputs, and a threshold that says what passing means. If you are choosing between models or about to upgrade one, a small eval of your own answers the question that matters, which is whether the new setup is better for this task. Public benchmarks cannot, as how to read an AI benchmark claim explains. OpenAI draws the same line: industry benchmarks compare models in isolation, while your own evals are "specific tests you implement to measure your LLM application's performance."^[3]

This guide covers the practical steps: define success, build the test set, choose graders, calibrate an LLM judge, and keep the suite running. The documentation of Anthropic and OpenAI agrees on most of it, and where they differ in emphasis we say so.

Quick answer

1. Write success criteria that are specific and measurable, usually several at once.
2. Collect test cases that look like your real traffic, plus the awkward edge cases.
3. Grade with code where you can, an LLM judge where you must, and humans to check the judge.
4. Set a pass/fail threshold, not just a score.
5. Re-run the identical suite whenever the model, prompt or retrieval setup changes.

Step 1: define success before you test

Anthropic's guidance says good criteria are specific, measurable, achievable and relevant.^[2] "Good performance" fails the first test; "accurate sentiment classification" passes. Even hazy goals can be quantified: the page's example replaces "safe outputs" with "less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter."^[2]

Most applications need more than one criterion. The same page lists common categories, and the table below pairs them with the guiding questions it gives.^[2]

The page's worked example combines four quantified criteria for a sentiment classifier: an F1 score of at least 0.85, 99.5% of outputs non-toxic, 90% of errors causing inconvenience rather than egregious harm, and 95% of responses under 200 ms.^[2] Our reading: the third one is the most unusual and the most useful, because it grades how the system fails, not only how often.

Step 2: build test cases that mirror production

Anthropic's advice is to design evals that mirror your real-world task distribution and "factor in edge cases," including irrelevant or nonexistent input, overly long input, harmful or poor user input, and ambiguous cases "where even humans would find it hard to reach an assessment consensus."^[1]

OpenAI names the opposite as a pitfall: datasets that "don't faithfully reproduce production traffic patterns," and waiting until you ship before writing any evals.^[3] Its advice to log during development and mine the logs for cases is a cheap way to start.^[3]

The two vendors differ slightly on size. Anthropic says to prioritise volume over quality, with "more questions with slightly lower signal automated grading" preferred to fewer hand-graded ones, and shows examples with 200 to 10,000 cases.^[1] OpenAI's guidance leans toward human-labelled ground truth plus calibration.^[3][4] These are compatible: use many automatically graded cases for coverage and a smaller human-labelled set to check that the automation measures what you think it does.

Whatever the size, keep the test set stable. The value of an eval is the comparison between runs, so changing the cases and the model at the same time tells you nothing.

Step 3: pick the cheapest reliable grader

Anthropic ranks grading methods by speed, reliability and scale:^[1]

The documentation's own advice is to choose "the fastest, most reliable, most scalable method" for each criterion.^[1] In practice that means a classification label or a required phrase is checked with code, a tone or coherence judgment goes to an LLM grader, and humans are reserved for calibration and the ambiguous cases.

Code-based graders go further than exact match. Anthropic's page also describes cosine similarity between sentence embeddings as a way to measure consistency, and ROUGE-L for summaries against a reference.^[1] Treat overlap metrics as signals, not verdicts, since OpenAI lists "relying solely on academic metrics like perplexity or BLEU score" among common mistakes.^[3]

Step 4: use an LLM judge, but verify it

Judging by model is attractive because it scales. Zheng et al. found that strong judges such as GPT-4 reached over 80% agreement with human preferences on their benchmarks, which they say matches the agreement between humans, and they also catalogued the weaknesses: position bias, verbosity bias, self-enhancement bias and limited reasoning ability.^[5] Those results are from 2023-era models and chat-style comparison questions; check agreement on your own task rather than assuming it carries over.

The vendor guidance converts those findings into rules:

• Use a different model as judge than the one that produced the output.^[1]
• Write a detailed rubric and ask for a constrained answer such as "correct" or "incorrect", or a 1 to 5 score with the endpoints defined.^[1]
• Prefer pairwise comparison or pass/fail over open-ended scores, which OpenAI says are more reliable.^[3]
• Control for length, because "LLMs bias towards longer responses in general."^[3]
• Let the judge reason first. Both vendors say reasoning before scoring improves results.^[1][3]
• Calibrate against people. OpenAI advises scaling up only once the judge consistently agrees with human annotations, using multiple reviewers and consensus votes.^[3]
• Validate the output. Anthropic's example code rejects a grade that is not a number instead of silently scoring it.^[1]

A minimal harness

The sketch below is our illustration of the structure, not vendor code. call_model is a placeholder for whichever SDK you use, so nothing here depends on a specific API. Python 3.10 or later.

import json
from collections import Counter


def call_model(model: str, prompt: str) -> str:
    """Replace with your SDK call. Must return the model's text."""
    raise NotImplementedError


def judge(judge_model: str, rubric: str, answer: str) -> str:
    prompt = (
        "Grade the answer against the rubric. "
        "Reply with exactly one word: pass or fail.\n"
        f"<rubric>{rubric}</rubric>\n<answer>{answer}</answer>"
    )
    verdict = call_model(judge_model, prompt).strip().lower()
    if verdict not in {"pass", "fail"}:
        raise ValueError(f"Unexpected grader output: {verdict!r}")
    return verdict


def run_eval(cases_path: str, candidate: str, judge_model: str) -> Counter:
    results = Counter()
    with open(cases_path) as f:
        for line in f:
            case = json.loads(line)  # {"input": ..., "expected": ..., "rubric": ...}
            answer = call_model(candidate, case["input"])
            if "expected" in case:  # code-based grading first
                ok = answer.strip() == case["expected"]
            else:  # LLM judge only where code cannot decide
                ok = judge(judge_model, case["rubric"], answer) == "pass"
            results["pass" if ok else "fail"] += 1
    return results

Run it with the same cases.jsonl against the current model and the candidate, and compare the pass rates with the threshold you set in step 1. For a stricter setup, run the judge twice with the two answers in swapped order when you compare models pairwise, which controls for the position bias Zheng et al. describe.^[5]

Step 5: keep it running

OpenAI calls evals "an essential component to building reliable applications," especially when upgrading or trying new models.^[4] That is the point of the suite: when a new model ships, such as the Claude Haiku 5.5 release, you can answer whether it is good enough and cheaper in an afternoon, then apply the cost arithmetic to decide whether to switch. Evals also give you a regression check on prompt edits, and they are the practical test for whether a mitigation in the hallucination guide actually helped.

What an eval cannot tell you

• It measures what you wrote down. A suite that omits a failure mode will pass it.
• Scores are not comparable across suites. A 92% pass rate on your cases says nothing about someone else's.
• Judges inherit bias. Calibration reduces it; it does not remove it.
• Small differences are noisy. With a few hundred cases, a one-point change may be within run-to-run variation. We did not run experiments for this article, so we cannot give a threshold; measure the variance of your own suite by running it repeatedly on the same model.

Sources and method

This guide is based on documentation and a paper opened on 2026-10-10 (listed above). The harness is our own illustration and was not run against a model; no performance results are claimed. Vendor documentation changes often, so re-check the linked pages before relying on specifics.