When a vendor says an agent "scores 60% on" a benchmark, three questions decide what that means: what counted as a pass, how many attempts the agent got, and whether one lucky success was enough. AI agent evaluation is the practice of answering those questions explicitly. This guide explains how agent benchmarks grade results, why two metrics called pass@k and pass^k can tell opposite stories about the same agent, and how four well-known benchmarks (τ-bench, SWE-bench, GAIA and OSWorld) decide what a pass is. It complements our guides to building LLM evals that survive production and to reading a benchmark claim.
What we read
We opened the τ-bench, SWE-bench, GAIA and OSWorld papers on arXiv and Anthropic's engineering post "Demystifying evals for AI agents" (9 January 2026), and checked each figure below against the paper text on 4 November 2026.[1][2][3][4][5] The benchmark numbers come from 2023 and 2024 and describe models of that period; they illustrate how the scoring works, not how current models rank. For current results, read the live leaderboards and the model makers' own pages.
Outcome or transcript: what gets graded
Anthropic separates two things an eval can look at. A transcript (also called a trace or trajectory) is the full record of a trial: outputs, tool calls, reasoning and intermediate results. The outcome is the final state of the environment. Its example: a flight-booking agent may say "Your flight has been booked", but the outcome is whether a reservation exists in the database.[5]
That distinction explains how the benchmarks work:
- τ-bench compares the database state at the end of a conversation with an annotated goal state.[1]
- SWE-bench applies the original pull request's tests: each task has at least one test that fails before the fix and passes after it, and a median of 51 further tests check that earlier behaviour still works.[2]
- GAIA gives each question one correct answer, and evaluation is done by quasi-exact match.[3]
- OSWorld attaches an initial-state configuration and a custom execution-based evaluation script to each task; the paper reports 134 unique evaluation functions.[4]
Anthropic's advice follows from this: checking for a fixed sequence of tool calls is "too rigid" because agents "regularly find valid approaches that eval designers didn't anticipate", so "it's often better to grade what the agent produced, not the path it took."[5] It adds that tasks with several components should get partial credit, and that graders should resist bypasses so that passing requires solving the problem.[5]
Three kinds of grader
Anthropic describes three grader types, each looking at part of the transcript or outcome:[5]
| Grader | Typical methods | Strength | Weakness |
|---|---|---|---|
| Code-based | String match, fail-to-pass tests, static analysis, outcome checks | Fast, cheap, objective, reproducible | Brittle to valid variations; limited for subjective work |
| Model-based | Rubrics, pairwise comparison, multi-judge consensus | Flexible, handles open-ended output | Non-deterministic, costlier, needs calibration against humans |
| Human | Expert review, spot checks, A/B tests | Gold-standard quality | Expensive and slow |
The guidance is to prefer deterministic graders where possible, use model graders where necessary, and use humans to validate.[5] The four benchmarks above lean on code-based grading, which is why they are cheap to rerun and why they only cover tasks whose success can be checked mechanically.
pass@k and pass^k: the same agent, opposite verdicts
Agents are stochastic, so one run per task is a noisy measurement. Two metrics describe repeated runs:
- pass@k is the chance that at least one of k attempts succeeds. It rises as k increases.[5]
- pass^k is the chance that all k attempts succeed. It falls as k increases. The τ-bench authors introduced it for settings like customer service, where consistency matters, and estimate it from n trials per task as the average over tasks of C(c,k)/C(n,k), where c is the number of successful trials.[1][5]
At k=1 the two are identical. Anthropic's worked example: with a 75% per-trial success rate, the probability of passing three trials in a row is 0.75³, about 42%.[5] The table below extends that arithmetic. It is our calculation, and it assumes every task has the same success rate and trials are independent, which real agents do not satisfy, so treat it as an illustration of direction, not a prediction.
| Per-trial success | pass@1 / pass^1 | pass@3 | pass^3 | pass@8 | pass^8 |
|---|---|---|---|---|---|
| 90% | 0.90 | 0.999 | 0.729 | 1.00 | 0.43 |
| 75% | 0.75 | 0.984 | 0.42 | 1.00 | 0.10 |
| 60% | 0.60 | 0.936 | 0.22 | 0.999 | 0.02 |
A real result shows the same shape. In τ-bench, the best agent, a gpt-4o function-calling agent with an average task success above 60% on the retail domain, saw its pass^8 fall below 25%. On the airline domain the same model solved 35.2% of tasks.[1] The authors conclude that it is hard to build agents with both high average success and consistency across trials.[1]
Which one you report should come from the product. Anthropic's rule: pass@k for tools where one success matters, pass^k for agents where consistency is essential.[5] A code assistant that drafts several patches for a human to choose from is a pass@k product. An agent that issues refunds without review is a pass^k product.
Reading a score: five checks
This checklist is our own synthesis of the sources, not a standard.
- What was graded? Final state, tests, an answer string, or a model's opinion of the transcript.
- How many trials, and which metric? A pass@8 and a pass^8 are not comparable. Ask for pass@1 as the baseline.
- Were the agent and environment fresh each trial? Anthropic warns that shared state between runs, such as leftover files or caches, can cause correlated failures or inflate performance.[5]
- Is the task set fair? The τ-bench authors manually examined 40 failed trajectories out of 115 and found 4 were caused by user-instruction typos or ambiguity, which they fixed.[1] Anthropic reports that Opus 4.5 initially scored 42% on CORE-Bench until a researcher found rigid grading (penalising "96.12" when expecting "96.124991…"), ambiguous task specs and stochastic tasks that were impossible to reproduce exactly; after fixes and a less constrained scaffold its score rose to 95%.[5] Read transcripts before trusting a number in either direction.
- What did a run cost? The τ-bench authors report $0.38 per task for the agent and $0.23 per task for the simulated user in the retail domain, about $200 for one trial per task.[1] Multi-trial metrics multiply that cost, and it is a reason many teams run few trials.
Capability evals versus regression evals
Anthropic also distinguishes two purposes. Capability evals ask what an agent can do well and should start at a low pass rate, giving a hill to climb. Regression evals ask whether it still handles everything it used to, and should sit near 100%; a drop signals something broke. Capability tasks with high pass rates can "graduate" into the regression suite.[5] This matters for reading vendor claims: a benchmark near saturation says little about capability, but a regression suite near 100% is exactly what you want to see.
Limits of this guide
- The benchmark papers are from 2023 and 2024. Their scoring designs still illustrate the concepts, but their scores are dated and the tasks may be saturated or contaminated for current models; we did not check current leaderboards.
- The pass^k estimator formula is as printed in the τ-bench paper; we did not re-derive it.
- Anthropic's post describes its own practice and recommendations; it is guidance, not a standard, and we did not run any evaluation ourselves.
- Agent benchmarks that need a live GUI or web environment, such as OSWorld, are harder to reproduce than test-based ones; the OSWorld paper reports humans completing over 72% of its 369 tasks against 12.24% for the best model then tested, a gap that has surely changed since.[4]
For how agents are built and attacked, see how AI agents work and how to secure them; for one agent type evaluated on a live screen, see computer use agents explained.
