Robot policy evaluation is the problem of measuring how well a learned robot controller does its job, and it is harder than it sounds. A model that picks up a can in a lab video may fail in the next room. This article explains how two research groups tried to make those measurements comparable: SIMPLER, which tests policies in simulation, and RoboArena, which crowd-sources real-robot tests.
Why a success rate is hard to read
If you read about a vision-language-action model (our explainer on vision-language-action models covers how they work), the headline is usually a success rate on a handful of tasks. That number depends on choices the reader rarely sees: which objects, which table, which lighting, how many trials, and who decided what counts as success.
The SIMPLER paper opens with this problem. Its authors say real-world evaluation of generalist manipulation policies does not scale and is hard to reproduce.[1] The RoboArena authors describe a related limit: conventional evaluation fixes tasks, environments and locations, and they propose crowd-sourcing evaluations across a distributed network instead.[2]
The same caution we apply to language models applies here. Our guide to reading an AI benchmark claim lists the questions to ask; for robots, the questions are about physical setup as much as about data.
SIMPLER: evaluate in simulation, then check against reality
SIMPLER is a set of simulated environments for evaluating manipulation policies on common real robot setups.[1] The design question is how to make a simulator trustworthy without building an exact digital twin of every lab. The authors identify two gaps, control and visual, and address each.
Closing the control gap
A real arm does not follow commands exactly as a simulated one does. The paper tunes the simulated controller's stiffness and damping parameters by simulated annealing so that simulated trajectories track real ones, a form of system identification. In the authors' ablation, adding this step lowered the ranking error metric MMRV to 0.031, compared with 0.070 and 0.100 for two settings without it.[1]
Closing the visual gap
Policies trained on real camera images can be sensitive to how a scene looks. The authors composite real backgrounds behind simulated objects ("green screening"), project real textures onto object meshes and tune the simulated robot arm's colour to match real footage. In their ablation the best combination gave an MMRV of 0.050, against 0.087 for the baseline, and partial combinations gave little or no improvement, so the techniques had to be applied together.[1]
What the correlation numbers say
The paper evaluates policies such as RT-1 checkpoints, RT-1-X, RT-2-X and Octo on Google Robot tasks (Pick Coke Can, Move Near, Open/Close Drawer) and on BridgeData V2 tasks with a WidowX arm, and compares simulated and real success. Two metrics are used: Pearson correlation (higher is better) and MMRV, a ranking-violation measure (lower is better).
| Method (Google Robot, 6 checkpoints) | Avg. Pearson r | Avg. MMRV |
|---|---|---|
| SIMPLER Visual Matching (recommended) | 0.924 | 0.056 |
| SIMPLER Variant Aggregation | 0.778 | 0.143 |
| Validation action error (baseline) | 0.308 | 0.375 |
Source: Table I of the SIMPLER paper as we read it on 2026-10-26.[1] The baseline row matters most: the paper says validation error is not a good proxy for real-world performance. That is why offline loss alone cannot rank robot policies.
The authors also report that SIMPLER tracks distribution shifts. Real-world success dropped 4% for solid table colours versus 25% for patterned textures, while simulation showed 2% and 24%.[1] Our interpretation: a simulator that reproduces which policies are fragile is useful even if absolute success rates differ.
The authors report the environments run at about 3.5k steps per second on an RTX 4090, a 7x speedup over real evaluation.[1] Everything here covers the tested robots and tasks; the paper does not claim the correlation holds elsewhere.
RoboArena: keep real robots, standardize the comparison
RoboArena goes the other way. It keeps physical robots and changes how results are collected. The authors crowd-source evaluations across a distributed network at seven academic institutions using the DROID robot platform, and report more than 600 pairwise real-robot episodes across seven generalist policies.[2]
The protocol
- A central server randomly samples two policies from the pool. Evaluators do not know which policies they run.
- The evaluator sets up a scene and writes a natural-language instruction. Both policies run back to back from matching initial conditions until success or a timeout.
- The evaluator records a progress score from 0 to 100, a binary preference, and a free-text explanation.
- Preferences are aggregated into a ranking. The paper compares Bradley-Terry variants, an Elo variant, a task-aware extension and a ranking by average progress score.[2]
Evaluators choose their own tasks, so coverage grows with the network instead of being limited to a fixed task list. Double-blind pairing guards against evaluators favouring a policy they built.
What it reports
As an oracle, the authors evaluated all remaining policies after each trial, giving 4,284 evaluations, and ranked by average progress. They compare RoboArena with a centralized procedure of 17 tasks and 44 episodes per policy. In a distribution-shift test the centralized approach scored a Pearson r of 0.692 and MMRV of 0.141; RoboArena scored 0.838 and 0.058.[2] The paper also says the method converges to high-quality rankings within about 100 pairwise comparisons.[2]
| Aspect | SIMPLER | RoboArena |
|---|---|---|
| Where it runs | Simulation | Real robots |
| Main strength | Cheap, reproducible, fast | Real hardware, diverse tasks |
| Main cost | Fidelity work per robot setup | Needs a network of willing labs |
| Validated on | Google Robot and WidowX tasks | DROID platform, seven policies |
A third kind: fixed simulated benchmarks
Some benchmarks avoid the real-world question and define everything in simulation. LIBERO, for example, provides 130 language-conditioned manipulation tasks in four suites with human-teleoperated demonstrations, built to study lifelong learning.[3] Its authors report that sequential finetuning beat existing lifelong learning methods on forward transfer and that naive supervised pretraining can hurt later learning.[3] Such a benchmark gives repeatable comparisons between algorithms, but its authors' target is knowledge transfer research, not a prediction of how a robot performs in a kitchen.
What these methods do not show
- Other robots. The RoboArena authors state that experiments used only DROID and that cross-embodiment evaluation is untested.[2]
- Adversarial evaluators. The same paper says robustness to deliberately misleading evaluators was not investigated.[2]
- Goodhart effects. The authors warn that a measure can stop being useful once it becomes a target.[2]
- Controlled experiments. Because tasks are not standardized, varying one condition at a time is hard.[2]
- Your task. Neither paper tells you how a policy handles your objects and your room.
How to read a robot result
- Find the robot and the exact task list.
- Check the number of trials per task and who scored them.
- Ask whether the comparison was blind and whether starting conditions matched.
- Prefer results from more than one lab or from a protocol like RoboArena over a single demo.
- Treat simulated numbers as useful for ranking when the paper validates them against real runs on that setup, as SIMPLER does.
Sources and method
We read the three arXiv pages and the HTML versions of the SIMPLER and RoboArena papers on 2026-10-26. We ran no experiments; figures are the authors' reports, and the table layout is our own. The RoboArena headline figure for the centralized comparison appears only in a plot in the paper, so we quote the table figures instead.
