Sim-to-real transfer means training a robot's model in a physics simulator and then running it on a physical robot. It matters because learning needs enormous amounts of experience, and a simulator can produce that experience faster and with less risk than hardware can. The catch is the reality gap: a simulator never matches the world in every detail, so a policy that works on screen can fail on a table. The most cited tool for narrowing that gap is domain randomization, and three papers show what it achieved and what it left undone.[1][2][3]
Why robots learn in simulation at all
The Rubik's cube paper states the problem plainly: learning requires vast amounts of training data, which is hard and expensive to acquire on a physical system, so collecting all data in simulation is appealing, provided the sim-to-real transfer problem is solved.[3] The scale on offer is large. The in-hand manipulation paper describes a pool of 384 worker machines with 16 CPU cores each that generate about two years of simulated experience per hour, with optimization on a single machine with 8 GPUs.[2] A real robot hand cannot collect experience at that rate.
Later work pushed the same direction with hardware. The Isaac Gym paper describes a platform where physics simulation and neural-network training both run on the GPU, and reports training speedups of two to three orders of magnitude over setups that use a CPU-based simulator with a GPU for the network.[4] That is a speed claim by the platform's authors about their own comparison, and it says nothing about transfer quality. It does show why the field keeps investing in simulators.
What domain randomization randomizes
The idea, in the words of the 2017 paper, is that "with enough variability in the simulator, the real world may appear to the model as just another variation."[1] Instead of training in one simulated world, you train across a wide range of them.
That paper randomizes how the scene is rendered: the number and shape of distractor objects, the textures of the table, floor and robot, the camera position, lighting, and the type and amount of image noise. The textures are deliberately non-realistic, generated by a simple random process, which let the authors train on hundreds of thousands of low-fidelity rendered images.[1]
The 2018 paper extends the idea from appearance to physics. It randomizes physical parameters at the start of every episode and holds them fixed through it. Its table lists scaling ranges for object dimensions (uniform 0.95 to 1.05), object and robot link masses (uniform 0.5 to 1.5), surface friction coefficients (uniform 0.7 to 1.3), joint damping and actuator force gains, plus noise on joint limits and the gravity vector. It also models effects the simulator lacks, such as motor backlash, action delays and noise, and the motion-capture system temporarily losing a marker.[2] The visual side randomizes camera positions and intrinsics, lighting, and materials and textures of all objects.[2]
A third piece matters for the policy itself. The authors use a recurrent network with memory (an LSTM) so the policy can identify properties of the current environment and adapt, since the randomized friction or mass cannot be read off directly.[2] The Rubik's cube paper reports clear signs of emergent learning at test time in this recurrent state, and argues that training an LSTM over an ADR distribution is implicit meta-learning.[3]
What the numbers say
Vision: 1.5 cm from a simulator alone
The 2017 paper trains an object detector on simulated images only and tests it on 480 real webcam images of geometric objects on a table, at 70 to 105 cm from the camera, with lighting not controlled. The detectors localized objects to within 1.5 cm on average in the real world, including with distractors and partial occlusion. The authors note the models were still over-fitting the simulated data, where error was 0.3 to 0.5 cm.[1]
The ablations are the instructive part:
- Performance degraded significantly when fewer than 1,000 unique textures were used; for their experiments a large number of random textures was necessary for transfer.[1]
- Including distractors during training appeared critical to resilience to distractors in the real world.[1]
- Camera-position randomization gave a slight accuracy boost, and pretraining on real images was not necessary once there were enough textures.[1]
A grasping demo then used the detector. For one object (a can of Spam from the YCB set) it picked up the target in 9 of 10 trials among previously unseen distractors.[1] That is a single small demonstration, and the authors frame the next step as making the technique reliable enough for contact-rich or higher-precision manipulation.[1]
Control: a block, 50 rotations, and the drop on hardware
The 2018 paper trains a policy entirely in simulation to reorient a block in a Shadow Dexterous Hand. Success is counted as consecutive rotations until the block is dropped, 80 seconds pass without a goal, or 50 rotations are reached. The table reports 100 trials in simulation and 10 trials per policy on the physical robot:[2]
| Task | Simulation median | Physical robot median |
|---|---|---|
| Block (state input) | 50 | 13 |
| Block (vision input) | 33 | 11.5 |
| Octagonal prism (state input) | 30 | 5 |
On the physical robot the block's individual trials ranged from 1 to 50 rotations. The authors conclude: "even though randomizations and calibration narrow the reality gap, it still exists and performance on the real system is worse than in simulation."[2] A sphere failed to get more than a few rotations in a row, which they speculate could stem from not randomizing rolling-related parameters.[2]
The vision result is also notable. The vision-based pose estimator, trained on synthetic images only, did slightly worse than motion-capture pose estimation on the real robot, but the authors call the difference surprisingly small.[2]
Automatic domain randomization
Choosing randomization ranges by hand is guesswork. The Rubik's cube paper introduces automatic domain randomization (ADR), which "automatically generates a distribution over randomized environments of ever-increasing difficulty." Together with a custom robot platform it let a humanoid hand solve a Rubik's cube using control policies and vision state estimators trained only in simulation.[3] We opened the abstract and introduction of that paper, not its full results, so we do not report its success rates here.
How to read a sim-to-real claim
Our interpretation of these papers is that randomization is a way of buying robustness with compute and design effort, not a guarantee. A few questions follow from what the authors themselves report.
- What was randomized, and was it enough? The sphere failure suggests that a missing parameter can cost the whole task.[2]
- How many real trials? Ten trials per policy on hardware is a small sample next to 100 in simulation.[2]
- Is the real number reported next to the simulated one? In these papers it is, and the gap is the finding.
- Is the result about perception or contact? Localizing objects from images transferred earlier and more simply than contact-rich manipulation, which the 2017 authors list as future work.[1]
The same scepticism applies to model leaderboards; see how to read an AI benchmark claim. Simulation is also only one source of robot data: vision-language-action models such as RT-2 and OpenVLA lean on web data and real demonstrations instead. And the cost logic of generating experience cheaply echoes the inference side of AI infrastructure, covered in inference serving explained.
Limits of this article
These are papers from 2017 to 2021, on specific robots and tasks. We did not test any of this ourselves, and we did not read the full text of the Rubik's cube or Isaac Gym papers beyond their abstracts and introductions. Simulators, robots and methods have moved on, so use these as the foundations of the idea, not a statement of the current state of the art. More on this beat lives on the robotics topic page.
