Robot training data is the recorded experience a robot learning system learns from: episodes in which a robot performs a task while its cameras and sensors are logged next to the commands it received. Unlike web text or images, real robot data does not exist until someone makes a robot act, which is why it is scarce. This explainer covers how labs collect it, what the largest shared robot learning datasets (Open X-Embodiment, DROID, BridgeData V2 and RT-1's data) actually contain, what their papers found about mixing data, and how to read a dataset-size claim.
Why robot data is different from web data
A language model's training text already exists. Robot data has to be produced. The DROID authors put the cost plainly: collecting manipulation data in diverse environments "poses logistical and safety challenges and requires substantial investments in hardware and human labour", so even the most general manipulation policies "are mostly trained on data collected in a small number of environments".[5]
The unit is also different. The Open X-Embodiment repository describes each dataset as "a sequence of episodes", stored in the RLDS episode format.[3] An episode is a time series: at each step, observations such as camera images and the robot's own state are paired with the action it was commanded to take. The RT-X models trained on that data take recent images and a language instruction and output a 7-dimensional end-effector action.[1]
One episode is not a fixed amount of data. ALOHA records at 50Hz, so an 8-14 second task becomes 400-700 time steps.[9] The Open X-Embodiment dataset sheet lists control frequencies of 15 for DROID, 5 for Bridge and 3 for RT-1's data.[4] BridgeData V2's project page gives an average trajectory length of 38 timesteps.[8] The hardware shapes what can be recorded at all, a point our guide to reading humanoid robot spec sheets makes from the other side.
How labs collect robot data
Teleoperation: a person drives the robot
ALOHA is a bimanual setup its authors built "within a 20k USD budget": an operator moves small WidowX leader arms, and two ViperX 6-DoF follower arms copy them joint by joint.[9] They chose joint-space mapping because off-the-shelf inverse kinematics "fails frequently" in fine manipulation.[9] The paper also shows what human data costs. The authors recorded 50 demonstrations per task (100 for one task), about 10-20 minutes of data per task, which took 30-60 minutes of wall-clock time "because of resets and teleoperator mistakes".[9] With "only 10 minutes worth of demonstrations", the paper's ACT policy learned real-world tasks such as opening a translucent condiment cup and slotting a battery "with 80-90% success".[9]
DROID scales the same approach across labs. The authors replicated one setup 18 times (a Franka Panda arm, a Robotiq gripper, two adjustable Zed 2 stereo cameras, a wrist camera and a height-adjustable desk on wheels), and operators steered the arm with Meta Quest 2 controllers.[5] Fifty collectors at 13 institutions gathered data over 12 months.[5][6]
RT-1's dataset came from a fleet of 13 robots over 17 months: "over 130k individual demonstrations constituting over 700 distinct task instructions".[10] BridgeData V2 used a WidowX 250 setup that "costs approximately $4,000 in total", teleoperated with a VR controller.[7][8]
Scripted and autonomous collection
In BridgeData V2, taking the person out traded quality for volume. Its authors "could partially automate the collection of pick-and-place data with a heavily randomized scripted policy"; it "fails frequently" but runs autonomously, and they note that methods such as offline RL can learn from such suboptimal data.[7] That script produced 9,731 of the dataset's 60,096 trajectories.[8]
QT-Opt is the large autonomous example: seven robots collected 580k grasp attempts through self-supervised collection, with a binary reward for lifting an object.[11] In the Open X-Embodiment sheet, by our count on 2026-11-04, 25 of 72 entries are labelled "Human VR", 15 "Scripted" and 13 "Expert Policy", with the rest spread across spacemouse, puppeteering, kinesthetic teaching and other methods.[4]
Simulation
Simulation generates episodes without tying up a physical robot, but transferring what is learned to real robots is its own discipline, covered in our sim-to-real explainer. One data point: in RT-1's experiments, adding simulated data left success on real objects at 90% against 92% without it, while success on objects that appeared only in the simulated data rose from 23% to 87%.[10]
Human video
Video of people handling objects is plentiful but contains no robot actions. R3M pre-trains a visual representation on the Ego4D human video dataset and uses it "as a frozen perception module" for a policy trained by behaviour cloning on robot demonstrations.[12] The authors note the embodiment in that video "differs from most robots". Across 12 simulated manipulation tasks they report that R3M improves task success by over 20% compared with training from scratch and by over 10% compared with CLIP and MoCo representations, and on a real robot they learned tasks in a cluttered apartment from "just 20 demonstrations".[12] Our reading is that human video, used this way, reduces how many robot demonstrations are needed rather than replacing them.
The shared datasets, side by side
Figures are as each source reports them, in that source's units.
| Dataset (first version) | Reported size | Robots and scenes | Collection | Licence as stated |
|---|---|---|---|---|
| Open X-Embodiment (Oct 2023) | "1M+ real robot trajectories"; 527 skills (160,266 tasks)[1][2] | 22 embodiments; 21 institutions in the paper, 60 pooled datasets from 34 labs on the project page[1][2] | Aggregation of existing datasets[2] | Repository: software Apache 2.0, "all other materials" CC BY 4.0; no per-dataset licence listed[3][4] |
| DROID (Mar 2024) | 76k successful trajectories, 350 hours; about 16k more labelled not successful[5] | One arm model (Franka Panda), 18 setups, 564 scenes, 52 buildings, 13 institutions[5] | Human teleoperation[5] | CC BY 4.0[5] |
| BridgeData V2 (Aug 2023) | 60,096 trajectories (50,365 teleoperated, 9,731 scripted)[8] | WidowX 250, 24 environments, 13 skills[8] | Teleoperation plus a scripted policy[8] | CC BY 4.0[8] |
| RT-1 (Dec 2022) | Over 130k demonstrations, over 700 task instructions[10] | 13 robots, 17 months of collection[10] | Human demonstrations[10] | Not checked separately; distributed within Open X-Embodiment[4] |
Why the numbers do not line up
- Units. Trajectories, episodes, demonstrations and hours are different measures. DROID's comparison table counts RH20T at 13k trajectories and notes that RH20T's own paper reports 110k by counting each camera stream separately.[5]
- Versions. DROID's first-version abstract says 84 tasks; the revised paper and project page say 86.[5][6]
- Growth. The Open X-Embodiment paper says 1M+ trajectories. The current version of DROID's comparison table lists it at 1.4M.[1][5] The official sheet showed a total of 2,419,193 episodes across 72 entries on 2026-11-04, including DROID in a later v1.1 batch.[4]
- Same name, different size. The sheet lists 73,499 episodes for RT-1's robot data (the paper: over 130k demonstrations) and 25,460 for Bridge, citing BridgeData V2 (60,096 trajectories). DROID's 92,233 is close to 76k successful plus about 16k unsuccessful episodes.[4][10][8][5] The materials we read do not explain the RT-1 and Bridge gaps.
What the aggregate is made of (our calculation)
From the sheet as downloaded on 2026-11-04: four entries (VIMA 660,103 episodes, QT-Opt 580,392, Language Table 442,226 and SPOC 233,000) make up about 79% of all listed episodes, and three of the four are labelled "Scripted" or "Expert Policy".[4] The paper describes Franka as the most common embodiment.[1] In the current sheet it appears in 28 of 72 entries but accounts for about 7% of episodes.[4] Our reading: the number of datasets per robot and the share of data per robot are different measures of diversity, and a claim should say which it uses.
What the papers found about mixing and scale
Open X-Embodiment. The project page says RT-1-X outperforms RT-1 or the original methods "by 50% in the small-data domain".[2] On the large Bridge and RT-1 datasets, "the RT-1-X model does not outperform the RT-1 baseline trained on only the embodiment-specific dataset", which the authors read as "underfitting for that model class"; the larger RT-2-X beat both.[1] On emergent skills (objects and skills present in Bridge data but not in RT-2's data, tested on the Google Robot), RT-2-X scored 75.8% against RT-2's 27.3%, and removing Bridge from training cut it to 42.8%.[1] The paper reports 3,600 evaluation trials across 6 robots.[1] Its stated limits: the experiments "do not consider robots with very different sensing and actuation modalities", do not study generalization to new robots, and give no criterion for when positive transfer happens.[1]
RT-1. "Removing 25% of the tasks while keeping 97% of the data" gave generalization equivalent to shrinking the dataset by as much as 49%, leading the authors to conclude that "data diversity is more essential than data quantity".[10] Mixing in successful QT-Opt grasps from a different robot raised bin-picking success from 22% to 39%, with a 2% drop on the original tasks.[10]
DROID. The authors co-trained diffusion policies with each batch split 50/50 between in-domain demonstrations and either DROID or a curated Open X-Embodiment subset of about 400K trajectories, across 6 tasks in 4 locations with 10 rollouts per setting. DROID beat the next-best method by 22 points of absolute success in distribution and 17 points out of distribution.[5] In a separate test, a 7,362-trajectory random sample of DROID beat an equally sized subset drawn from 20 scenes on the out-of-distribution tasks, "suggesting the scene diversity of DROID is one of the driving factors".[5]
BridgeData V2. The authors found "performance improves with both model size and dataset size and diversity", and list as limits that tasks are "generally low-precision" and that the data was "collected entirely at a single institution".[7]
Our reading is that across these papers, variety in tasks and scenes helped more than repetition, and data from other robots helped when the model had the capacity to use it. But every result comes from a handful of robots, tasks and trials chosen by the authors. None of it is a scaling law. Our explainer on how robot policies are evaluated covers why such success rates are hard to compare.
How to read a robot-data claim (our checklist)
- What is the unit? Trajectories, episodes, demonstrations, hours or camera streams, and at what control frequency.
- Which version and subset? Paper, project page and dataset sheet can disagree, as the Open X-Embodiment and DROID figures above show.
- Are failures included? DROID releases about 16k unsuccessful episodes alongside its 76k successful ones, and BridgeData V2's scripted policy "fails frequently".[5][7]
- How many robots, and how much data each? Count of embodiments says little if one or two entries dominate the episodes.
- Better than what? Open X-Embodiment's 50% is stated for the small-data domain; DROID's 22 points are absolute success over 10 rollouts per setting. Ask for trials, tasks and baseline.
- Which licence covers which part? DROID and BridgeData V2 state CC BY 4.0. An aggregate's top-level licence statement does not, on its own, tell you the terms each contributing lab set, so check each source.[5][8][3]
For the models trained on this data, see our explainer on vision-language-action models. More coverage is on the robotics and research topic pages.
