A vision-language-action model (VLA) is a model that takes a camera image and a natural-language instruction and produces robot actions. The idea is to start from a vision-language model that has already learned from web-scale images and text, then teach it to control a robot, so that the robot inherits general knowledge instead of learning each task from scratch. The term comes from the RT-2 paper, and three papers since then show how the approach was built, opened up and changed.[1][2][3]
The problem VLAs try to solve
Classic robot learning trains a policy for a task on data collected for that task. A policy that folds one kind of towel on one kind of table tells you little about a different towel. Language models and vision models went the other way: pretrain on very large, varied data, then adapt. A VLA applies that recipe to control. The pretrained model already recognises objects and understands instructions, and the robot data supplies the missing piece, which is how to move.
This is a research direction, not a finished product category. For the parallel in software agents, where a model chooses actions through tools, see how AI agents work, and how to secure them. A VLA differs in that its actions are physical and its feedback loop runs on a camera.
RT-2: actions as text
The RT-2 paper argues that vision-language models trained on web-scale data can be used directly for end-to-end robot control. Its authors co-fine-tune such models on robot trajectory data together with internet-scale vision-language tasks.[1] The key design choice is in the paper's own words: they "express the actions as text tokens and incorporate them directly into the training set."[1] A robot action becomes a short string of tokens, so the same model can answer a question about an image or emit a movement, depending on the prompt.
The authors report 6,000 evaluation trials, improved generalisation to novel objects, and the ability to interpret commands that were not in the robot training data. They also report that adding chain-of-thought reasoning lets the model handle multi-stage semantic tasks, with the example of choosing a rock as an improvised hammer.[1]
OpenVLA: an open version
The OpenVLA authors say existing VLAs were hard to adopt because they were closed and because earlier work had not shown efficient ways to fine-tune them.[2] Their model is a 7B-parameter open-source VLA trained on 970k real-world robot demonstrations, built on a Llama 2 language model with a visual encoder that combines DINOv2 and SigLIP features.[2]
The reported comparisons are the useful part:
- It outperforms the closed RT-2-X (55B) by 16.5% absolute task success rate across 29 tasks, with 7x fewer parameters.[2]
- When fine-tuned, it outperforms from-scratch imitation learning methods such as Diffusion Policy by 20.4%.[2]
These are the paper's own figures, from the authors' evaluation tasks. They are strong evidence that a small open VLA is competitive on those tasks. They are not a guarantee on yours.
pi-zero: changing the action output
The pi-zero paper keeps the starting point, a pretrained vision-language model, to "inherit Internet-scale semantic knowledge." What it changes is how actions come out: the authors "propose a novel flow matching architecture built on top of a pre-trained vision-language model."[3] They train on data from multiple dexterous robot platforms, including single-arm robots, dual-arm robots and mobile manipulators, and evaluate zero-shot ability after pretraining. The tasks listed include laundry folding, table cleaning and assembling boxes.[3]
Our interpretation is that this marks a design split in the field. Text-token output is simple and reuses language model machinery, while a dedicated action-generation head can be a better fit for smooth, high-frequency motion. The pages we read do not compare the two designs head to head, so treat that as a reading of the architectures, not a measured result.
Shared data: Open X-Embodiment
A model needs varied robot data, and one lab owns only so many robots. The Open X-Embodiment collaboration assembled a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160,266 tasks). Its high-capacity model, RT-X, "exhibits positive transfer and improves the capabilities of multiple robots."[4] The paper's argument is that robot learning could follow language and vision toward general pretrained models.[4]
What the papers do not show
- Different robots, different tests. Each paper evaluates on its own robots and tasks, so the numbers above cannot be compared across papers.
- Success rates are task-level. A higher success rate on a defined task set says little about reliability, safety or cost in open-ended work.
- No claim of general household or factory capability. None of the abstracts we read make one, and we do not either.
- Rapid change. This field moves quickly, so check current papers before relying on any of these specifics.
For why headline numbers deserve scepticism, see how to read an AI benchmark claim. More on this beat lives on the robotics topic page.
