Key takeaways
- A VLA is a vision-language model retrained to output actions, such as robot arm moves or keypresses, instead of only text.
- RT-2 and OpenVLA turn each action dimension into one of 256 bins and write it as a token; π0 instead generates smooth 50-step action chunks with a flow-matching action expert.
- The main limits are speed, reliability (OpenVLA usually stays below 90% success) and skills that rarely transfer beyond the motions or games in the training data.
A VLA, or vision-language-action model, is an AI model that looks at camera images, reads an instruction and outputs actions: how a robot arm should move, or which keys a game agent should press. So what is a VLA model in practice? It is a vision-language model retrained so that its answers are motor commands instead of sentences.
The name comes from Google DeepMind's RT-2 paper of July 28, 2023. The idea has since spread to open models anyone can download, and this week to a miHoYo model that plays video games from raw screen pixels.
How a VLA differs from a VLM
A vision-language model (VLM) takes in images and text and writes text. Show it a kitchen and it describes the counter.
A VLA adds one more output. Ask it to "put the apple in the bowl" and it answers with numbers that move a gripper toward the apple. Each answer covers a fraction of a second: the model sees a new camera frame, produces the next action, and repeats until the task is done.
Starting from a VLM means the model already knows what an apple is and what "the smallest object" means, from web data. Robot data is scarce and expensive, so the robot doesn't have to learn every word and object from its own demonstrations.
How actions become tokens
Language models write one token, a short chunk of text, at a time. The first VLAs wrote actions as tokens too.
RT-2 describes an arm's action with eight numbers: six for how the gripper moves and rotates, one for how far it opens, and one for whether the task is finished. Each continuous number is cut into 256 bins, so it becomes an integer from 0 to 255, and the model writes the eight integers like a short line of text. PaLI-X, one of RT-2's base models, already has tokens for integers up to 1,000. PaLM-E doesn't, so the team overwrote its 256 least-used tokens and gave them new meanings as action bins.
OpenVLA, an open model from Stanford, UC Berkeley, Toyota Research Institute, Google DeepMind, Physical Intelligence and MIT, does the same, but its bins span the 1st to the 99th percentile of its training actions, so rare extreme values don't waste precision.
Physical Intelligence's π0 (pi-zero) skips discrete tokens. It attaches a 300 million parameter "action expert" to a 3 billion parameter PaliGemma VLM. The expert uses flow matching, a relative of diffusion that turns random noise into smooth continuous actions, and outputs 50 future steps at a time. The paper says this lets π0 control robots at up to 50 Hz, which matters for work like folding laundry.
What VLAs are trained on
Every VLA starts from a VLM trained on web images and text, then learns from recordings of a robot or a person doing tasks, each paired with its instruction.
| Model | Released | Size | Robot or action data |
|---|---|---|---|
| RT-2 (Google DeepMind) | July 2023 | 5B, 12B and 55B | 13 robots over 17 months in an office kitchen |
| OpenVLA | June 2024 | 7B | 970,000 robot trajectories from Open X-Embodiment |
| π0 (Physical Intelligence) | October 2024 | 3.3B | Over 10,000 hours, 7 robot configurations, 68 tasks |
| Gamer (miHoYo) | November 2026 | 8B and 35B (MoE) | Gameplay, about 1,000 annotated hours of Honkai: Star Rail |
RT-2 trained on robot and web data at the same time, which the paper calls co-fine-tuning. Robot data was about half of the mix for the PaLI-X models. OpenVLA took 64 A100 GPUs for 14 days. π0 fine-tunes on 5 to more than 100 hours of data for each hard task.
What the results show
RT-2. In settings the robot had never seen, the 55B model succeeded 62% of the time, against 32% for RT-1, Google's earlier robot model. On skills that appear only in web data, such as moving an object onto a particular number, it averaged 60% against 17%, over about 6,000 trials in total.
OpenVLA. Across 29 tasks on two kinds of robot, it scored 16.5 points higher in absolute success than the 55B RT-2-X, with about 7 times fewer parameters. A 4-bit version kept its success rate (71.9% against 71.3%) while cutting GPU memory from 16.8 GB to 7.0 GB.
π0. The team shows it folding laundry, bussing tables and building boxes. The paper says it beats OpenVLA and Octo "by a large margin" on tasks out of the box, in comparisons the π0 team ran itself.
Gamer: a VLA for video games
Gamer, from miHoYo's Honkai AI R&D team, was submitted to arXiv on November 13, 2026. It looks at raw screen pixels and outputs keyboard and mouse input, with no access to the game's internal state.
Its action tokens descend from RT-2's. Every 200 milliseconds it writes one text span: the mouse movement, then six groups of pressed keys, one per 33 millisecond slice, all as ordinary tokens in the Qwen vocabulary. Like RT-2, its pre-training mixes gameplay with general vision-language data. On seven kinds of in-game minigames, its average success rose from 15.0% after pre-training to 49.7% after fine-tuning and reinforcement learning.
On four public game benchmarks played in real time, the 8B Gamer averages 29.5 and the 35B version 31.5, according to the paper. GPT-5.6-Sol averages 17.6 and Claude Opus 5.0 17.3 in the same table, though both beat Gamer on one benchmark, GameWorld. The team's serving engine, Streaming-vLLM, cut the 35B model's time per step on an Nvidia H200 from 1,306 ms to 65 ms. The repository links the weights under Apache-2.0.
The limits of VLAs today
- No new motions from the web. RT-2's authors write that web data teaches new concepts, not new physical skills. Wiping and folding towels were among its failures.
- Speed. RT-2's 55B model ran at 1 to 3 Hz from cloud TPUs. OpenVLA runs at about 6 Hz on an RTX 4090. That is why π0 predicts 50 actions at once and Gamer needed a custom engine.
- Reliability. OpenVLA usually stays below 90% success, its authors note, and π0's paper says not every task works reliably.
- Narrow transfer. Gamer's best results are on held-out tasks inside the game it trained on most. More pre-training data stopped helping on distant, unseen games.
- Self-reported numbers. Every result above comes from the team that built the model.
Recent work targets these limits: PDRVLA adds a small physics module to a robot VLA. The thing to watch is whether open checkpoints, like OpenVLA and π0 in Physical Intelligence's openpi, lead to independent tests on robots their makers never trained on.
Frequently asked questions
What does VLA stand for in AI?
VLA stands for vision-language-action. The term comes from Google DeepMind's RT-2 paper of July 2023, which described models that take in images and text and output robot actions as tokens.
What is the difference between a VLA and a VLM?
A VLM reads images and text and writes text. A VLA starts from a VLM and is trained on recordings of tasks so it also outputs actions, such as gripper movements or keyboard and mouse input.
Are there open-source VLA models?
Yes. OpenVLA, a 7B model, released its checkpoints and code, and Physical Intelligence publishes π0 checkpoints in its openpi repository under Apache-2.0. Gamer's repository links its game-playing model weights on Hugging Face.
What is a VLA used for in robotics?
VLAs control robot arms and mobile robots from camera images and spoken or typed instructions. The papers show tasks such as picking and placing objects, folding laundry, bussing tables and building boxes.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlarXiv (Google DeepMind)
- RT-2: New model translates vision and language into actionGoogle DeepMind
- OpenVLA: An Open-Source Vision-Language-Action ModelarXiv
- OpenVLA project pageOpenVLA team
- π0: A Vision-Language-Action Flow Model for General Robot ControlarXiv (Physical Intelligence)
- openpiPhysical Intelligence
- Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game AgentarXiv (miHoYo)
- Gamer on Hugging Face PapersHugging Face
- Gamer repositoryGitHub



