Vision-Language-Action

A robotic arm is supposed to place an apple from the table into a bowl. In the past, this required three separate programs: one that finds the apple in the camera image, one that interprets the sentence, and one that controls the motors. A Vision-Language-Action model does all of this in one piece. It continuously receives images and the task as text, and directly outputs control commands, for example how far each joint should rotate. The three words in the name stand for exactly these three parts: vision, language, action. Such models are the core of what is currently being developed as the foundation for human-like robots.

Why robots have so far failed at the real world

Industrial robots have been very good at a single task for decades. They weld the same seam millions of times, but always in the same spot. As soon as a workpiece shifts by two centimeters, the grip misses. The movements are hard-programmed and don’t really know the environment.

Vision-Language-Action models are meant to replace this rigid behavior. They learn from many examples what a task typically looks like, instead of running through a fixed sequence. That’s why such a model can also handle an apple it has never seen before. Experts call this transfer to new situations generalization.

Economically, this is really the point. A robot that can be told tasks in plain sentences doesn’t need expensive reprogramming for every variation. Companies like Tesla, Figure, or Google are investing billions in the technology for this reason. Whether the models will become reliable enough for continuous operation, however, remains an open question.

From camera image to motor command

The foundation is usually an already-trained model that understands images and text together. Such models have learned from millions of photos with captions what objects look like and what they’re called. This knowledge is valuable: the model already knows what an apple is before it has ever controlled a robotic arm. All that’s left to teach it is how to derive movements from this understanding.

For this, it is further trained with recorded robot operations. Humans guide the arm through the task via remote control, while camera images and joint positions are recorded. From such recordings, the model learns the relationship between image, task, and the next sensible movement. You can think of it like a driving student who learns by watching and imitating rather than through rules.

In operation, this then runs in a fast loop. The model sees an image, outputs a small movement step, sees the new image, and corrects. This happens several times per second, often between ten and fifty times. This is exactly where a major difficulty lies: large models compute slowly, but an arm that reacts too late knocks the apple over.

Humanoids, warehouses, and the limits of demo videos

In the news, Vision-Language-Action models mostly appear in connection with humanoid robots. These are machines with two arms and two legs that fold laundry or sort boxes. Pilot projects are also underway in warehouses and manufacturing. Google released one of the first well-known models of this kind with RT-2, and numerous successors now exist.

Skepticism is warranted when watching the videos. Many demonstrations take place in tidy rooms, on the twentieth attempt, and sometimes in fast-forward. The difference between a successful demonstration and a device that works reliably eight hours a day is enormous. Another common misconception is that a language model in a chat is already nearly a robot control model. Text can be corrected afterward; a wrong arm movement cannot.

The biggest bottleneck right now is data. Text is practically unlimited on the internet, but recordings of real robot activities have to be laboriously generated. That’s why research groups now collect their recordings in shared datasets and partly train in simulations. How well what is learned in simulation transfers to real hardware is among the open questions of the field.