Vision-Language-Action Model

A vision-language-action model controls a robot. It receives three things: images from a camera, a sentence in plain language, and information about how the robot is currently positioned. From this, it calculates how the arm or the chassis should move in the next moment. If you tell such a robot “Put the cup in the sink,” it looks for the cup in the camera image and reaches for it. The name describes exactly these three parts: Vision for seeing, Language for language, Action for movement. Experts usually abbreviate the term as VLA.

Robots that are no longer hard-programmed

Classic industrial robots execute fixed, pre-programmed movements. An engineer sets each position down to the millimeter. This works excellently as long as nothing changes. If the workpiece is shifted two centimeters, the robot grasps at empty air. For a factory floor this is manageable, for a cluttered kitchen it is not.

This is exactly where the appeal of VLA models lies. They are meant to handle situations that no one has described beforehand. The goal is a robot that you simply tell a task to, instead of programming it. In robotics, this long-unsolved problem is called generalization: the ability to transfer what has been learned to new objects and spaces.

That is why a lot of money is flowing into this area. Companies like Google DeepMind, Nvidia, Tesla, and Figure are working on such systems, often in connection with humanoid robots. The economic idea behind it is big: language models have transformed office work, VLA models are supposed to do the same for physical labor. Whether this succeeds is still open. Previous demonstrations mostly take place in controlled environments.

From camera image to joint angle

The foundation is usually an already fully trained model that processes images and text together. Such models have learned from millions of photos with captions what a mug, a handle, or a drawer is. This prior knowledge is crucial: it comes from the internet, not from a laboratory. Without it, every robot would have to learn about the world from zero.

Then comes the decisive conversion. The model is further trained with recordings of real robot movements. Humans guide a robot arm by remote control through tasks while cameras record everything. From such recordings, the model learns which movement fits which situation. The output is no longer a text answer, but a sequence of numbers: positions, angles, gripper state.

In operation, this runs as a fast loop. Capture image, calculate a short segment of movement, execute it, capture new image. This repeats many times per second. A common misconception is that the model plans the entire task through once. In fact, it constantly corrects itself based on what the camera is currently showing. The biggest bottleneck is the training data: text is practically unlimited on the internet, but recordings of robot grasps have to be laboriously produced one by one.

Where these systems appear today

In everyday life, one rarely encounters VLA models so far. They are found in research robots, in warehouses under test operation, and in demonstration videos from manufacturers. Well-known examples are Google’s RT-2, the open model OpenVLA, and Nvidia’s GR00T series for humanoid robots. Car manufacturers are also experimenting with similar approaches for driving.

In the news, these systems usually appear in connection with humanoid robots and large funding rounds. When reading about them, a critical question is worthwhile: is this a previously rehearsed sequence or a genuinely new task? The difference is enormous and is rarely made clear in promotional videos. Demonstrations in which someone spontaneously places an unfamiliar object on the table are the ones that carry real weight.

The term should be distinguished from a pure language model like ChatGPT, which only generates text. Likewise from a classic image processing system, which recognizes objects but controls nothing. What is special about the VLA model is the continuous connection from image all the way to movement within a single trained system.