Perception-Language-Action

A Perception-Language-Action model – PLA model for short – combines three capabilities in a single system: perceiving, understanding, and acting. The model sees its environment, for example via a camera. It understands language commands, such as “Clear the table.” And it directly controls the motors of a robotic arm. In the past, three separate programs handled these tasks one after another. A PLA model does everything in a single step.

Why a shared model makes a difference

When three separate systems are chained together, errors propagate. The vision system misidentifies an object, the language system then plans the wrong action, and the robot grabs the wrong spot. Every interface between two systems is a new source of error.

A PLA model processes all three levels together. Meaning and context from the language command can directly influence how the model interprets the camera image – and vice versa. The result is more robust than any chain of individual parts.

On top of that comes flexibility. A PLA model can respond to new commands that never appeared during training, because it understands language and situation together. A classic control program, by contrast, only knows the commands someone explicitly programmed into it.

From camera image to movement: the internal process

The model receives several inputs at the same time: images from the camera, sensor data about the robot’s current posture, and a language command as text. These different data types are translated into a shared internal representation – researchers call this a shared representation space. In doing so, the model learns to think of the camera image and the meaning of language in the same “language.”

From this shared representation, the model computes an output: concrete control commands for each motor. These commands are not abstract plans but direct numbers – how far joint two rotates, how firmly the finger grips. This final step is called the action head, the part of the model specialized in action.

One important difference from pure language models: PLA models must respond in real time. A chatbot is allowed to think for a second. A robotic arm that’s supposed to catch a glass doesn’t have that time. This places high demands on computing speed and model size.

PLA models in products and research

In 2023, Google's robotics research published RT-2, one of the best-known examples: a model that processes camera images and language commands together and directly derives robot movements from them. It was able to solve tasks it had never been explicitly trained on – for example, sorting an object by its color, even though it had only learned to sort by shape.

Companies such as Figure AI or 1X Technologies are also working with similar architectures for humanoid robots – that is, robots with a human-like body shape. Their goal is a robot that can be taught a new task through natural language, without having to be reprogrammed.

In public discourse, the term mainly appears in reports on industrial robotics and autonomous vehicles. Self-driving cars face a similar problem: they see the road, hear instructions from the navigation system, and must steer and brake – all at the same time and without delay. The PLA architecture is therefore not a niche topic in robotics research, but a building block of many technologies currently entering the market.