Robotics Foundation Model
A robotics foundation model is a large AI model that has learned from vast amounts of motion, image, and language data to control robots. Instead of being reprogrammed for every single task, such a model is meant to handle many different tasks and often various types of robots as well.
Robots were long built and set up for exactly one task. A welding robot in a car factory repeats the same movement millions of times, but can do nothing else. A robotics foundation model takes the opposite approach. It is a very large computer program that has learned from enormous amounts of data, namely from videos of robotic arms, from camera images, and from text instructions. From this data, it derives which movement fits which situation. A human says in ordinary language what should happen, and the model generates from that the individual control commands for the motors.
Why robots have failed at every new task so far
The classic approach is expensive. For every new activity, someone has to define the motion sequences, test them, and fine-tune them. If the workpiece shifts by two centimeters, the robot grabs the wrong spot. This is precisely why industrial robots almost always stand behind barriers in a strictly ordered environment. A messy kitchen or a full warehouse is an unsolvable problem for them.
Foundation models are meant to solve this problem the way it has already worked for language. A chatbot was not programmed individually for every possible question. It learned general patterns from a very large amount of text and therefore also manages new questions. Applied to robots, this means: a model that has learned grasping itself should also be able to grasp a cup it has never seen before.
Economically, this is the real appeal. If robots can take on new tasks without months of setup, they become worthwhile even for small businesses and for fields like care work or logistics. That is why billions of dollars are currently flowing into companies working on such models. Whether the technology will deliver on its promise, however, remains an open question.
From camera to motor command
At its core, the model processes three kinds of inputs. It sees camera images of the current scene. It reads an instruction in everyday language, such as the sentence: Put the cup in the sink. And it knows the current position of its own joints. From this, it calculates many times per second how the joints should move next. Because of this combination, these are also called vision-language-action models, that is, models for image, language, and action.
The training data comes largely from humans. An employee guides the robotic arm through a task via remote control, and all movements are recorded. Such recordings are called demonstrations. Several research groups have pooled their data to reach millions of such episodes. Training also takes place in simulations, that is, in physically modeled virtual worlds where thousands of runs can happen in parallel.
This is also where the greatest weakness lies. Text is practically unlimited on the internet, but real robot data is not. Every demonstration costs time, hardware, and personnel. On top of that comes the so-called reality gap: what works in simulation fails in the real world due to friction, lighting, or a slippery grip. Another misconception is the assumption that such a model understands physics. It predicts statistically probable movements, and it regularly gets them wrong.
Humanoids, warehouses, and headlines
Robotics foundation models are most visible in humanoid robots. Companies like Figure, Tesla with Optimus, or Agility show videos of machines sorting boxes or folding laundry. Chipmaker Nvidia offers a foundation with its GR00T model family that other manufacturers can adapt. Google DeepMind is working on the RT and Gemini Robotics series, while the start-up Physical Intelligence is developing open models for many types of robots.
In everyday life, this technology is barely encountered so far. Realistic settings are warehouses, factories, and test labs, where mistakes cost little. When news talks about embodied AI, it is usually referring to exactly this topic: AI that doesn’t just write text but moves things in the physical world.
When reading such reports, a critical question is worth asking. Is the robot in the video operating autonomously, or is a human controlling it remotely? And is the speed real or sped up? Both are frequently left unmentioned in demonstration videos. The gap between an impressive clip and a product that works reliably eight hours a day is still quite large.