Action Prediction

A human often can tell what another person is about to do. Someone reaches for the door handle, so they’re about to open the door. Someone looks over their shoulder and switches on the turn signal, so they’re about to change lanes. Action Prediction refers to computer methods that replicate exactly that: they observe a situation and predict the next action before it happens. The input is usually video footage, sometimes also sensor readings, such as speed or distance. The result is not certainty, but an assessment expressed as probabilities.

The time advantage as the actual benefit

Many technical systems are only safe if they react early enough. At 50 km/h, a car needs about 14 meters to come to a complete stop after braking. If it only detects a pedestrian once they’re on the road, it is often too late. If, however, it predicts that the person is about to step off the curb, it gains a second. That second is the whole point of the exercise.

The time advantage also matters in human-machine collaboration. A robot in a workshop is supposed to hand over a tool while the person is still reaching out their hand. If it waits until the movement is complete, it appears sluggish and useless. Prediction turns a merely reactive device into one that thinks along.

It is important to distinguish this from mere recognition. Action recognition answers the question: What is happening right now? Action Prediction answers: What will happen next? The first task looks backward and is therefore considerably simpler. The second works with incomplete information and can, in principle, never be entirely certain.

Inferring the future from movement patterns

The foundation is almost always training with a very large number of examples. A program is shown millions of video clips in which actions unfold. For each clip, what happened afterward is recorded. The program searches on its own for regularities between the beginning and the continuation. It can then estimate a continuation for new, unfamiliar scenes.

The temporal context is decisive here. A single image reveals little, whereas a sequence of images reveals a lot: direction, speed, acceleration, gaze direction. Modern systems therefore process entire sequences and weigh which details are currently important. They often predict several possible continuations simultaneously, each with a probability value. A car then plans not for a single course of events, but for the riskiest among several.

A common misconception is that the system calculates the future. It merely computes what mostly followed in similar situations. In the case of unusual behavior, it is therefore often wrong. And it inherits the biases of its training data: anyone who only collects videos from cities with wide streets ends up with a model that estimates poorly in narrow old-town alleys.

From assistance systems to surveillance cameras

Action Prediction is most visible in road traffic. Emergency braking assistants and driver assistance systems constantly estimate where pedestrians, cyclists, and other cars are moving. Manufacturers such as Tesla, Waymo, or Mobileye advertise exactly this in their presentations. In technical texts, this is often referred to as trajectory prediction, that is, predicting the future path of movement.

There are also applications in factories, in care settings, and in sports. Robotic arms adjust their speed to match a person’s hand movement. Fall detection in hospitals should ideally announce a fall, not just report it after the fact. Analysis software in soccer estimates from a player’s running direction where a pass will be played.

Its use in video surveillance is discussed critically. There, software is meant to predict suspicious behavior, such as theft in a store. Such systems infringe on fundamental rights and are frequently wrong about individuals. The European AI Act therefore classifies parts of these applications as high-risk or bans them outright. When action prediction is debated in business news, it is usually these borderline cases that are at issue.