JEPA

JEPA stands for “Joint Embedding Predictive Architecture”. It is a blueprint for programs that learn something about the world by themselves from images or videos. The idea behind it: the program sees part of an image and is supposed to guess what the hidden rest looks like. However, it does not paint out this rest. Instead, it merely predicts roughly what is to be seen there — in the form of a highly condensed description. This approach was proposed by the French researcher Yann LeCun, who for a long time led AI research at Meta.

LeCun’s objection to language models

Today’s chatbots are trained by predicting the next word of a text at each step. This works astonishingly well for language. With images and videos, however, this recipe often fails. Anyone tasked with predicting the next video frame pixel by pixel wastes their entire computing power on details that nobody can know: the exact shape of every leaf, the noise in the shadow, the color of a car on the horizon.

LeCun has argued for years that models therefore fail to build genuine understanding of the world. They learn surfaces instead of relationships. A human, by contrast, knows that a released ball will fall — without imagining exactly what every single pixel looks like in the process. JEPA is meant to hit exactly this level: predictions about the essentials, not about the surface.

This debate matters because it represents a strategic bet. A large part of the industry is investing billions in ever larger language models. LeCun considers this path a dead end on the way to machines with common sense. JEPA is his counter-proposal, and whether it will pay off remains open.

Predictions in the space of meanings

The core of the method is a so-called encoder. This is a part of the network that translates an image into a long list of numbers. This list is called an embedding and compactly describes the content: for example “dog, from the side, in motion, on grass”. Fine details are deliberately lost in the process.

During training, the system receives two crops of the same image. One crop is visible, the other is hidden. Both are passed through encoders. An additional component, the predictor, is now supposed to guess the embedding of the hidden part from the embedding of the visible part. So what is compared are two lists of numbers, never two images.

A comparison makes the difference clear. A reconstructing model must completely redraw a hidden face. A JEPA only has to say: “There is a face looking to the left.” This is a much more honest task, because it does not even demand unpredictable details in the first place. The best-known pitfall is called collapse: the model could cheat and output the same list of numbers for every image, in which case the prediction is always correct. Technical safeguards are used to counter this, such as a slightly delayed copy of the encoder.

From research paper to robotic arm

In practice, JEPA is encountered mainly in publications by Meta. Well known are I-JEPA for single images and V-JEPA for videos, published since 2023. The trained models are freely available and are being reused by research groups worldwide. A finished product for end users does not yet exist.

The practical benefit lies in preparation. A JEPA learns without labeled data, that is, without humans manually tagging thousands of images with keywords. After that, just a few examples suffice to adapt it to concrete tasks — such as recognizing actions in surveillance videos or in sports footage.

The approach is particularly interesting for robotics. A robot must estimate what happens when it pushes an object. Exactly such short-term predictions about the near future are the promise of JEPA. In news articles, the term therefore usually appears in connection with “world models”, meaning AI systems intended to internally represent physical relationships.