MTEB

Some AI models don’t answer questions but instead translate texts into long lists of numbers. These lists of numbers are called embeddings, and they work like a location marker on a map of meaning: texts with similar content end up close together. This can be used, for example, to build a search that still finds something even when the user uses different words than the text being searched for. MTEB is a public test that makes such models comparable. The abbreviation stands for Massive Text Embedding Benchmark, roughly “huge test for text number lists”. The test consists of over a hundred individual tasks, and at the end each model receives a score.

Why a single score decides millions

Anyone building a company search, a chatbot with document access, or a product recommendation needs an embedding model. There are hundreds of them, from large providers and from the open-source community. Without a shared test, every company would have to spend weeks experimenting on its own. MTEB delivers a first answer in minutes.

Before MTEB, almost every provider tested its model on self-chosen tasks. Naturally, every model looked good under those conditions. A shared test with fixed rules makes marketing claims verifiable. That’s why the MTEB ranking now appears in almost every announcement of a new embedding model.

However, the limits of the test matter. A high score says nothing about speed, price, or memory requirements. A model ranked first can be ten times as expensive as one ranked twelfth. In practice, developers therefore often choose a smaller model with a slightly lower score.

What tasks make up the score

MTEB consists of several task types. In search, the model receives a query and a large pile of texts. What’s measured is whether the matching texts end up at the top. In clustering, the model is supposed to sort texts into meaningful topic groups without any guidance.

Other tasks test similarity. Two sentences are presented, and humans have previously rated their similarity on a scale. The model must come as close as possible to this human judgment. Still other tasks test classification, for example whether a product review is positive or negative.

In the end, the results of all tasks are averaged into a single score. This is convenient, but it’s also the biggest weakness. A model can be weak at search and strong at clustering, and the average hides that. Anyone who needs a model for search should therefore only look at the search tasks. The results are available individually; you just have to expand them.

The leaderboard and its pitfalls

MTEB is best known for its ranking on the Hugging Face platform, a kind of large online warehouse for freely available AI models. Anyone can have their model tested and added to the list. The list changes almost weekly. When tech news reports that a new model is “leading in embeddings”, this ranking is usually what’s meant.

Originally, MTEB was strongly focused on English. Meanwhile, there are extensions for German, French, Chinese, and many other languages. For German-language applications, it’s worth looking at these sub-rankings, since an English front-runner can drop significantly in German.

A typical mistake is confusing it with tests for language models like ChatGPT. MTEB does not measure answer quality or logic. It only measures how well meaning is translated into numbers. Another pitfall: because the test data is public, providers can specifically train their models on it. Experts call this overfitting, and it’s a reason to never take leaderboard rankings as sole proof.