LMArena

LMArena is a freely accessible website for comparing AI chatbots. You type in a question and receive two answers from two different systems. At first, you don’t find out which systems answered. You simply choose the answer you think is better, and only afterward are the names revealed. From many millions of such votes, the site calculates a leaderboard. The project originated at the University of California, Berkeley, originally under the name Chatbot Arena.

Why the leaderboard makes the industry nervous

Until recently, AI systems were tested almost exclusively with fixed test tasks. Such tests consist of catalogs of questions with known model solutions, for example math problems or multiple-choice exams. The problem: these questions eventually end up on the internet, and the systems learn from the internet. A high score can then mean that a system has already seen the solutions.

LMArena avoids this because the questions come from real users and are constantly new. Something else is also being evaluated: not the formally correct solution, but which answer people actually find more helpful. This is closer to everyday life, where there is usually no single correct answer.

The stakes are correspondingly high. When a company releases a new model, its position on the Arena leaderboard is often the first headline. Stock prices and investor sentiment react to it. Precisely for this reason, the site also faces criticism: companies can secretly test many preliminary versions and only let the best one compete publicly.

From duel to score: the Elo principle

The leaderboard is based on a method from chess, the Elo system. Every participant has a score. Whoever wins a duel takes points away from the loser. How many depends on who was considered the favorite: a win against a highly rated opponent yields a lot, a win against a weak one yields little.

A single vote says almost nothing, because taste fluctuates. Only after tens of thousands of duels per model do the values stabilize. LMArena therefore always states a margin of fluctuation, for example plus/minus five points. If two models lie within this range, they are practically equally strong, even if one ranks higher in the table.

A well-known weak point is the tendency toward attractive packaging. People prefer long, friendly-worded, well-structured answers. Most voters do not check whether the content is actually correct. The operators try to factor this out and additionally offer sub-leaderboards, for example for coding, mathematics, or individual languages.

What the Arena ranking means for users and investors

In news reports, LMArena is usually encountered in a sentence like: The new model takes the top spot on the Arena leaderboard. In presentations by Google, OpenAI, Anthropic, or Chinese providers such as DeepSeek and Alibaba, the ranking is also shown as proof of progress. Anyone reading such reports should check the point gap and not just the order.

You can use the site even without voting. You can pit two systems against each other on purpose and see for yourself how differently they respond to the same question. This is one of the simplest ways to directly compare free and expensive models.

A common mistake is to treat the leaderboard as a quality seal. It measures what volunteers prefer in short conversations. It does not measure reliability over long tasks, safety, data protection, or operating costs. For a purchasing decision within a company, the Arena ranking is therefore only one of several reference points.