Elo Rating

The Elo rating is a number that indicates how strong someone is in direct comparison with others. It doesn't arise from a test with fixed tasks, but from many duels. After each duel, the winner gains points and the loser loses exactly as many. What matters is how surprising the result was: whoever wins against a much weaker opponent gets barely anything, whoever beats a much stronger one gets a lot. The system was invented by the Hungarian-American physicist Árpád Élő for chess. Today the same calculation is also used to compare computer programs that write texts or generate images.

Why chatbots don't get a school grade

For many tasks, performance can be measured easily. An arithmetic test has right and wrong answers, you count the hits and you're done. With a text program, it's more difficult. Whether an explanation is good, friendly, or understandable can't be objectively checked off.

This is exactly where comparison helps. People are bad at saying whether an answer deserves 78 or 84 out of 100 points. But they are very good at saying which of two answers they prefer. From thousands of such individual judgments, the Elo rating builds a ranking.

That's why Elo numbers now appear in almost every report about new AI models. Companies advertise that their system sits a few points ahead of the competition in a ranking. But anyone who understands the number also recognizes its limits: it measures what test subjects prefer, not what is factually correct. A model that phrases things beautifully while making mistakes can still end up on top.

The math behind the points

Before each duel, the system calculates an expectation. A win probability results from the difference between the two scores. A lead of 100 points roughly means a 64 percent chance of winning, a lead of 400 points about 91 percent. The score alone therefore says nothing; only the gap to others is meaningful.

After the duel, the system compares expectation and outcome. If the result was as predicted, almost nothing changes. If it was surprising, the numbers shift noticeably. How strongly is governed by a factor often called the K-factor. A large K-factor makes the values fluctuate quickly, a small one makes them sluggish but stable.

With AI rankings, this works as follows: a tester asks a question and receives two answers from two models, without knowing which one wrote which. They choose the better one. This result counts like a chess game. The starting value is usually 1000 or 1500 points, after which each model settles at its own level.

From the chess ranking to the AI arena

In chess, the Elo number has been the official standard since the 1970s. Club players are often between 1200 and 1800 points, grandmasters above 2500. Many computer games with online rankings also use this principle or a further development of it to put together fair opponents.

In the AI world, the method became known through public comparison platforms where anyone can vote. Millions of such votes produce rankings that are cited in trade articles and marketing materials. Sometimes the top spots are separated by only ten points. Such a small gap often falls within the range of statistical uncertainty and effectively means a tie.

A common misconception is to treat the Elo number as a measure of intelligence. It only describes performance against the specific competition under the specific test conditions. If the field of participants or the type of questions changes, the values shift. It is nonetheless useful because it condenses a great many individual judgments into a single, comparable number.