Intelligence Index

Anyone who wants to know which AI program is better faces a measurement problem. There are dozens of tests: math problems, coding tasks, knowledge questions, logic puzzles. One program wins at calculation, another at coding. The Intelligence Index solves this by combining many such tests into a single number. One program might then score 60 out of 100 points, another 45. It is, in a sense, an overall grade on a report card, formed from many individual grades.

What a single score does to the market

Such rankings help decide where money goes. Companies buying AI must choose from a dozen providers and don’t have time for their own tests. They look at the ranking. A provider that climbs a few spots gets mentioned in news articles and picks up new customers.

That’s why the scores show up immediately in marketing. When a company unveils a new model, the index number appears on slide two. Stock analysts also use it as an argument for whether an AI provider is technically ahead or falling behind. A number that is really just a test result thus becomes an economic signal.

This produces a well-known side effect. Anyone who knows what is being tested can optimize for it. Some providers train their models specifically on tasks that resemble the test tasks. The score then rises without the model actually getting better in everyday use. Experts call this overfitting to the test — comparable to a student who only memorizes old exam papers.

What tests make up the score

The foundation is individual standardized tests, known in the trade as benchmarks. Each consists of a fixed set of tasks with known solutions. The model works through all the tasks, after which the correct answers are counted. One well-known test contains difficult exam questions from university subjects, another contains real coding problems from software development.

The index takes several of these results and forms an average from them. The values are usually first brought onto a common scale, often from 0 to 100. Some operators weight individual tests more heavily if they are considered more informative. This selection and weighting is exactly where the subjective part lies: someone who counts coding tasks twice produces a different ranking than someone who favors knowledge questions.

It is important to distinguish this from user ratings. On leaderboards like the Chatbot Arena, people vote on which of two answers they like better. The Intelligence Index, by contrast, measures objectively verifiable correctness. Both methods regularly produce different winners, because likeable answers and correct answers are not the same thing.

Where the index numbers show up

The term is encountered most often in tech and financial news. Sentences like “the model achieves 70 points on the Intelligence Index, putting it ahead of the competition” have become standard. Charts plotting score against price per query almost always come from such index operators as well.

Within companies, the index serves as a preliminary filter. An IT department takes the three top-ranked models and then tests them on its own tasks. That makes sense, because the index says nothing about how well a model handles the specific documents of an insurance company or a government agency.

For private users, a simple rule of thumb remains. The score roughly shows which league a model plays in, nothing more. Differences of two or three points won’t be noticeable in chatting. A jump from 30 to 60 points, on the other hand, is clearly noticeable, for example in multi-step calculation tasks.