Teaching to the Test

The term originally comes from schools. If a teacher knows which types of tasks will appear in the exam, she can have students practice exactly those. Grades then rise without the class having actually understood the subject better. The very same thing happens with artificial intelligence, that is, with computer programs that learn from examples. Such programs are evaluated using standardized sets of tasks. Whoever knows these tasks and optimizes specifically for them gets high scores — even if the program performs worse in real-world use.

Why top scores on leaderboards prove little

AI companies almost always advertise with numbers. A new model achieves 92 percent on a math test, another one beats it with 94 percent. Such comparison tests are called benchmarks. They are publicly accessible so that all providers can use the same tasks. But this very openness is also the problem: what is public can be deliberately trained on.

For investors and journalists, this is tricky. A benchmark number seems objective, like a measured top speed of a car. In reality, it’s more like a lap time on a track that the manufacturer knew in advance and practiced on for weeks. Two models with almost identical scores can differ significantly in everyday use. That’s why experts warn against basing investment decisions solely on leaderboards.

The effect also has a flip side in the other direction. A model can be very useful in everyday use and still perform only mediocrely on tests. Anyone who only looks at the table overlooks such cases. The number doesn’t measure the capability itself, only the performance on a particular set of tasks.

How test tasks leak into training

The most harmless way is by chance. Large language models are trained on huge amounts of text from the internet. Many benchmark tasks are posted online somewhere, complete with solutions, for example in forum posts or on programming platforms. If they end up in the training material, the model has simply memorized the answers. Experts call this data contamination, meaning the contamination of training data with exam material.

There are also deliberate variants. A provider can compile training data so that it closely resembles the types of tasks in the test. Formally, the test hasn’t been copied, but the result is still artificially inflated. Between clean preparation and cheating lies a wide gray area here. That’s precisely why the accusation is hard to prove and keeps coming up regularly in the industry.

There are several countermeasures. One can keep part of the tasks secret and never publish them. One can create tests after a model’s release, so that the tasks can’t possibly be in the training data. Or one can slightly reword known tasks: if only numbers and names are changed and the score collapses, the knowledge was apparently just memorized. This trick is a common test for teaching to the test.

From school report cards to product announcements

You’ll most often encounter this term in reports about new AI models. After every major announcement, experts discuss whether the stated figures hold up. Independent retests often follow, in which a model performs worse than advertised. The gap between announcement and verification is a good warning sign for readers.

Outside of AI, you know the principle from the education system. After the PISA studies, some school systems aligned their teaching heavily toward the tested competencies. Something similar happens in companies when employees only improve the metric by which they are measured. Behind this lies a general rule: as soon as a metric becomes a goal, it no longer works as a metric.

In practice, a simple reflex helps. Whenever you see an impressive number, ask who devised the test and whether the tasks were known beforehand. And try a model out on your own, unusual task. What works on a question you made up yourself is something no one could have practiced in advance.