SimpleQA

SimpleQA is a collection of around 4,300 short factual questions used to test computer programs that write text. Such programs can formulate a fluent answer to any question, even if they don’t actually know the facts. That is exactly what SimpleQA is meant to expose. Each question has only a single correct answer that can be looked up in a reliable source. One example would be: In what year was a particular person first elected to office? Because the answer is so brief, it can be clearly scored as right or wrong. The company OpenAI published the test in the fall of 2024 and made it freely available.

Making made-up facts visible

The biggest practical problem with today’s language programs is not a lack of language ability, but fabricated content. Experts call this hallucination: the model produces a statement that sounds plausible but is factually wrong. For an essay, that might be annoying. In a doctor’s office, a law firm, or a bank, it can become expensive. That is why measurements are needed that specifically capture this weakness.

SimpleQA is deliberately built to be hard. The questions were chosen so that the strongest models at the time often answered them incorrectly. At the time of release, the best systems scored well below half of correct answers. A test that everyone passes immediately would be useless, because it would show no differences.

It is also important to note what SimpleQA does not measure. The test says nothing about programming ability, mathematics, or the understanding of long texts. It measures factual knowledge on short questions. Anyone who infers general intelligence from a good SimpleQA result is overstretching the claim.

Three ratings instead of just right and wrong

The procedure is simple. The model being tested receives a question without any aids and answers in a few words. A second program then compares the answer to the stored model solution. This grader is itself a language model, because rigid text comparisons would fail on minor details. “1961” and “in the year 1961” should, after all, both count as correct.

Ratings fall into three categories: correct, incorrect, or not answered. This third category is the actual trick. A model that honestly says it doesn’t know the answer when uncertain is not treated the same as a model that guesses. This makes it possible to separate knowledge from overconfidence.

From these values, two metrics are calculated. One is the share of correct answers out of all questions. The other looks only at the cases in which the model answered at all. A system can be mediocre on the first number and still very reliable if it scores strongly on the second. To ensure the questions were truly unambiguous, several independent people had to arrive at the same answer during creation.

SimpleQA in model announcements

You mainly come across the name when a company introduces a new model. In the tables of such announcements, SimpleQA usually appears alongside tests for mathematics and programming. Companies like OpenAI, Google, and Anthropic use it to demonstrate that their new system makes things up less often. Trade articles and financial reports also cite these figures.

A related term is benchmark, meaning a standardized comparison test for AI systems. SimpleQA is one of many, and such figures should be read with caution. A well-known pitfall is called contamination: if the test questions accidentally end up in a model’s training data, good results are worthless. Also, the test applies without internet access. An assistant that is allowed to search the web in everyday use often performs better in practice than its SimpleQA score would suggest.