Ground Truth

Anyone who wants to check whether an answer is correct needs something to compare it against. That is exactly what ground truth is: the information that counts as correct. In a photo, for example, that might be the note “a dog can be seen here,” verified by a human. For a weather forecast, it is the weather that actually occurs later on. So one compares a machine’s output with this fixed solution and counts the matches. The English term literally means “truth on the ground” and comes from land surveying: satellite images were checked against measurements taken on site.

No measurable progress without verified answers

A program that learns from examples needs those examples paired with the respective correct solution. Otherwise it doesn’t know in which direction to improve. The same applies to later evaluation. Statements like “the model recognizes 94 percent correctly” only make sense if someone has previously defined what “correct” means.

That is why the quality of this reference is often more important than computing power. If five percent of the examples are mislabeled, the learning system adopts these errors. It treats them as the rule and reliably repeats them. In practice, many apparent model errors actually lie in the data.

A second problem is bias. If faces were labeled predominantly by light-skinned people, recognition performs worse for other people. The model hasn’t become “unfair” — it has simply learned exactly what was presented to it as the truth.

How the correct answer even comes into being

In the simplest case, humans label the data by hand. They draw boxes around pedestrians in video footage or mark comments as offensive. This work is called annotation or labeling and is expensive. Large image datasets contain millions of such entries.

To ensure the labeling is reliable, several people look at the same example. If they agree, the label is considered certain. If they disagree, an expert decides or the example is discarded. How strongly the raters agree is reported as a separate metric.

Sometimes reality provides the solution on its own. Whether a loan was repaid is established after three years. Whether someone clicked on an ad is logged. Such automatically generated reference data is cheap, but not always what one actually wanted to measure. A click doesn’t mean the ad was good. It’s also important to note: ground truth is not always the objective truth, but often just the best available determination. Whether a text sounds “polite” is a matter of interpretation.

From driver assistance systems to chatbot rankings

In autonomous driving, camera images are cross-checked with laser scanners and manual work to record exactly where cars and cyclists were positioned. In medicine, the finding from a tissue sample often serves as the reference for an X-ray. In both cases, obtaining this data is the most labor-intensive part of the project.

The well-known rankings for language models are also based on this. A test resembling a collection of school assignments includes the model answer for each question. The percentage figure in a press release is the proportion of matches against these solutions. Because such tasks are posted on the internet, they can accidentally end up in training data. Then a model appears stronger than it actually is. Experts call this contamination.

With open-ended texts, there is often no single correct answer. There, the fixed solution is replaced by human ratings comparing two answers. If you read in the news that a model achieves a certain score, it’s worth asking: What was it measured against? Who determined the solutions?