Validation set

Programs that learn from examples need a great many such examples. Before learning begins, these examples are divided into several batches. The program practices with the largest batch, called the training data. A second, smaller batch is deliberately kept locked away during practice: the validation set. Developers use it to check in between how well the program performs on tasks it has never seen before. Comparing both results reveals whether the program has learned general relationships or is merely parroting back its practice examples.

The test against mere memorization

A model can be almost error-free on its practice exercises and still be useless. Experts call this overfitting. In that case the model has also learned the peculiarities of the training data, even its random quirks and errors. On new data, performance collapses. Without a validation set, this collapse would only be noticed once the model is actually deployed.

A comparison from school makes this clear. Anyone who memorizes all the answers to the practice worksheets scores 100 percent on those worksheets. Whether real understanding lies behind it only shows up in the class test with new tasks. The validation set is exactly such an interim test, one that may be taken as often as desired.

The distinction from the test set is important. The test set is a third batch that remains untouched until the very end. The validation set, by contrast, is looked at a hundred times during development. As a result, the model inevitably gets adjusted somewhat to fit this data. Only the test set, ideally used exactly once, delivers the honest final grade.

Splitting, measuring, turning the dials

A common split is roughly 80 percent training, 10 percent validation, and 10 percent test. What matters is that the split happens randomly and that no example ends up in two batches. Such overlaps are called data leakage and make the result look better than it is. For data with a time dimension, such as stock prices, the split is instead made by date: train on the past, validate on the future.

During training, the program regularly calculates its error on the validation set. Typically this error first decreases, reaches a minimum, and then rises again. It is exactly at this turning point that overfitting begins. Developers stop training there, a practice called early stopping.

The validation set is also used to compare different design variants. Various model sizes and learning rates are tried out, and the variant with the best validation result is kept. If the amount of data is small, cross-validation is used: the data is split into five or ten parts, and each part serves once as the validation set. The overall result is the average of all runs and is therefore more stable.

Why benchmark records should be read with caution

News about new AI models often cites percentage scores from well-known benchmark collections. Many of these collections openly provide a validation set while keeping the actual test set secret. Anyone quoting a number should state which part it was measured on. Values from the public validation set regularly turn out better than the later official results.

A well-known problem with large language models is contamination. Because they are trained on enormous amounts of text from the internet, test tasks along with their solutions may have ended up in the training material. In that case the validation set no longer tests anything, it merely measures memory. Providers therefore continually create new, fresh test tasks.

The term also appears in areas without chatbots. Banks validate creditworthiness models, clinics validate image-analysis programs, insurers validate claims forecasts. Regulators frequently require proof that the model has been tested on withheld data. The validation set is thus not merely a technical precaution but often a documentation requirement.