Synthetic Data

Synthetic data are data that no one has measured or written in the real world. They are generated by computer, but are meant to look like real data. One example: instead of photographing ten thousand pictures of real traffic signs, a program is used to compute ten thousand images of street scenes. Not a single real sign appears in these images, yet a program can still learn to recognize signs from them. The counter-term is real data or actual data: everything that comes from measurements, cameras, forms, or texts produced by humans. Synthetic data are therefore not a fraud, but a deliberately manufactured substitute supply.

When real data is missing, too expensive, or forbidden

Modern learning programs need enormous quantities of examples. For many tasks, these quantities simply do not exist. A self-driving car also has to be able to handle an overturned truck in fog. Such situations are, fortunately, rare, and precisely for that reason they are missing from recordings of real test drives. On a computer, they can be generated as often as desired.

A second reason is data protection. Hospitals have patient records, banks have account transactions. Neither can simply be handed over to developers, because real people stand behind them. A synthetic dataset can reproduce the statistical patterns without a single real person appearing in it. Researchers can then compute without violating personal privacy rights.

The third reason is money. Paying people to label images or write example answers often costs several euros per item. A program generates the same quantity for a fraction of a cent. Large AI companies are now also using synthetic data because publicly available material on the internet is gradually being exhausted.

From simulation to a model that writes its own practice exercises

There are roughly three approaches. The first is simulation: the world is rebuilt on a computer, somewhat like in a video game, and virtual photos are taken within it. Because the program built the scene itself, it knows exactly where each object is located. The labeling of the data thus comes along free of charge. This is a major advantage over real photos, where humans have to mark every object by hand.

The second approach is statistical reconstruction. A real dataset is analyzed and new rows are generated with the same distributions and relationships. A synthetic customer dataset then has the same age distribution and the same typical revenue patterns as the original, but no real names. The third approach is the most important today: an already-trained language model writes training material for another model. It formulates tasks, solves them, and explains the solution path.

A well-known problem lurks here, often called model collapse. When models learn only from models, errors and quirks reinforce themselves with each round. Rare but important cases gradually disappear from the data. For this reason, synthetic data is checked with filters or mixed with real data. A common misconception is that synthetic data is automatically neutral. It inherits the biases of the source it comes from.

Where synthetic data is already in use today

They are most visible in automated driving. Manufacturers have their systems drive millions of simulated kilometers before a real car rolls onto the road. This is also common in medical technology, for instance when a program is supposed to recognize rare tumor forms and only a few hundred real images exist worldwide for this purpose.

In business news, the term usually appears in connection with the training of large language models. Several well-known models were explicitly further trained with computer-generated tasks and solutions. An entire industry has emerged around this process, in which companies sell synthetic datasets. Anyone reading such reports should pay attention to two questions: where does the source of the data come from, and how is it verified that what is generated is actually correct.