Data Scarcity

Programs like ChatGPT don’t learn from rules that someone has programmed into them. They learn from huge amounts of examples: texts, images, audio recordings, measurement values. As a rule, the more such examples there are, the better the results become. Data scarcity means: there aren’t enough of these examples, or not enough of them in usable quality. The problem occurs in two forms. Sometimes material is missing for a very specific field, for instance a rare disease. But sometimes almost all of the freely available material in the world has been used up.

Why the supply of texts is finite

The largest AI systems were trained on almost everything publicly available on the internet. This includes Wikipedia, news sites, forums, digitized books, and program code. Experts estimate the size of this usable body of text at a few trillion words. This amount does keep growing, but only slowly. The hunger of the models is growing considerably faster.

Until now, a simple rule of thumb applied: more data and more computing power yield a better model. This rule only works as long as there is fresh supply. This is exactly where the term data wall comes in. It describes the point at which models can no longer be improved simply by feeding them even more text.

On top of this comes a legal problem. Many publishers, image agencies, and platforms are now blocking the crawling programs of AI companies. Others demand payment for their content. As a result, the freely accessible portion of the internet is shrinking for AI companies, even though the overall body of content itself is not getting smaller.

What companies are doing about the data shortage

The most obvious approach is to buy data. OpenAI, Google, and others are signing contracts with news organizations, forum operators, and archives. A second approach turns to material that was not previously available as text: old books, videos, podcasts, scientific papers. A video, after all, provides not just images but also spoken language and subtitles.

The third approach is called synthetic data. Here, one AI itself generates the examples used to train another AI. This may sound like a trick, but it works quite well in areas with clear control. For math problems or program code, it is possible to check whether a generated solution is correct. For open-ended questions without a clear-cut answer, this kind of checking is difficult.

This is why researchers warn about so-called model collapse. If models are repeatedly trained on the outputs of their predecessors, their errors and biases become reinforced. Rare, unusual cases disappear from the material. The result resembles a photocopy of a photocopy: with each pass, details are lost.

Data scarcity in medicine, robotics, and stock market news

The shortage is felt most strongly where there was never much data to begin with. For a disease with only a few hundred documented cases worldwide, it is hardly possible to train a reliable diagnostic model. The situation is similar for minor languages. For English, enormous amounts of text exist, whereas for Icelandic or Swahili there is far less.

In robotics, the situation is even tighter. A robotic arm that is supposed to learn to grasp objects needs recordings of real movements. These are not generated incidentally on the internet, but only in labs and factories, in real time. This is why companies do a great deal of practice in simulations, that is, computer replicas of reality.

In business news, the term usually comes up as an argument about stock prices. When analysts ask whether the AI boom will continue, the question is often whether enough training material keeps coming in. Multi-million-dollar contracts between AI companies and media houses are also a direct consequence of this scarcity. Data has thus turned from a byproduct into a tradable raw material.