Scraping

With scraping, a program automatically reads out websites and saves their content. A human would open a page, read it, and type out the interesting parts. A scraper does exactly that, only thousands of times per minute and without pause. The program calls up an address, fetches the page text, searches out the desired parts, and writes them into a table or database. It’s important to note the difference from an official interface: with an interface, the provider deliberately makes its data available for retrieval. With scraping, the program simply takes whatever is on the publicly visible page.

Why AI models wouldn’t exist without scraping

Large language models learn from text, and from an extreme amount of text at that. The training data of well-known models comprises many billions of words. No one could gather such quantities by hand. They come almost entirely from scraped websites, forums, news portals, and online encyclopedias. The best-known example is Common Crawl, a freely available archive that has been storing large parts of the open internet for years.

This gives rise to a conflict that has shaped the industry since 2023. Newspapers, photo agencies, and authors argue that their content was used without permission and without payment. There are now numerous lawsuits against AI companies, including one by the New York Times against OpenAI. At the same time, licensing agreements are emerging in which publishers receive money for access to their archives. For investors, this is a real cost factor: training data was free for a long time and is now becoming a paid-for raw material.

Scraping is not only an AI topic, however. Price comparison portals, real estate search engines, and market research firms build their entire business on regularly reading out other people’s websites.

From page request to finished table

Technically, scraping proceeds in three steps. First, the program sends a request to the server, just as a browser would. The server responds with the page’s source code, i.e., the code from which the browser assembles the visible page. In the second step, the scraper searches this code for patterns, for example the spot where the price is always located. In the third step, the result ends up structured in a file.

It becomes difficult when content is only generated in the browser by additional programs. In that case, the simple source code isn’t enough, and the scraper has to control a complete browser in the background. This is considerably slower and more elaborate.

Many site operators also defend themselves against automated access. They block addresses that send too many requests too quickly, or put up puzzles that are difficult for machines. One voluntary rule is the robots.txt file: this is where a site writes down which programs should not read it out. This file is a request, not a technical barrier. A common misconception is therefore that scraping is automatically permitted as long as it works technically. Whether it is permissible is decided by copyright law, data protection law, and the site’s terms of use.

Recognizing scrapers in everyday life

Anyone comparing a flight or a hotel is using results from scrapers. The same applies to portals that collect gas prices or used-car listings. Search engines also work with a related technique: their crawlers move through the web and capture pages so that they can later be found. The difference lies in the purpose. A crawler wants to discover pages for the index; a scraper wants to extract and keep specific content.

In the news, the term usually comes up in disputes. Reports about blocked AI bots, about new licensing agreements between publishers and AI companies, or about harvested profile data from social networks are all about scraping. In the last case, one speaks of data scraping of personal data, which in Europe quickly collides with the General Data Protection Regulation.

The technique is also widespread on a small scale. A school project that fetches weather data from a website every day and turns it into a chart is scraping. The effort involved is minimal, but the legal question remains the same.