Web Scraping

Web scraping means: a program calls up websites and automatically reads out their content. A human would open a page, read off the price, and type it into a spreadsheet. The program does exactly that, only a thousand times over and in a matter of minutes. The end result is an organized collection of data, for example a table with product names, prices, and dates. The word comes from the English “to scrape,” meaning to scratch or scrape off. What is meant is that the desired information is extracted from the finished, formatted page.

Why half the internet gets copied out

A great deal of valuable information is openly available online, but not in a usable form. An online shop displays its prices on nicely designed pages, not as a downloadable file. Anyone who wants to compare prices across many shops therefore has to collect them themselves. This is exactly what comparison portals for flights, hotels, or electronics live on.

In recent years, a second reason has become even more important. Large language models, i.e. AI systems like ChatGPT, learn from enormous amounts of text. A large portion of these texts come from the open internet and were collected via scraping. Without this collection work, today’s AI models would not exist.

This gives rise to an ongoing dispute. Publishers, forums, and image databases see their content siphoned off for free and then exploited in commercial products. Several newspapers and artists have therefore sued AI companies. Other providers now prefer to sell their archives for licensing fees rather than simply let them be copied. Legally, the situation is not uniform: scraping is not automatically permitted just because a page is publicly accessible.

From page source code to spreadsheet

Technically, the process happens in three steps. First, the program requests a page from the server, just as a browser would. It receives HTML back, i.e. the blueprint of the page with text and formatting instructions. In the second step, the program searches within it for the desired spots. In the third step, it saves the findings in an organized way.

The searching works via patterns in the page structure. A price on a shop page usually always sits in the same spot in the blueprint, for instance in an element labeled “price.” The scraper is configured to this pattern and then finds it again on all product pages. An additional program, the crawler, clicks its way through the links beforehand so that the scraper knows about all the pages in the first place.

In practice, this is fragile. If the shop changes its layout, the pattern no longer fits and the scraper delivers nonsense or nothing at all. Many pages also only assemble their content within the browser, so no data is present yet in the initial blueprint. In that case, an actual browser has to be remotely controlled, which is slower. Operators also defend themselves with image puzzles, blocking of addresses, or speed limits per visitor. A file called robots.txt additionally states which areas automated visitors should avoid. It is a request, not a technical barrier.

Price portals, bots, and news about data deals

In everyday life, one usually encounters scraping as a finished product. Flight and hotel comparisons, real estate portals, and price histories for electronics are based on it. Search engines, too, are at their core gigantic crawling and scraping machines that capture billions of pages. Anyone who types an address into a maps app is often using data that was gathered together from many sources.

In business news, the term mainly comes up in two contexts. First, in lawsuits and licensing agreements between media companies and AI providers. Second, with platforms that suddenly make their interfaces for developers more expensive or shut them down in order to slow down the siphoning of data. Reddit and X have done this in a publicly visible way.

It is important to distinguish this from hacking. Scraping only accesses content that every visitor can see; it does not bypass any password. It becomes problematic due to the sheer volume, violated terms of use, or personal data such as profile photos. For school projects, a simple rule therefore applies: query sparingly, cite the source, and prefer official data interfaces when they exist.