Crawler

A crawler is a program that visits internet pages without human assistance. It downloads a page, stores its content, and looks at the links pointing from that page to other pages. It then automatically follows these links, and on the new pages the same process starts all over again. In this way, a crawler works its way through millions of pages, step by step. The name comes from the English word for creeping or crawling: the program essentially crawls along the web of links. Another common name is spider, since it moves through the web of pages and links.

No search engine and no language model without crawlers

When you google something, Google isn’t searching the internet itself. Google is searching its own copy of the internet. This copy was assembled by a crawler, often weeks or months earlier. That’s why search results are sometimes outdated: the crawler hasn’t been back since the last change.

In recent years, a second purpose has emerged. Large language models like ChatGPT learn from enormous amounts of text, and much of this text comes from the open web. It was collected by crawlers. Well-known collections such as Common Crawl comprise billions of pages and are freely available.

This is exactly what has sparked a dispute. Publishers, photographers, and authors see their content reappear in AI models without having been paid for it. Several media companies have therefore filed lawsuits, while others have signed licensing agreements with AI companies. Who is allowed to crawl which data has since become a highly relevant economic issue.

From the queue to the polite request

Technically, a crawler maintains a long queue of addresses. It takes the address at the top, fetches the page, and stores the text in memory. It appends all the links it finds on the page to the back of the queue. Then the next address takes its turn. Since this goes on endlessly, in practice thousands of crawler processes run simultaneously.

This creates two problems. First, the same pages keep turning up again and again, so the crawler has to remember what it has already visited. Second, it would bring a small web server to its knees if it sent hundreds of requests per second. Responsible crawlers therefore deliberately wait between two requests to the same page.

There is also a kind of house rule. In a file called robots.txt, an operator can specify which areas of their site should not be accessed. However, this is merely a request, not a technical barrier. Those who don’t comply are usually locked out through other means, such as blocking the access address.

Crawlers in the news and in everyday life

You rarely notice directly that a crawler is currently at work. It becomes visible in the results: in search hits, in price comparison portals, in travel portals that line up flight prices from many providers side by side. Even the small preview with an image and title that appears when sharing a link is fetched by a program of this kind.

In the news, you usually encounter crawlers under their names. GPTBot belongs to OpenAI, Googlebot to Google Search. Major news sites blocked GPTBot in large numbers starting in 2023. According to research findings, a significant proportion of the highest-reach news sites now block at least one AI crawler.

A common misconception: a crawler is not the same as a scraper. The crawler discovers pages and follows links. A scraper pulls specific data from a known page in a targeted way, such as all the prices in a shop. In practice, the two are often combined, and in everyday usage the terms tend to blur together.