Web crawler

A web crawler is a program that visits web pages on its own and saves their content. It works like an extremely fast reader that never gets tired. On every page, it collects the references to other pages contained there, in other words the clickable links. These addresses go onto a waiting list, and the program then fetches them one after another as well. In this way it works its way from page to page across the web and can visit millions of addresses in a short time. No human sits at a screen clicking along.

No crawler, no search engine

When you google something, Google doesn’t search the internet. It searches its own copy of it. This copy is called an index, and it was compiled beforehand by crawlers. That’s the only reason a search result appears in a fraction of a second. A page that has never been visited by a crawler simply doesn’t show up in the results.

For a few years now there has been a second major reason. AI language models like ChatGPT learn from enormous amounts of text, and much of this text comes from the open web. It, too, was gathered by crawlers. Well-known collections such as Common Crawl comprise billions of pages. This has turned crawlers into the raw-material suppliers of the AI industry.

This is precisely what has sparked a dispute. Newspapers, forums, and image platforms see their content being scraped up for free and used in products from which they earn nothing. Several publishers have therefore filed lawsuits, while others have signed licensing agreements with AI companies. On finance and tech sites, the term is thus usually discussed in connection with copyright and money, not technology.

From the waiting list to the index

At its core, the process is simple. The crawler starts with a list of known addresses. It downloads the first page, saves the text, and extracts all the links. New addresses are added to the back of the waiting list, while ones already visited are filtered out. Then it moves on to the next address, and it does so in thousands of processes simultaneously.

Things get tricky with prioritization. The web is practically infinite, and many pages change daily. So the crawler has to decide what to fetch first and how often to come back. A news homepage might be visited several times an hour, while an old hobby site might be visited once a year. It also has to keep up its pace without overloading other people’s servers.

There is a voluntary traffic rule for this. In the robots.txt file, any website can specify which areas bots should not access. Reputable crawlers respect this, but it cannot be technically enforced. A common misconception is therefore that robots.txt provides protection. It’s more of a sign that can also simply be ignored.

Where crawlers turn up in everyday life

You encounter crawlers constantly without ever seeing them. Price comparison portals use them to read out shops' current offers. Travel portals use them to collect flight prices. Even the small preview with an image and title that appears when you share a link in a chat comes from an automatic retrieval of the page.

Website operators see crawlers in their access logs. There you’ll find a name like Googlebot, Bingbot, or GPTBot. Many sites now specifically block the bots of AI companies while still letting classic search engines in. That’s because search engines send visitors back, whereas AI chatbots have so far done so far less often.

Two terms that are often mixed up should be kept distinct. Crawling means systematically finding and fetching pages. Scraping means extracting specific data from a page in a targeted way, such as all the prices in a table. In practice, both steps often happen together, but they don’t mean the same thing.