Crawling

Crawling

Crawling refers to the automatic scanning of the internet by a program that calls up pages one after another, stores their content, and continually follows new links. Search engines and AI companies use crawlers to collect vast amounts of text – as the basis for their index or their training data.

A crawler is a program that visits web pages on its own. It downloads a page, reads it, and then follows the links found on that page – working its way from page to page. This process is called crawling. The program itself is also called a spider or bot. Search engines like Google use crawlers to find out which pages exist at all and what is on them. Without crawling, there would be no search results.

Why crawlers are what makes the web usable in the first place

The internet consists of hundreds of billions of pages. No person and no team could manually review this volume. Crawlers handle this automatically, around the clock. Without them, the web would be a giant archive with no table of contents.

For AI companies, crawling has taken on a second meaning. Large language models – programs like ChatGPT that understand and generate text – need enormous amounts of text for training. A large portion of this text comes from the public internet and was collected via crawlers. The dataset “Common Crawl,” for example, contains snapshots of billions of web pages and is used by many AI labs as a basis for training.

Crawling thus determines not only what a search engine finds, but also what an AI model has ever read – and therefore what it knows or doesn’t know.

How a crawler proceeds page by page

It starts with a list of known starting addresses. The crawler fetches the first address and receives the page’s HTML code in return – this is the source code from which a browser builds a web page. From this source code, the crawler extracts all links and adds them to the queue. It then processes the next address, and so on.

To avoid running in circles, it keeps track of which addresses it has already visited. In addition, a website can use a file called robots.txt to specify which areas the crawler may enter and which it may not. Many websites exclude certain directories – such as an internal members area – or block individual crawlers entirely. However, whether crawlers respect these rules is their own decision; it cannot be technically enforced.

In practice, large companies don’t operate a single crawler but thousands of instances running in parallel on distributed servers. Google states that it crawls many billions of pages every day. The collected raw data is then indexed, meaning it is processed so that it can later be searched quickly.

Where crawling appears in news and products

In tech news, crawling mainly comes up when there are disputes over data rights. Since it became clear that large language models' capabilities stem from crawled text, publishers and authors have been complaining that their content was used without compensation. Some major news websites have since explicitly blocked access for AI crawlers.

For businesses, crawling is also its own field of commerce. Search engine optimization services – meaning measures to help a website rank higher in results – use their own crawlers to check whether a site is technically well-built and can be properly captured by Google.

Price comparison portals and news aggregators also use crawling: they automatically read product prices or headlines from other sites. This regularly raises legal questions – because not every technically possible form of data collection is actually permitted.

Related Products

Latest News

Subscribe free. Unsubscribe the second it sucks.

High-signal news across AI, business, UX, and tech. Every morning.