AI Crawler

An AI crawler is a program that visits web pages on its own and saves what’s written there. It works like an extremely fast reader who wanders through the internet link by link. Such programs have existed for decades: Google uses them to build search results. What’s new is the purpose. Companies like OpenAI, Anthropic, or Meta use them to collect text, images, and program code in order to train their AI systems, meaning to feed them with examples. Some of these programs also fetch content live when a user is just asking a question. That’s why people now talk about AI crawlers and no longer just search engine bots.

The dispute over other people’s content

For website operators, a basic principle has shifted as a result. With a search engine, the deal was clear: the bot takes the text, and in return the search engine sends visitors back. An AI crawler also takes the text, but the user later gets a finished answer in the chat window. Often they no longer click on the source at all. Newspapers and specialist portals are losing readers and advertising revenue as a result.

This has turned into real legal cases. The New York Times sued OpenAI and Microsoft because its articles had allegedly ended up in training data without permission. Other publishers took the opposite route and sold licenses: Axel Springer, Reddit, and the news agency AP have signed contracts with AI companies. Content that used to simply be there has suddenly become an economic good with a price tag.

On top of that, there’s a practical problem. AI crawlers sometimes fetch millions of pages per day, straining the servers that deliver a website. Wikipedia and several open-source projects have reported significantly increased costs. For small sites, such a surge can completely bring down the technical infrastructure.

How a crawler moves through the web

The process is simpler than one might think. The crawler starts with a list of addresses. It fetches the first page, saves the content, and notes all the links found on it. These links get added to the back of the list. Then it moves on to the next address. This way, the program works its way through the web like a snowball.

So that a server can recognize who’s making the request, every crawler identifies itself by name. At OpenAI, for example, it’s called GPTBot, at Anthropic it’s ClaudeBot. Websites can specify in a small text file called robots.txt which bots they don’t want. However, this file is only a polite request, not a barrier. A program can ignore it or register under a false name.

That’s why many sites now resort to harsher measures. Providers like Cloudflare block suspicious access directly at the network level. Others deliberately serve recognized bots nonsensical text so that the training data becomes unusable. By the way, an AI crawler shouldn’t be confused with the model itself: the crawler only collects the material; the actual learning happens later in a separate step.

AI crawlers in everyday life and in the headlines

You usually notice the work of these programs only indirectly. When a chatbot summarizes a news item from this morning and links to the source, a crawler had previously fetched that page. The AI answers at the top of Google search results are also based on collected web content. Anyone running their own website or blog will find these bots in their access logs under names like GPTBot or CCBot.

In business news, AI crawlers mainly come up in connection with three topics: lawsuits from publishers, new licensing deals, and blocking technology as a business model. Cloudflare made headlines in 2025 because the company blocks AI bots by default for new customers and is planning a pay-per-request paywall. This could permanently change access to web content.

For you as a reader, one thing matters most. Whatever you publicly post online can be captured by such programs and reused within an AI system. So far, there is no reliable way to undo this after the fact.