robots.txt

There are constantly programs roaming the internet that automatically access websites and store their content. Such programs are called crawlers or bots. Search engines need them to know which pages even exist. The robots.txt file is a simple text file that a website operator places at the very top of their directory. It states which areas of their site these programs should access and which they should not. It is always found in the same place: after the website’s address, followed by a slash and the file name, so for example example.com/robots.txt.

A house rules sign without a door lock

The file works like a sign at the entrance: “Please do not enter the basement.” Whether to comply is up to the visitor. Technically, the file prevents nothing. Any page marked as forbidden in robots.txt remains accessible to any normal browser. Anyone who really wants to protect content needs a password or an access barrier on the server.

Nevertheless, the file has been an important part of the web since the 1990s. Major search engines voluntarily comply with it because they don’t want to risk their reputation. Operators also use it to reduce server load. A bot that fetches a thousand pages every second can noticeably slow down a small website.

Since the boom of language AI, the file has become political. Companies like OpenAI or Anthropic collect huge amounts of text from the web to train their models. Many newspapers and publishers don’t want to allow this without payment. They enter the names of these AI crawlers into their robots.txt and block them. Studies show that a large proportion of the most important news sites now do this.

Structure of the file: User-agent and Disallow

The file consists of a few lines of plain text that anyone can write with a text editor. A rule group begins with the line “User-agent”. Behind it is the name of the program the rule applies to. Google's search engine bot, for example, is called “Googlebot”, OpenAI’s collector is called “GPTBot”. An asterisk instead of a name means: applies to all.

Below that follow lines with “Disallow” and a path. “Disallow: /internal/” means: hands off everything located in this subdirectory. With “Allow” individual exceptions can be granted again. An empty Disallow means everything is permitted. Many files also contain a reference to the sitemap, i.e. a list of all pages the operator wants to have found.

A common misconception: blocking a page in robots.txt does not hide it from the search engine. The address can still appear in the results if other pages link to it. To really keep a page out of the index, a different instruction is needed directly in the page’s source code, the so-called noindex tag. And because the file is publicly readable, a ban sometimes reveals exactly what one wanted to hide.

Dispute over AI training and paid crawling

The topic becomes most visible in reports about publishers and AI companies. In 2023, OpenAI introduced GPTBot and stated that it respects robots.txt. Shortly afterward, Reuters, the New York Times and many German media outlets blocked this name. Later, contracts were made in which AI companies pay for access to archives. The file is the technical tool with which an editorial team expresses its no or yes.

It also plays a role as evidence. If a crawler accesses a blocked page anyway, this can be read in the server logs. Exactly such logs were the basis for public accusations against individual AI providers in 2024. Providers of network infrastructure such as Cloudflare go further and technically block unwanted bots instead of merely politely asking.

For you personally, the file is above all a glimpse behind the scenes. You can call up the address with /robots.txt on any website and immediately see which bots are welcome there. On large portals these are often hundreds of lines. Anyone running their own website or blog quickly comes into contact with the file themselves, because website builders and content management systems create it automatically.