RFC 9309

Every website can provide a small text file called robots.txt. This file always resides in the same place, namely directly beneath a website’s main address, for example at “example.com/robots.txt”. Automated programs, so-called web crawlers, which systematically visit and read pages, are supposed to read this file first and then abide by it. RFC 9309 is the document that, since August 2022, has set out as an official internet standard exactly how this file must be structured and how crawlers should interpret it. RFC stands for “Request for Comments” — this has been the name for decades for the technical specifications on which the internet is built.

Why a 30-year-old custom only now became a standard

The robots.txt convention was introduced in 1994 by Martijn Koster. Back then it was an informal proposal, not a binding set of rules. Nevertheless, most major crawlers adhered to it — out of habit, not obligation. The problem: when something is not precisely written down anywhere, everyone interprets it a little differently.

Google, which took the lead in driving the standardization effort, documented over the years how differently various crawlers interpreted the same robots.txt file. Some ignored certain notations, others handled wildcard characters differently. RFC 9309 eliminates this gray area. Among other things, it specifies how crawlers must handle conflicting rules, how large a robots.txt file may be, and what happens if the file cannot be reached at all.

This matters to website operators because their instructions are now followed more predictably. For crawler developers, the standard creates a clear basis against which they can measure their software.

Structure and logic of robots.txt

A robots.txt file consists of blocks, each beginning with “User-agent”. This entry specifies which crawler the following rule applies to. It is followed by “Allow” and “Disallow” lines, which permit or block specific address paths. The wildcard “*” for User-agent applies to all crawlers that do not have their own block. A simple example: “User-agent: * / Disallow: /internal/” means that no crawler should visit the /internal/ area.

RFC 9309 clarifies several points that were previously ambiguous. If an “Allow” rule and a “Disallow” rule concern the same path, the longer rule wins — because it is more specific. If the robots.txt file is unreachable and the server responds with an error, the crawler must react differently depending on the error code: a temporary error means waiting for the time being; a permanent error means treating all areas as allowed.

It is also important to note what RFC 9309 expressly does not regulate: it does not require any crawler to respect the file. It only describes how the file must be correctly interpreted if a crawler chooses to take it into account. Compliance is voluntary — but whoever reads robots.txt must follow RFC 9309.

RFC 9309 and the training of AI models

Since 2023, RFC 9309 has appeared in the news mainly in connection with AI. Companies like OpenAI, Google, and Anthropic deploy crawlers to collect text from the web in order to train their language models with it. Many website operators — including newspapers, publishers, and platforms — do not want this. They therefore enter the crawler names of these companies into their robots.txt and block them.

This raises a fundamental question that RFC 9309 itself does not answer: do all AI crawlers actually abide by the rules? Several investigations have shown that some providers have at least partially ignored robots.txt blocks. The standard regulates the how, not the whether — and enforcement mechanisms are lacking. The debate over whether a violation of robots.txt is legally relevant is running in parallel with the technical discussions in various countries.

RFC 9309 is thus more than a dry technical document. It is the central rulebook in a dispute over who may use the open web — and for what purpose.