llms.txt

llms.txt is a small text file stored on a website. It is always located in the same place, namely directly behind the main address: for example example.com/llms.txt. In this file, the operator describes in a few lines what the site is about. To do this, they list the most important subpages along with a brief note on what can be found there. The whole thing is not meant for humans, but for programs like ChatGPT or other chat assistants that read web pages and formulate answers from them. The idea was proposed in September 2024 by the Australian developer Jeremy Howard.

Why websites are hard for machines to read

A normal website consists of the actual text only to a small extent. The rest is menus, advertising space, cookie notices, footers, and invisible instructions for the browser. A human effortlessly filters this out. A program, on the other hand, first has to laboriously sort out which part is actually the content.

There is also a second problem: language models can only process a limited amount of text at once. This limit is called the context window. A large website with thousands of pages never fits into that. An llms.txt reduces this to a clear list that might be one or two screen pages long.

For companies, there is also a concrete interest behind this. More and more people are asking their questions to a chat assistant instead of a search engine. Anyone who wants their product or documentation to be accurately represented in the process has a motive to make it easy for the programs. Wrong answers about one’s own company cost trust.

Structure of the file

The file is written in Markdown. This is a very simple notation in which headings are marked with hash symbols and lists begin with hyphens. It can be opened and read with any text editor. That is exactly the point: no special software and no complicated technology is needed.

The structure follows a fixed pattern. Right at the top is the name of the project as a heading. Below that follows a short paragraph explaining in two to three sentences what it is about. After that come sections with links, each supplemented with a short clause of explanation. A section called Optional may contain content that a program can omit if there is a lack of space.

There is often an additional file called llms-full.txt. It contains not just links, but the complete text of the website in a tidied-up form. A common misconception is confusing it with robots.txt. robots.txt forbids programs from accessing certain areas. llms.txt does the opposite: it invites them in and shows the way.

Adoption and open questions

llms.txt is most commonly found among providers of developer tools. Companies like Anthropic, Stripe, or Cloudflare provide their technical documentation this way. The reason is obvious: programmers often have code written by an AI assistant. If it finds the current instructions, it makes fewer mistakes.

In the news, the term usually appears in connection with a dispute. Publishers and website operators are discussing whether they should supply their content to AI companies at all. Some block the programs out, others hope for visibility. llms.txt clearly stands on the side of those who want to be found.

Important for context: this is a proposal, not a rule. No body has adopted it, and the major AI providers have not yet publicly confirmed that their systems actually evaluate the file. Google even expressed explicit skepticism in 2025. Anyone who creates an llms.txt is thereby investing half an hour of work in a possibility that still has to prove itself.