What is llms.txt?
The reference version: what the specification actually says, what changed in v2 on 10 August 2026, who publishes one, and what a real file looks like on a real domain.
llms.txt is a markdown file that tells an AI agent what a site contains and where to find the useful version of it. The proposal is by Jeremy Howard, published September 2024 and revised to v2 on 10 August 2026. It can sit at the site root or at any path, and it covers the pages beneath it.
- Current version
- v2, modified 10 August 2026. Most writing about the file describes v1.
- Required
- One H1. Everything else in the format is optional.
- Where it goes
/llms.txt, or any subpath. The most specific file covering a URL wins.- Real adoption
- 11.5% of 1,048 top sites we crawled serve a real one. A status code check says 27%.
What the specification actually requires
The format is short enough to state completely, and most articles about it state it loosely, so here it is in the order the proposal gives. A conforming file contains: an optional byte order mark; an H1 with the name of the project or site, which the spec calls the only required section; a blockquote holding a short summary; then zero or more markdown sections of any type except headings; then zero or more H2 sections, each containing a file list.
A file list is a markdown list where every item carries a required hyperlink in the form [name](url), then optionally a colon and notes about that link. That colon is load bearing in practice even though the spec marks the notes optional: a bare list of links tells an agent where things are and nothing about which one answers the question it currently has.
The one thing to take from the shape is that the file is an index, not a corpus. The proposal is explicit that the file should stay small enough to fit in context, with the detail behind the links, fetched only when needed. A file that inlines everything it could have linked to has misunderstood the job, and there are some very large ones in the wild doing exactly that.
There is a companion convention, llms-full.txt, holding the full text rather than links to it. It is aimed at developer documentation and is overkill for most businesses, which is the same conclusion the proposal reaches by a different route.
The H1 is the only hard requirement
Everything else, including the summary blockquote, is optional in the specification. That is worth knowing before you accept a validator that fails your file for a missing section the spec does not require.
It can live at a path, not only the root
A file at /docs/llms.txt covers everything under /docs/. Where several files could apply, the agent should use the most specific one. This is what lets somebody who controls a directory but not a domain participate at all.
Markdown, deliberately, rather than XML or JSON
The proposal picks markdown because the intended reader is a language model, while noting the format is precise enough for ordinary parsers and regular expressions to read too. It is the rare file meant to be read by both without a second serialisation.
It is for inference, not training
The proposal says its expectation was that the file would be useful at inference time rather than for training, and that this is how it has been used. If your reason for publishing one is to influence what a model was trained on, that is not what this file does.
What changed in v2, on 10 August 2026
The revision is recent enough that almost everything written about llms.txt, including our own earlier article, describes the previous version. Four changes matter if you already have a file.
Discoverability now has an answer. The commonest request after two years of adoption was: given a page, how does an agent find its markdown version, or the llms.txt that covers it, without guessing? v2 answers with standard link relations. rel="alternate" type="text/markdown" points at a page's markdown version and rel="describedby" points at the llms.txt covering it, supplied either as HTML <link> elements or as an HTTP Link: response header. The header form is the interesting one operationally: it can be set at the web server or CDN, works for non HTML resources, and needs no change to any page.
Both markdown URL forms are now valid. v1 specified appending .md to the whole page URL, giving page.html.md. Publishing tools shipped the other form, replacing the extension to give page.md. v2 accepts both rather than pretending the tools were wrong.
Subpath files are defined rather than merely permitted. A file covers the pages under its path and the most specific applicable file wins. The proposal gives the reason for choosing this over the /.well-known/ convention of RFC 8615: well known URIs exist only at the origin root, so an author who controls a path but not the host, such as a project on GitHub Pages, could never publish one there.
The Optional section lost its mechanical meaning. Under v1 it told the context expansion tooling what it could omit. That tooling is no longer part of the proposal, so the section is now a convention for secondary links and nothing more. If you built anything that treats it as a machine instruction, it is now reading a heading with no defined semantics.
Who actually publishes and reads one
This is the question the file lives or dies on, and it is the one most pages about llms.txt answer with a vibe. Here is what we could verify by fetching things on 13 August 2026, and where we could not verify, that is said rather than smoothed over.
The AI labs publish them for their developer documentation. The proposal claims OpenAI, Anthropic and Gemini do. Fetching the three URLs it links: OpenAI returns a real markdown file at developers.openai.com/llms.txt, Anthropic returns one at docs.anthropic.com/llms.txt, and the Gemini URL could not be checked from outside at all, because an unauthenticated request enters a sign in redirect loop and never returns a body. Two verified, one unverifiable, zero assumed.
Note where those files are. anthropic.com/llms.txt is a 404; the file is on the documentation subdomain. That is the normal pattern rather than an exception, and it is why a check that only tries the apex domain will tell you a company has no file when it has one.
Chrome does audit for it, and the detail matters. Lighthouse has an llms.txt audit under its agentic browsing checks. What the audit does is flag a server error when fetching the file. If the file is simply absent and the server returns a clean 404, the audit is marked Not Applicable, because providing the file is described as optional. So Chrome checks for one and Chrome does not mark you down for not having one, and anybody quoting the first half as pressure to publish has stopped reading one sentence early.
What no operator documents is the part that matters most. None of the major assistants has published documentation saying its retrieval pipeline fetches your /llms.txt while answering a question about you. Coding agents and documentation tooling use these files heavily and visibly. A chat assistant deciding how to describe your company is a different pipeline, and for that one the evidence is still absent. Google states the opposite outright for its own AI features: no AI text file is required.
How many sites really have one, measured rather than repeated
We crawled 1,048 of the most visited websites on 11 August 2026, reading the robots.txt, the llms.txt and the homepage of each. 121 of them serve a real llms.txt, which is 11.5%.
The interesting number is the one next to it. A further 162 domains return HTTP 200 for /llms.txt while serving an HTML page, because their server answers every unknown path with the site rather than a 404. A checker that trusts the status code and never looks at the body would have reported adoption at 27%, more than double the truth. If you take one operational thing from this page, take that: a check that reads the response rather than the status code is not a pedantic distinction, it is the difference between 11.5% and 27%.
Two more findings from the same crawl. The median real file is about 8.5 KB, which is an index rather than a corpus and matches what the proposal asks for. And 11 sites publish an llms.txt while blocking an AI crawler in the same robots.txt, which is a locked door with a welcome mat in front of it.
The full method, the denominator and the CSV are published with the study, so any of it can be recomputed rather than taken on trust.
llms.txt against the files you already serve
None of these replaces another. The useful question is not which to have but which question each one answers, because they answer four different ones.
| File | Answers | Read by | Enforced |
|---|---|---|---|
| robots.txt | Who is allowed to fetch what. | Every well behaved crawler. | By convention, and RFC 9309 defines it. |
| sitemap.xml | Every URL you would like indexed. | Search engines. | Yes, as a discovery input. |
| llms.txt | Which pages are worth reading, and what you are. | Coding agents and documentation tooling, verifiably. Chat assistants, undocumented. | No. It is a proposal, not a standard. |
| llms-full.txt | The whole text, inlined, with no fetching needed. | The same tools, when they want everything at once. | No, and it is overkill outside developer docs. |
| A page's .md twin | This page, without the navigation and scripts around it. | Agents that follow the v2 link relations to find it. | No, but v2 gives it a discoverable location. |
robots.txt is permission, sitemap.xml is inventory, and llms.txt is editorial judgement. Only the third one asks you to say which of your pages actually matter, which is why writing one takes an hour and copying one takes a minute and is worth nothing.
Writing one that conforms, in the order that avoids rework
Under an hour for most sites, and most of that hour is spent deciding what belongs in it rather than typing.
- 01
Check the door is unlocked first
A file listing your best pages, on a domain that refuses the crawler, is decoration on a locked door. Run the AI crawler checker before writing a line. Eleven of the sites in our crawl got this exactly backwards.
- 02
Write the H1 and the summary as if they will be quoted alone
The H1 is the only section the spec requires and the blockquote under it is the passage most likely to be lifted whole. Put what you do and what it costs in the first forty words, in flat declaratives, with every figure next to the thing it describes.
- 03
Choose twenty links, not four hundred
A file that lists everything has made no editorial decision and is worth what your sitemap is worth. Give every link the colon and a sentence saying what is behind it and when somebody would want it.
- 04
Decide root or subpath deliberately
One file at the root for a business site. A file per documentation area if the areas have genuinely different readers, since v2 resolves overlap by using the most specific file. Do not create subpath files to seem thorough.
- 05
Add the v2 link relations while you are in there
Point
rel="describedby"at the file and, if you publish markdown twins,rel="alternate" type="text/markdown"at those. Setting theLink:header at the CDN does the whole site at once and touches no page. - 06
Generate it from the source your product pages use
A hand written copy of a price list is a copy that is wrong the first time a price changes, and a file that is confidently wrong at the moment of citation is worse than no file. If it cannot be generated, put a date in it and say which page is authoritative when the two disagree.
How to tell whether yours is really being served
Fetch it and read the body. That is the whole method, and it is the one thing the tooling in this category most often gets wrong. Our crawl found 162 domains that answer 200 for a file they do not have, because their server returns the homepage for every unknown path, and every one of them would pass a status code check.
So the test is: does a fetch of /llms.txt return markdown that starts with an H1, rather than HTML that starts with a doctype? Our llms.txt checker reads the response for exactly this reason, and reports your robots.txt and sitemap status alongside it, because those three answer the discovery question together.
Two further checks worth making by hand. Serve it as text/plain or text/markdown rather than as HTML, since a file served with the wrong content type is a file some clients will not treat as text. And check the subdomain your documentation actually lives on, not only the apex, because that is where most of the real files in our crawl turned out to be.
What you cannot measure is whether it changed anything, and no honest page will tell you otherwise. There is no report, no header and no referrer that says an assistant read your llms.txt before writing an answer. Publish it for the cost, which is close to nothing, and never for a number you have been promised, because nobody can produce that number.
Check the file and the door in the same sitting
The file is the last thing to fix, not the first. These four cover the things that decide whether it could ever matter.
Related reading: llms.txt examples from real sites, whether llms.txt actually works, the llms.txt generator prompt, who blocks the AI crawlers.
Questions people ask about llms.txt
What is an llms.txt file?
A markdown file that tells an AI agent what a site contains and where the useful version of each thing is. It holds an H1 with the site name, a short summary, and lists of links with a sentence of context each. It was proposed by Jeremy Howard in September 2024 and revised to v2 in August 2026. It is a convention rather than a ratified standard.
Where do I put the llms.txt file?
At the root of your domain, as /llms.txt, or at any path within it. A file at /docs/llms.txt covers the pages under /docs/, and where more than one file could apply an agent should use the most specific. The subpath option exists so that somebody who controls a directory but not a whole host, such as a project on GitHub Pages, can publish one at all.
What is required in an llms.txt file?
Only the H1 naming the site or project. The specification marks every other part optional, including the summary blockquote and the link sections, though a file with only an H1 is conforming and useless. In practice: H1, one blockquote of summary, then H2 sections of links each carrying a colon and a sentence about what is behind it.
What changed in llms.txt v2?
Four things. Link relations were added so an agent can discover a page's markdown version and the llms.txt covering it, via rel="alternate" type="text/markdown" and rel="describedby", in HTML or an HTTP Link header. Both markdown URL forms, page.html.md and page.md, became valid. Subpath files were properly defined, with the most specific file winning. And the Optional section lost its mechanical meaning when the context expansion tooling left the proposal.
Do ChatGPT and Claude read llms.txt?
No operator of a major assistant has documented that its retrieval pipeline fetches your llms.txt while answering a question. Coding agents and documentation platforms do use these files, visibly and heavily, and both OpenAI and Anthropic publish one for their own developer docs. Treat any claim that a chat assistant reads yours as unverified until an operator says so.
How many websites have an llms.txt?
We crawled 1,048 of the most visited sites in August 2026 and 11.5% served a real one. A further 162 returned HTTP 200 for the file while actually serving an HTML page, so a checker trusting the status code would have said 27%. Adoption figures in this category are routinely inflated by exactly that mistake.
Does llms.txt help SEO?
It has no effect on Google rankings, and Google states directly that no AI text file is needed to appear in its AI features. Chrome's Lighthouse does include an llms.txt audit, but it only flags a server error and marks the audit Not Applicable when the file is simply absent, so it is not a penalty either. The case for the file is cost, not evidence.
What is the difference between llms.txt and llms-full.txt?
llms.txt is an index: short, and pointing at the detail. llms-full.txt inlines the full text so nothing has to be fetched. The proposal is explicit that the index should stay small enough to fit in context, which is why the full variant is aimed at developer documentation and is usually the wrong choice for a business site.
Is llms.txt the same as robots.txt?
No. robots.txt is permission and is honoured by well behaved crawlers under a defined protocol. llms.txt is editorial judgement about which of your pages are worth reading, and nothing enforces it. Publishing an llms.txt while robots.txt blocks the crawler achieves nothing, and 11 of the sites in our crawl were doing exactly that.
Check the file, and the things that decide whether it matters
The audit reads your llms.txt body rather than its status code, checks each AI crawler separately, and tells you whether the pages it points at survive without JavaScript. Free, no card, and the report is yours whether or not we ever speak.