robots.txt examples from real websites

Read from live domains on 15 August 2026 and quoted verbatim, including the parts we then point at. Nothing on this page was written by us.

A robots.txt is a list of groups. Each one names the crawlers it applies to and gives them path rules, and a Sitemap line applies to the whole file wherever it sits. The eight files below are real, and between them they show every shape worth knowing: the three line version, the total block, the multi-agent group, and the rules nobody ever pruned.

By Honeydmaxx, Content Writer

Files read 15 August 2026

What 14 real files had in them

Counted over the 14 domains listed at the foot of this page, all of which served a file. A small sample, and it is stated as one: it is here to show the range of what people do rather than to be a rate about the web.

5 of 14name at least one AI crawler and disallow it entirely

10 of 14declare a Sitemap line

2 of 14carry a Content-Signal directive

3 linesthe shortest file in the sample, on shopify.com

One clarification the count needs. Reddit is not in the five: it blocks every crawler with a single wildcard rule rather than naming any AI agent, so a stricter reading is that five sites made a decision about AI crawlers specifically and one made a decision about everybody. The wider version of this question, measured across 1,048 sites rather than fourteen, is in who blocks the AI crawlers.

shopify.com

The whole file, on a company worth billions

Three lines, and worth showing first because of how much this page could otherwise imply. A robots.txt does not have to be long, clever or exhaustive. Two paths that produce nothing useful to a crawler are kept out and everything else is left alone. Most sites need something close to this and end up with something forty times the length.

https://shopify.com/robots.txtThe whole file · read 15 August 2026

User-agent: *
Disallow: /authentication/
Disallow: /*/account

openai.com

Allow everything, and say where the sitemap is

The shape almost every business site should have. Allow: / is redundant, since nothing is blocked by default, and stating it costs nothing and removes an argument. The Sitemap line is the part that earns its place: it is a non-group directive, so it applies to the whole file regardless of where it sits, and it is how a crawler that arrived without knowing your structure finds the list of what you want indexed.

https://openai.com/robots.txtThe whole file · read 15 August 2026

User-agent: *
Allow: /
Disallow: /microsoft-for-startups/

Sitemap: https://openai.com/sitemap.xml

reddit.com

A total block, stated as a position

Two lines of rules and five of explanation. Disallow: / under the wildcard group is the most severe thing this file can say, and here it is deliberate and documented, which is the distinction that matters. The same two lines are also the most common accident on the web: shipped from a staging environment, they remove a site from search entirely and nothing visibly breaks. The difference between the catastrophe and the policy is the comment above it.

https://reddit.com/robots.txtThe whole file · read 15 August 2026

# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy

User-agent: *
Disallow: /

theguardian.com

Many agents, one group

The most important shape on this page, and the easiest to misread. Consecutive User-agent lines do not start new groups. They accumulate, and the rules that follow apply to all of them at once, which is how one Disallow: / governs seventeen named AI crawlers here. A parser that treats each line as its own group will report this file as naming seventeen agents and blocking none of them. Ours did, on the first pass, which is why this example is here.

https://theguardian.com/robots.txtExcerpt · read 15 August 2026

User-agent: CCBot
User-agent: TurnitinBot
User-agent: PetalBot
...
User-Agent: PerplexityBot
User-agent: anthropic-ai
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: Google-CloudVertexBot
...
Disallow: /

cloudflare.com

Explicitly allowing AI crawlers

The opposite position to the Guardian, held just as deliberately. Every named group here is redundant in the strict sense, because those crawlers were already allowed by the wildcard group and by the absence of any rule. It is still worth writing: an explicit group is a statement of intent that survives the next person to edit the file, and it stops a future blanket block from being applied without somebody noticing what it covers.

https://www.cloudflare.com/robots.txtExcerpt · read 15 August 2026

User-agent: *
Allow: /

Sitemap: https://www.cloudflare.com/sitemap.xml

# Allow AI crawlers to access markdown versions of pages
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Google-Extended
Allow: /
...
# Content Signals - declare AI content usage preferences
Content-Signal: ai-train=yes, search=yes, ai-input=yes

vercel.com

A third position, and a directive Google does not read

Two things worth noticing. Content-Signal is not part of the robots exclusion protocol: it is a separate convention for stating how content may be used, and this one says search yes, AI answers yes, AI training no — a distinction robots.txt itself cannot express. And the bare Disallow: with nothing after it means "disallow nothing", so it does no harm and no work sitting above a list of real disallows.

https://vercel.com/robots.txtExcerpt · read 15 August 2026

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

# Allow robots to be able to crawl the OG Image API route
Allow: /api/og/*
Allow: /api/docs-og*
...
Disallow:
Disallow: /api/
Disallow: /oauth
Disallow: /confirm
...
Sitemap: https://vercel.com/sitemap.xml

stripe.com

Rules outliving what they were written for

Three things this file demonstrates at once. A Sitemap line works at the top, before any group, because it is not part of a group. A group for ia_archiver is a rule written for a crawler from an earlier era of the web, still being served to everybody who asks — robots.txt files accumulate and nobody prunes them. And Crawl-delay is a directive Google states it does not support, though Bing and several smaller crawlers do honour it.

https://stripe.com/robots.txtExcerpt · read 15 August 2026

Sitemap: https://stripe.com/sitemap/sitemap.xml

User-agent: ia_archiver
Allow: /docs/api
Allow: /docs/api$
Disallow: /docs
Disallow: /docs$
...

User-agent: *
Allow: /docs
Allow: /docs$
Disallow: /bitcoin/refund
...

User-agent: rogerbot
Crawl-delay: 2

developer.mozilla.org

A Sitemap line inside a group

Unconventional and correct. Sitemap is a non-group directive, so its position makes no difference and it is not scoped to the group it appears inside. It reads as a mistake and is not one, which is a useful thing to know before you open a pull request about somebody else's file. Note also /*/files/: the wildcard matches any single path segment, which is how one rule covers every locale on a site with dozens of them.

https://developer.mozilla.org/robots.txtThe whole file · read 15 August 2026

User-agent: *
Sitemap: https://developer.mozilla.org/sitemap.xml

Disallow: /api/
Disallow: /*/files/
Disallow: /media

A correction, because it changed a number on this page

The first version of this count said the Guardian named ten AI crawlers and blocked none of them. That was wrong, and it was wrong for a reason worth publishing: the parser treated every User-agent line as the start of a new group.

Consecutive User-agent lines do not start new groups. They accumulate, and the rules underneath apply to all of them at once. Correcting that moved the Guardian's file from 39 groups to 3, and from zero blocked AI crawlers to seventeen. It also changed the headline figure for the whole sample.

This is the same class of mistake as reading a robots.txt by its status code rather than its body, and it has the same consequence: a tool that is confidently wrong about a file you cannot easily check by hand. If you are counting anything across robots.txt files, check your parser against a multi-agent group before you trust the output.

Build yours rather than copying one

Copy the shape, not the contents. A publisher's block list expresses a decision about their business, and pasting it into yours is the single most common way a site ends up blocking crawlers nobody meant to block.

Generate the file from your own answers.The robots.txt generator writes named groups per AI crawler, offers the disallow paths your platform actually creates, and resolves the sitemap line. It runs in your browser and sends nothing anywhere.
Confirm it is served as a file, not as a route.The robots.txt checker reads the response body rather than the status code, which is what catches a framework answering every unknown path with your homepage.
Check the effect on each assistant separately.Rules are one thing and their consequence is another. TheAI crawler checker evaluates your live file once per named agent, the way a crawler does.

Questions people ask

What does a robots.txt file look like?

A plain text file of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow lines giving path rules. A Sitemap line can appear anywhere and applies to the whole file. Comments start with a hash. There is nothing else to it.

Can one group name several crawlers?

Yes, and this is the detail that catches people and parsers alike. Consecutive User-agent lines accumulate into one group, and the rules underneath apply to all of them. The Guardian uses this to govern seventeen named AI crawlers with a single Disallow line.

What is the simplest valid robots.txt?

A single group allowing everything, plus a sitemap declaration. Shopify's own file is three lines long and blocks two paths. A short file is not a lazy one: it means the site has few URL spaces worth keeping crawlers out of, which is the normal case.

Where does the Sitemap line go?

Anywhere. It is a non-group directive, so it is not scoped to whatever group it appears near and its position makes no difference. Stripe puts it at the very top before any group, and MDN puts it inside the wildcard group. Both are correct.

Does Google support Crawl-delay?

No. Google's robots.txt documentation states plainly that fields such as crawl-delay are not supported. Bing and several smaller crawlers do honour it, so it is not useless, and it is not a way to control Googlebot. Crawl rate for Google is managed through Search Console and by making pages cheaper to serve.

What is a Content-Signal line?

Not part of the robots exclusion protocol. It is a separate convention for stating how content may be used, with values such as search, ai-input and ai-train set independently, which is a distinction robots.txt itself cannot express. Two of the fourteen files we read carried one. Support is emerging rather than established.

Should I copy one of these files?

Copy the shape, not the contents. A publisher's file expresses a decision about their business that is probably not yours, and the most common way a site ends up accidentally blocking AI crawlers is a block list pasted from somewhere that had a reason for it. Build yours from your own answers instead.

Sources

Every file was requested directly from the domain on 15 August 2026. These are live documents and every one of them will change. Read them yourself rather than trusting this snapshot.