28.3% block an AI crawler. 1.5% block Googlebot.
We fetched robots.txt, llms.txt and the homepage of 1048 of the most visited websites, twice, on one day. Every figure below is recomputable from the CSV, and there is no form in front of it.
Of the 810 sites in this sample that publish a robots.txt, 229 disallow at least one major AI crawler and 12 disallow Googlebot. Blocking an AI crawler is not a locked front door, it is a decision: 217 sites let Google's search crawler in and shut an AI crawler out. The decision is also less precise than it looks. 145 of them block at least one crawler that fetches pages to answer live questions, which is the crawler that decides whether an assistant can cite you, and only 84 block the training crawlers while deliberately leaving every citation crawler open.
- Block an AI crawler
- 229 of 810
- Block Googlebot
- 12 of 810
- Publish an llms.txt
- 121 of 1048
- Thin without JavaScript
- 21.4%
28.3% of sites with a robots.txt disallow at least one of eleven AI agents.
1.5%. The control, and the reason the figure above is a choice.
11.5%, counting only files that are files. 162 more returned 200 and served HTML.
of 556 homepages carried under 200 words with no JavaScript executed.
Blocking an AI crawler is a decision, not a wall
The single most useful comparison in this data is the gap between 28.3% and 1.5%.
A site that blocks everything is a site with an unusual robots.txt, and there are almost none: 12 of 810 disallow Googlebot from the homepage. 217 sites, 26.8%, disallow an AI crawler while allowing Googlebot. Somebody sat down and wrote those lines.
It is also not a token gesture. Among the 229 sites that block anything, the median blocks 7 of the eleven AI agents we tested. The common pattern is a pasted list rather than a considered one, which matters for the next finding.
Only 267 of the 810 files, 33%, name any AI agent at all. For the other two thirds, whatever happens to these crawlers is decided by rules written before any of them existed.
Most of them are blocking the wrong bots
Training crawlers and citation crawlers are different agents, owned by the same companies, doing opposite jobs. The data says the distinction is not widely understood.
GPTBot is blocked by 21.4% of sites with a robots.txt. OAI-SearchBot, the crawler that fetches pages so ChatGPT can answer with them, is blocked by 10%. The same split appears at Anthropic: ClaudeBot 20.7%, Claude-SearchBot 10.9%.
Read one way, that gap is encouraging: it looks like the web has settled on do not train on us, do cite us. Read against the raw counts, it is thinner than that. Only 84 sites, 10.4%, actually hold that position cleanly. Another 145 block at least one crawler that feeds live answers, usually as collateral from a broad rule.
The most expensive version of the confusion is Google-Extended, blocked by 19.3%. It governs Gemini and other Google AI products. It does not govern AI Overviews, which are assembled from the ordinary Google index, so blocking it does not remove you from the surface most people are trying to influence, and the only directive that would also removes you from search. We wrote the eligibility rules up in how to rank in AI Overviews.
| Crawler | Owner | What it is for | Blocked | Named in file | What blocking it actually does |
|---|---|---|---|---|---|
| CCBot | Common Crawl | Training | 186 23% | 187 | Crawls for the Common Crawl public archive, which many model builders train on. Blocking it removes you from a dataset, not from an answer. |
| Bytespider | ByteDance | Training | 174 21.5% | 160 | ByteDance. Feeds TikTok and Doubao products. The most aggressive crawler in most server logs. |
| GPTBot | OpenAI | Training | 173 21.4% | 212 | OpenAI training. Blocking it does not remove you from ChatGPT answers; OAI-SearchBot is the one that does. |
| ClaudeBot | Anthropic | Training | 168 20.7% | 193 | Anthropic training. Same distinction: Claude-SearchBot is the retrieval side. |
| Google-Extended | Training | 156 19.3% | 187 | Governs Gemini and other Google AI products. It does not control AI Overviews, which run on the ordinary Googlebot index, and confusing the two is the most expensive mistake in this table. | |
| meta-externalagent | Meta | Training | 146 18% | 135 | Meta. Feeds Llama training and Meta AI. |
| PerplexityBot | Perplexity | Search and citation | 130 16% | 173 | Perplexity retrieval. Blocking it removes you from an assistant that shows its sources. |
| Applebot-Extended | Apple | Training | 122 15.1% | 124 | Apple Intelligence training. Ordinary Applebot, which powers Siri and Spotlight, is separate. |
| ChatGPT-User | OpenAI | User fetch | 102 12.6% | 147 | Fetches a page because a user asked ChatGPT to open it. Blocking it breaks a link somebody chose to follow. |
| Claude-SearchBot | Anthropic | Search and citation | 88 10.9% | 94 | Anthropic retrieval for Claude search results. |
| OAI-SearchBot | OpenAI | Search and citation | 81 10% | 129 | OpenAI retrieval for ChatGPT search results. The crawler that decides whether you can be cited. |
| Bingbot | Microsoft | Search | 13 1.6% | 94 | Ordinary Bing search, and the index behind Copilot. Also a control. |
| Googlebot | Search | 12 1.5% | 123 | Ordinary Google search. In the table as a control: it is what a locked front door looks like. |
Counts are out of the 810 sites that publish a robots.txt, evaluated for the homepage path with RFC 9309 group selection and longest match rules, using the same parser as our AI crawler checker. Googlebot and Bingbot are controls. Named in file counts sites whose robots.txt mentions the agent at all, allowed or disallowed.
llms.txt: 11.5% have one, and a status code check would say 27%
121 of 1048 sites publish a real llms.txt. The median file is 8.3 KB, and the adopters are what you would guess: documentation platforms, developer tools and large ecommerce.
The methodological result beside it is worth as much. 162 further domains answered HTTP 200 for /llms.txt and served an HTML page, usually a homepage or a soft 404 from a server that answers every unknown path. Counting status codes rather than reading responses would have produced an adoption rate of 27%, 2.3 times the real figure. Any llms.txt statistic you read that does not say how the file was validated should be treated as this number, not ours.
And the contradiction: 11 of the 121 sites publishing an llms.txt block at least one AI crawler in the same robots.txt. A file asking assistants to read the good pages, served from a domain telling them not to fetch anything. Both were almost certainly added by different people, months apart, neither aware of the other. It is the cheapest thing on this page to check on your own domain, and our llms.txt checker reads the response rather than the status code for the reason above.
The specialists behave differently, and mostly correctly
The same three fetches, the same day, against the 98 agencies selling generative engine optimisation that we audited in study 1.
- Publish an llms.txt
- 49 of 98
- Block an AI crawler
- 8 of 95
- Block a citation crawler
- 0
- Thin without JavaScript
- 0
50%, against 11.5% of the most visited websites.
8.4%, and all eight block only training crawlers.
None. Not one of them blocks a crawler that feeds live answers.
Median 1483 words on the homepage with no JavaScript executed.
This is the cleanest result in the study and it goes the way the category would want. The people selling AI visibility publish the optional file at 4.3 times the rate of the wider web, and none of them has accidentally blocked the crawler that decides whether they can be cited, which 145 of the biggest sites on the internet have.
With one wrinkle. 6 agencies publish an llms.txt and block an AI crawler in the same robots.txt, the same contradiction as above, in the sample least entitled to it.
Allowed in, and there is nothing to read
Access is only the first gate. We counted the words a fetcher sees on each homepage with no JavaScript executed, which is what most retrieval passes actually receive.
The median is 742 words, which is healthy. 119 of 556 homepages, 21.4%, carry fewer than 200 words, and 18.7% carry fewer than 100. Some of those are deliberately minimal by design, a search box or a login. Many are not: 79 pages carried the markers of a single page application in the HTML, and 27.8% of those were under 200 words, which is the render gap, visible from outside, on the most visited sites on the web.
The order of work follows from the three findings together: let the right crawlers in, make sure there is text for them when they arrive, and only then worry about the optional file. Our render gap checker covers the middle step and the free audit covers all three on one URL.
Methodology
The sample. Two frames. Tranco list 645KX, generated 11 August 2026, top 1500 rows, which is a reproducible dated ranking combining five providers rather than anybody's proprietary top sites list. And the 98 GEO agencies from study 1, reused unchanged so the two are comparable. Both frames are published as a plain text file with their selection rules.
The denominator, stated twice because it is the number most easily got wrong. 1500 domains were attempted and 1048 serve an HTML homepage. The 452 excluded are nameserver, CDN and ad delivery hosts with no website at all, and they are in the CSV with the reason. Rates over 1500 would be about 30% too low. Blocking rates are then over the 810 that publish a robots.txt, because a site with no file blocks nothing.
Everything was fetched twice, and the second pass changed a headline. Pass one declared itself as a research crawler and 189 of 1048 answered 401, 403 or 429. On its own that reads as one site in six blocking crawlers at the edge. Repeating every fetch with a Chrome user agent from the same machine and the same address showed that only 22 of those 189 served the browser. The rest refuse this address whatever it calls itself, which is datacentre IP filtering rather than a decision about bots. So the honest version of that figure is 2.1%, not 18%, and the robots.txt rules below are read from the browser pass, since a site that refused our crawler its robots file has not thereby stopped having one.
Words were counted only where counting words works. Splitting on spaces does not count words in Chinese, Japanese or Thai, and a global top sites list is full of them. Counted naively, 38.8% of homepages looked nearly empty. The content figures are computed over the 556 pages whose visible text is at least 80% Latin script; 238 pages are excluded and marked in the CSV. Text is extracted from the raw HTML with scripts, styles and comments removed. No browser, no rendering, which is the point: it approximates what a fetcher without a JavaScript engine receives.
robots.txt is evaluated, not grepped. Group selection picks the most specific matching user agent, and Allow and Disallow are resolved by longest match with Allow winning ties, per RFC 9309. It is the same parser behind our AI crawler checker, so this study and the tool we sell cannot disagree about what a rule means. A response is only treated as a robots.txt if it is not HTML and contains at least one directive line.
One path, one day. Every rule is evaluated for /, and everything was collected on 11 August 2026 within a few hours.
What this study does not show
It does not show what any crawler actually did. robots.txt is a stated policy. Whether a given crawler obeys it, and whether a WAF sitting in front of the site enforces something stricter, are different questions that need server logs to answer. Ours cannot see yours.
It is the homepage and the root path. A site can allow the homepage and disallow the blog, or the reverse. Rates here would move on a different path, and the direction is not predictable.
Tranco is not a sample of businesses. It ranks global traffic, so this frame is international, heavily technical, and over represents platforms and infrastructure. Read the headline as "the most visited domains on the web", not "typical company websites". The agency frame is the opposite: small, specialised and self selected by listicles.
Fetched from one machine on one network. Geography and IP reputation change what a server returns, which the two pass comparison above makes visible rather than hides.
It is a snapshot. One day. robots.txt changes quietly and often. Every figure is dated for that reason, and the scripts that produced them are the ones we will rerun.
Use this, and cite it like this
Published under CC BY 4.0: republish any of it, including commercially, as long as you credit the source. No form, no email wall, no request to link. If you are writing about AI crawlers and need a number, take one.
Visibility100x (2026). Who blocks the AI crawlers, and who publishes an llms.txt: robots.txt, llms.txt and homepage text measured across 1048 of the most visited websites. Published 11 August 2026. https://visibility100x.com/research/ai-crawler-blocking/
Questions this data answers
How many websites block AI crawlers?
Of 810 of the most visited websites that publish a robots.txt, 229 disallow at least one major AI crawler from the homepage, which is 28.3%. Over the same sample, 1.5% disallow Googlebot. Measured 11 August 2026 across 1048 sites drawn from the Tranco top 1,500.
Which AI crawler is blocked most often?
CCBot, the Common Crawl crawler, at 23% of sites with a robots.txt, followed by Bytespider at 21.5% and GPTBot at 21.4%. The crawlers that fetch pages to answer live questions are blocked far less: OAI-SearchBot at 10% and Claude-SearchBot at 10.9%.
Does blocking GPTBot remove you from ChatGPT?
No. GPTBot is OpenAI's training crawler. OAI-SearchBot is the one that fetches pages for ChatGPT search results, and ChatGPT-User fetches a page when a user asks the assistant to open it. Blocking GPTBot while allowing those two is a coherent position: do not train on us, do cite us. In this sample 84 sites hold exactly that position.
How many sites have an llms.txt file?
121 of 1048, which is 11.5%. That figure only counts files that are actually files: a further 162 domains returned HTTP 200 for /llms.txt while serving an HTML page, so a checker that trusted the status code would have reported 27% adoption, more than double the truth.
Does blocking Google-Extended remove you from AI Overviews?
No, and this is the most costly confusion in the table. Google-Extended governs Gemini and other Google AI products. AI Overviews and AI Mode are built from the ordinary Google index, so ordinary Googlebot access is what governs them. Blocking Google-Extended to stay out of AI Overviews does not work, and blocking Googlebot to achieve it removes you from search entirely.
How much of a homepage can an AI crawler read without JavaScript?
The median homepage in this sample carried 742 words of text with no JavaScript executed. 21.4% carried fewer than 200 words and 18.7% fewer than 100. Among pages with single page application markers in the HTML, 27.8% were under 200 words.
Related: study 1, 98 GEO agencies audited, what each AI crawler is and does, and whether llms.txt does anything yet.