11 August 2026
1048 of the most visited websites: who blocks the AI crawlers
28.3% of sites with a robots.txt block at least one AI crawler, against 1.5% that block Googlebot, so it is a decision rather than a locked door. 145 of them block a crawler that feeds live answers, which is usually not what they meant. 11.5% publish a real llms.txt.
- Method
- robots.txt, llms.txt and the homepage fetched for every domain, twice: once as a declared crawler and once as a browser. Rules evaluated per RFC 9309.
- Data
- Every domain and column published as CSV, both sample frames documented, CC BY 4.0.