Set each AI crawler separately, declare your sitemap, and copy out a file that says what you meant. It runs in your browser and sends nothing anywhere.
A robots.txt tells crawlers what they may fetch, and it does it per crawler. The part most generators get wrong is that a bot with a group of its own ignores theUser-agent: * rules entirely, so a single wildcard block cannot express “index me on Google, do not train on me”. This one writes named groups, which is the only shape that can.
Your rules
Nothing here affects whether a page can be indexed if it is linked from elsewhere. These are the URL spaces that waste a crawl rather than the pages you want hidden.
Every agent starts allowed, which is also what happens if you serve no file at all. Blocking a training crawler and blocking a retrieval crawler are different decisions: the first affects whether a model learns about you, the second affects whether an assistant can quote you this week.
Googlebot ignores this. Bing and several smaller crawlers honour it. Setting it because a server is struggling is a workaround for a capacity problem rather than a fix.
# robots.txt
# Generated by visibility100x.com/robots-txt-generator/
User-agent: *
Disallow:
Save this as robots.txt at the root of your domain, served as plain text. A file in a subdirectory is read by nothing, and one your framework returns as HTML is read as nothing.
Everything that goes wrong with a robots.txt goes wrong in one of these four places.
A crawler picks the one group whose user agent matches it most specifically and obeys that group alone. It does not merge the wildcard rules in underneath. This is set out inRFC 9309, and it is why a Disallow: /wp-admin/ under User-agent: * stops applying to GPTBot the moment GPTBot gets a group of its own.
Within a group, the rule with the longest matching path decides, whichever order the lines are in. So Allow: /blog/public/ beats Disallow: /blog/for a URL under the first, and shuffling the lines changes nothing. People reorder these files expecting a change and get none.
A disallowed URL can still be listed in results if something links to it, because the crawler was never allowed in to read the noindex you were relying on. Blocking and de-indexing are opposite instructions that have to be given in the right order: allow the crawl, then serve the directive.
Compliance is voluntary. Well behaved crawlers honour it and nothing stops one that does not, so this file is for managing crawl behaviour rather than for protecting anything. Private data needs authentication. This needs a text editor.
When we ran the same check over 1,048 of the most visited websites,28.3% blocked at least one major AI crawler while 1.5% blocked Googlebot, and 26.8% did the first while permitting the second. Read the files and almost none of it looks like a decision. It arrives with a security plugin, a hosting default, or a list copied from a site that had a reason.
That is why every agent in the form above starts allowed, which is also what happens if you serve no file at all. A generator that pre-ticks “block” would be manufacturing more of exactly what we measured. It is also why the agents are labelled with what they feed rather than only by name: blocking a training crawler and blocking a retrieval crawler have completely different consequences, and a list of user agent strings with no indication of which is which is how these files end up wrong.
Publishers whose revenue depends on people arriving at the page have a real argument for blocking. A business that wants to be recommended by name does not. Both are defensible; neither should be arrived at by accident. The full measurement is inwho blocks the AI crawlers, andwhich crawler belongs to which assistanthas each operator's own documentation behind every agent named above.
Three checks, in this order, and all three are free.
/robots.txt and serves HTML. Therobots.txt checker reads the body rather than the status code, which is the distinction that catches it.A plain text file at the root of a domain that tells crawlers which parts of the site they may request. It controls crawling, not indexing, and it is a request that well behaved crawlers honour rather than an access control. Anything you genuinely need to keep private needs authentication, not a Disallow line.
Not strictly. With no file, everything is allowed, which is correct for most small sites. It becomes worth having the moment you want to declare a sitemap, keep crawlers out of search result pages and other infinite URL spaces, or give a specific AI crawler different rules from the rest.
Give the crawler its own named group and disallow everything in it. The generator above does this, and the reason it uses named groups rather than one wildcard rule is that a crawler obeys the most specific group matching it and ignores all the others, so a bot with its own group never reads the wildcard rules at all.
No. Google-Extended governs Gemini training and grounding in some of Google’s other products. AI Overviews follow ordinary Googlebot access and the standard snippet controls, so blocking Google-Extended costs you one thing and does not achieve the other. Getting these two confused is the most common mistake in this file.
No, and this catches people out. It stops the page being fetched. A blocked URL that is linked from elsewhere can still be listed, with no description, precisely because the crawler was never allowed to read it. To keep a page out of the index, allow the crawl and serve a noindex directive so the instruction can actually be read.
At exactly /robots.txt on the host, served as plain text. A file in a subdirectory is read by nothing, and a framework route that answers that path with HTML is not a robots.txt at all even though it returns 200. Check the response body rather than the status code.
No. Everything on this page runs in your browser and calls nothing, because assembling a text file from a form is deterministic string building and there is no reason for a server to be involved. Close the tab and nothing about what you typed exists anywhere.
Free account, no card. Paste your URL and get a real, scored report of your AI and search visibility.
Measuring rankings in