Can AI Read Your Website?

Check in one click whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI bots are allowed to read your site.

Measuring rankings in

Runs the full audit on your live site. Your report opens in your dashboard.

What this checks

This fetches your live robots.txt and evaluates it the way a crawler does, once for each of seven named AI user agents: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bytespider. It applies the real matching rules, so a Disallow under a wildcard group counts against every bot that has no group of its own, and a bot with its own group is judged only on that group. If you serve no robots.txt at all, every crawler is allowed by default and the check says so.

Why it matters

An assistant that cannot fetch your pages cannot quote them. This is the one part of AI visibility that is binary rather than a matter of degree: no amount of good content compensates for a Disallow line, and the block is usually accidental. It arrives in a staging config, a security plugin, a template copied from a site that made a deliberate choice, or a robots.txt written years before these crawlers existed.

How to read a bad result

Blocked bots are listed by name, so you know which assistants are affected rather than just that something is wrong. Three or more blocked is scored as a failure because at that point most of the generative surface is closed to you. Fixing it is one edit to robots.txt and takes effect on the crawler's next visit, which is usually days rather than weeks. Blocking some crawlers on purpose is a legitimate choice, and the report tells you the state rather than telling you what to want.

Crawler access is binary and it is one of three checks in the engine carrying the heaviest weight, alongside reachability and indexability. The full audit runs it with the other thirty-five, so you find out in the same pass whether the pages those crawlers are now allowed to fetch are worth fetching.

All 36 checks, free Plans

A failing result, and what it costs

A robots.txt from a WordPress site whose security plugin has a “block AI scrapers” toggle. Nobody in the marketing team knew it was on.

Googlebot is fine, because the only rule it matches is the wildcard group and that group blocks nothing but the admin path. GPTBot and CCBot each have a group of their own, and a bot with its own group is judged on that group alone, so the wildcard rules above are irrelevant to them. The report names both agents rather than saying access is restricted, because the two blocks are different decisions: CCBot feeds a training corpus, GPTBot fetches pages OpenAI may cite. Reversing one and not the other is a coherent position; having neither on purpose is not.

User-agent: *
Disallow: /wp-admin/

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

How to fix it

Decide per agent, not in general.Write down which of the seven you mean to allow. Training crawlers and retrieval crawlers have different consequences, and a blanket answer is almost always somebody else's default rather than your decision.
Remove the group rather than adding an Allow line.Deleting the GPTBot group returns it to the wildcard rules, which is usually what you wanted. An explicit empty Disallow: under the agent does the same thing if you prefer the rule to be visible.
Check what the plugin or CDN is doing.Security suites, hosting dashboards and CDN bot-management rules all write these lines. Edit the source that generates the file, or your change is reverted on the next update.
Re-request the file and read the body.A file that returns HTML is not being served as robots.txt at all, and no crawler is reading what you think it is.
Wait for the recrawl.The change takes effect on the crawler's next visit, which is days rather than weeks. Nothing needs resubmitting.

Questions people ask

Does blocking GPTBot remove me from ChatGPT?

It stops OpenAI crawling new pages from your site, so new content cannot be cited. It does not retroactively erase what a model already learned during training, and it does not remove third party pages that talk about you. In practice a block means you lose ground slowly rather than disappearing at once.

Which AI crawler matters most?

GPTBot and OAI-SearchBot for ChatGPT, Google-Extended for Gemini and AI Overviews, ClaudeBot for Claude and PerplexityBot for Perplexity. If you only fix one thing, unblock the two that feed the assistant your buyers actually use.

Should I block AI crawlers?

Publishers whose revenue depends on people arriving at the page have a real argument for blocking. Businesses that want to be recommended by name do not. Both are defensible, which is why this tool reports the state of your robots.txt rather than grading it.