Fetch and parse your live robots.txt: whether it exists, whether it is valid, which crawlers it restricts and whether it points at your sitemap.
Measuring rankings in
Runs the full audit on your live site. Your report opens in your dashboard.
This requests robots.txt from the root of your domain, parses it with the same grouping and precedence rules a crawler applies, and reports three things: whether a valid file came back at all, which user agent groups it defines and what they disallow, and whether it declares a Sitemap line. It runs beside the AI crawler check so you can see the general rules and their specific effect on assistants in one report.
robots.txt is the first file a crawler requests and the cheapest place on a site to cause serious damage. A single Disallow: / left over from a staging environment removes a site from search entirely, and because nothing visibly breaks it can survive for months. The Sitemap line is the other half: it is how a crawler that arrived without knowing your URL structure finds the list of everything you want indexed.
A missing robots.txt is a warning rather than a failure, because nothing is blocked by default and plenty of healthy sites do not have one. What deserves immediate attention is a broad Disallow, a rule that blocks the assets a page needs to render, or a file that returns your homepage instead of plain text, which means it is not being served correctly and no crawler is reading what you think it is reading.
This page leads with one dimension, and the audit behind it is the whole engine: Visibility100x scores36 checks on every page it crawls, across crawlability, on page SEO, structured data, AI and GEO, content, media, performance and security. Running it here gives you the same report a paying customer gets, on a free account with no card.
No. It stops the page being crawled. A blocked URL that is linked from elsewhere can still appear in results, with no description, because the crawler was never allowed to read it. To keep a page out of the index, allow the crawl and serve a noindex directive.
Not strictly. Without one everything is allowed, which is fine for most small sites. It becomes worth having when you want to declare a sitemap or keep crawlers out of search result pages and other infinite URL spaces.
At the root of the host, exactly /robots.txt, served as plain text. A file in a subdirectory is not read by anything, and neither is one that a framework returns as HTML.
More free tools in crawling and indexing, and the rest of the set at all free tools.
Free account, no card. Paste your URL and get a real, scored report of your AI and search visibility.
Measuring rankings in