AI Crawler robots.txt Checker

What this robots.txt checker tells you

It reads your robots.txt and evaluates it for 18 crawlers: 12 that fetch pages for AI search and answers, and 6 that collect pages for model training. For each one you see allowed or blocked, and the line in your file that decided it.

It uses the same evaluator as the access check in a 99Visibility audit, including the RFC 9309 rules for a robots.txt that is missing, forbidden or failing. So this free check and a paid audit always agree about what your file says.

Why AI crawlers need their own check

AI search and AI training are separate crawlers with separate controls. OpenAI documents it plainly: sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers", and "each setting is independent of the others", so allowing GPTBot does nothing for search and blocking it does not remove you (OpenAI).

A common mistake is a blanket block written to keep AI training out. It also removes the site from ChatGPT search, Claude search and Perplexity, because it never names which bot does what.

How to read your result

• Allowed: a rule in your file lets this crawler fetch your homepage.
• Allowed (no rule): nothing in your file applies, so the crawler may fetch by default.
• Blocked: a Disallow rule, either in the crawler's own group or the User-agent: * group, stops it. The deciding line is shown.

The check evaluates your homepage path. A rule that blocks only /blog/ or /pricing will not show here, so read your file for paths you care about.

What robots.txt cannot tell you

robots.txt is permission, not access. A crawler your file allows can still be turned away by your firewall or CDN with a 403. OpenAI asks for both: allow OAI-SearchBot, and "confirm that the website host or content delivery network allows traffic" from its crawler (OpenAI).

This matters more since September 2026, when Cloudflare began blocking AI training and agent crawlers by default on ad-monetized pages, for new customers, new sites and free-plan sites. A site can lose AI access without anyone editing robots.txt. Here is what changed and how to check it.

This free tool reads one file and requests nothing else from your site. A 99Visibility audit requests your site as each AI crawler and shows every place robots.txt says yes and the network says no.

Crawlers that may not follow robots.txt

Some agents fetch a page because a person asked for it, and their vendors say robots.txt may not apply. OpenAI says this of ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt (Perplexity). The checker marks these so a "Blocked" result for them is not read as a guarantee.

Fixes that come up most

1. Remove a blanket block you did not mean. A Disallow: / under User-agent: * blocks every crawler, search included.
2. Allow the search crawlers by name if you block training. Name OAI-SearchBot, Claude-SearchBot and PerplexityBot in their own groups.
3. Decide training separately. Blocking GPTBot, ClaudeBot or Google-Extended keeps your pages out of training. It does not remove you from AI search.
4. Make robots.txt answer 200 or 404. A 5xx or a timeout tells crawlers to treat your whole site as disallowed until it recovers.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

That file keeps OpenAI's training crawler out and lets the three search crawlers in. Every other bot falls back to your User-agent: * group, or to no rules if you have none.

Does robots.txt block Google AI Overviews?

No token does. AI Overviews and AI Mode use Googlebot and Google's search index, and Google says a page must be indexed and eligible for a snippet to appear (Google). Google-Extended controls Gemini and Vertex AI training and grounding; Google states it does not affect Google Search. To keep a page out of AI Overviews, the controls are nosnippet, max-snippet or noindex, which also change how it appears in normal results.