AI Crawler Checker

What the AI crawler checker tests

Enter a page address and the AI crawler checker runs three tests on it. It reads your robots.txt and evaluates it for 18 AI crawlers against the exact path you entered, not only your homepage. It then requests the page once as a normal browser and once under the name of each of the 16 crawlers that fetch pages, with the user-agent each vendor publishes.

The third test reads the page for the controls that stop engines quoting it: noindex, nosnippet, max-snippet:0 and data-nosnippet, in the HTML or in an X-Robots-Tag header. The robots.txt evaluation is the same code a 99Visibility audit runs, including the RFC 9309 rules for a file that is missing, refused or failing.

How to read the result

• The verdict names the most serious problem first: a robots.txt that fails, a search crawler that robots.txt blocks or your firewall turns away, a noindex, then anything we could not confirm.
• AI search crawlers counts how many of 12 got the page: the search crawlers, dual-purpose crawlers such as Googlebot and Bingbot, and the agents that open a page when someone asks an assistant about it.
• Training crawlers blocked counts the 6 that only collect pages for model training. Blocking them is a choice, not an error.
• robots.txt and snippet controls say whether the file answered and which quoting controls the page carries.
• The full report lists every crawler with the robots.txt line that decided it, what your firewall answered, and what blocking it costs, linked to the vendor's own page and marked where the reading is ours rather than the vendor's.

When robots.txt says yes and the firewall says no

robots.txt states a preference; your firewall, CDN or bot protection decides what is served. The checker flags a disagreement when robots.txt allows a crawler, a normal browser gets the page, and the same request under that crawler's name gets a 403 or a challenge page.

OpenAI names the cause in its guidance on allowing its crawlers: services such as Cloudflare and Akamai "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors" (OpenAI). Google sets the same two conditions for AI Overviews and AI Mode, asking that crawling be allowed "in robots.txt, and by any CDN or hosting infrastructure" (Google).

Defaults move too. Since 15 September 2026, Cloudflare's Block and Block on pages with ads settings also apply to Googlebot, Bingbot and Applebot, and a new domain that says it runs ads starts with AI training disallowed and agent crawlers blocked on its ad pages (Cloudflare). Our guide to Cloudflare's AI crawler settings walks through each one.

When we ran this check on the 43 brand websites in our database on 14 September 2026, 8 of the 19 sites behind Cloudflare refused at least one crawler their robots.txt allowed. Five of the eight refused only Bytespider, and one refused seven AI search and agent crawlers, OAI-SearchBot and PerplexityBot among them (our write-up). That is a small sample of our users' sites, not the web.

Training crawlers and search crawlers are separate

Blocking a training crawler does not take you out of AI search. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", and that "each setting is independent of the others", so a GPTBot block keeps your pages out of training while ChatGPT search can still use them (OpenAI).

Anthropic runs Claude-SearchBot and Claude-User apart from its training crawler, ClaudeBot (Anthropic), and Google says Google-Extended "does not impact a site's inclusion in Google Search" (Google). The mistake to look for is a rule written to keep training out that also catches a search crawler, such as Disallow: / under User-agent: * with no group of their own for the search crawlers.

What this check cannot see

• IP verification. Our requests carry each crawler's name but come from our servers, not from the vendor's published IP ranges. A firewall that verifies crawlers by IP, the way Google tells site owners to check Googlebot (Google), may refuse us and still admit the real crawler, so a refused Googlebot or Bingbot is marked unconfirmed, and a pass proves nothing about a rule keyed to IP addresses.
• Agents that skip robots.txt. Perplexity says its user fetcher "generally ignores robots.txt rules" (Perplexity), and OpenAI says robots.txt rules "may not apply" to ChatGPT-User. For those agents, the firewall's answer counts for more than the file.
• JavaScript. The checker reads the HTML your server sends and runs no scripts, so it cannot tell you whether a crawler sees text your page builds in the browser.
• Other pages and other bots. It tests one URL against the 18 crawlers in its list. A rule can block /blog/ and allow /pricing, so run your key pages one at a time.

Run it on every audit

Every 99Visibility audit runs this same check on your brand's website and turns what it finds into fixes in the audit's fix list, on every plan, Free included. The Crawlability page in your dashboard keeps each check you run there, so you can see the day a firewall rule changed. Compare plans and pricing, or read your rules on their own with the robots.txt checker for AI crawlers.