AI Crawler Access Check
The access check answers one question per AI crawler: can it fetch your site? It reads your robots.txt rules for each crawler, requests your page as that crawler to see what your firewall or CDN does, and looks for tags that stop your content being quoted. Everything it reports is subtractive: something that blocks a crawler, never a tweak that promises a boost.
Where it runs
- Crawlability (
/dashboard/crawlability): the AI crawler access card, and Access over time with Run and save a check. Site health is also on this page. - Dashboard: the AI crawler access card.
- Every audit runs the check alongside the AI platforms and feeds the crawler-access recommendations.
The check is on every plan. It needs the brand's website URL and is rate-limited per user per hour; the error says how long to wait.
Crawlers checked
| User agent | Vendor | Product | Role | Firewall probe |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search | retrieval | Yes |
GPTBot | OpenAI | OpenAI model training | training | Yes |
ChatGPT-User | OpenAI | ChatGPT on-demand browsing | retrieval | Yes |
ClaudeBot | Anthropic | Claude model training | training | Yes |
Claude-SearchBot | Anthropic | Claude search | retrieval | Yes |
Claude-User | Anthropic | Claude on-demand browsing | retrieval | Yes |
PerplexityBot | Perplexity | Perplexity search index | retrieval | Yes |
Perplexity-User | Perplexity | Perplexity on-demand browsing | retrieval | Yes |
Googlebot | Google Search, AI Overviews, AI Mode | both | Yes | |
Google-Extended | Gemini and Vertex AI training/grounding | training | robots.txt only | |
Bingbot | Microsoft | Bing, Copilot, ChatGPT browsing | both | Yes |
Applebot-Extended | Apple | Apple foundation model training | training | robots.txt only |
Meta-WebIndexer | Meta | Meta AI search | retrieval | Yes |
Meta-ExternalAgent | Meta | Meta AI training and indexing | both | Yes |
Bytespider | ByteDance | ByteDance/TikTok model training | training | Yes |
CCBot | Common Crawl | Common Crawl corpus (used by many LLM trainers) | training | Yes |
Amazonbot | Amazon | Alexa answers and Amazon services | both | Yes |
DuckAssistBot | DuckDuckGo | DuckAssist answers | retrieval | Yes |
Role retrieval fetches pages to answer a user now; training collects data for future models; both does both. Blocking a training crawler does not remove you from that vendor's live answers, and allowing one does not get you cited.
What is tested
robots.txt
Each crawler's rules are evaluated for the page's path. A robots.txt that returns 5xx or cannot be fetched counts as blocking every crawler. One that returns 404 or 410, or 401 or 403, counts as no rules.
Firewall and CDN
The page is requested once per crawler with that crawler's user agent, plus once as a normal browser for control. A crawler counts as blocked on HTTP 401, 403, 429 or 503, or on a bot-challenge page from Cloudflare, Akamai, Imperva or Vercel. Requests run a few at a time with a per-request and an overall timeout; crawlers not reached in time are reported "Not run".
Suppression
Meta robots tags (robots, googlebot, googlebot-news, bingbot), the X-Robots-Tag header and data-nosnippet. Severity: noindex or none critical; nosnippet and max-snippet:0 high; a short max-snippet medium.
Output
| Field | Meaning |
|---|---|
robots.perAgent | Per crawler: Allowed, Blocked or No rule. |
waf.perAgent, waf.control | Per crawler: the status or challenge vendor, Timed out, or Not run. The browser control's result. |
suppression | Each suppression finding with its severity. |
disagreements | Crawlers robots.txt allows but the firewall blocks, while the browser control got through. The finding no other tool reports. |
summary | Retrieval crawlers reachable out of the total, blocked by robots.txt, blocked by firewall, a blanket disallow, and whether the control was blocked. |
The card shows these as "N of M retrieval crawlers can reach you", a per-crawler table (Crawler, Vendor, Role, robots.txt, Firewall, If blocked) and an Accidental suppression list.
Limits of the check
- A pass is not proof. The request carries the crawler's user agent but not its IP addresses, so a firewall that allowlists by IP can treat the real crawler differently.
- If the control request is refused, firewall verdicts are not crawler-specific until that is fixed. The card says so.
- robots.txt is necessary, not sufficient. Some user-triggered fetchers say they may not follow it, which is why the firewall test exists.
Score not moving after a site change? Score not improving after your changes starts with this check.