AI Crawlers and robots.txt, Bot by Bot

Somewhere in your robots.txt there may be a block that someone pasted in a year or two ago, from a post about stopping AI companies scraping your site. It names five or six AI user-agents and disallows all of them. Whoever added it believed they were opting out of model training. In the same lines, they opted out of ChatGPT search and the Perplexity index.

That is the most common real-world access error, and it is silent. Nothing breaks. The site loads, Google still ranks it, and no dashboard shows a warning. ChatGPT search stops showing your pages, and you find out months later when a prospect mentions that ChatGPT recommended your competitor.

This lesson sits in Stage 1 of the model from Where AI Answers Come From: access. Access is the stage with the strongest evidence behind it, and robots.txt is its first gate. By the end you will have read your own file the way each crawler reads it.

Three kinds of AI bot, three different jobs

AI vendors do not send one crawler. They send several, each with its own job and its own robots.txt token. Sort them into three kinds before you change anything.

• Training crawlers collect pages that may be used to train future models. GPTBot and ClaudeBot are the main ones, and Google-Extended is the equivalent control for Gemini. What they collect can only ever reach Stage 0, the model's built-in knowledge, and only after a future training run. Blocking them has no documented effect on AI search.
• Search crawlers build the index an AI product searches when it answers with live sources: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Meta-WebIndexer for Meta AI, plus Googlebot and bingbot behind Google's and Microsoft's surfaces. These decide whether you can be retrieved and cited at all.
• User-triggered fetchers go out when a person asks the assistant to read a page, or the assistant opens one while answering: ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher. They act for a user, and several vendors say robots.txt may not bind them.

The rule that follows is simple. A robots.txt decision applies to one bot and does not transfer. Blocking a training bot does not block search. Allowing a training bot does not buy search. OpenAI says so in writing.

Bot by bot: what each block costs

Here is every crawler that matters for AI visibility, with the consequence its own vendor documents. Where a vendor does not state the consequence, the table says so.

Token	Vendor	Kind	What blocking it does
OAI-SearchBot	OpenAI	Search	OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
GPTBot	OpenAI	Training	Signals that your content should not be used to train OpenAI foundation models. OpenAI claims no search effect, in either direction.
ChatGPT-User	OpenAI	User-triggered	"Not used to determine whether content may appear in Search." OpenAI adds: "Because these actions are initiated by a user, robots.txt rules may not apply."
ClaudeBot	Anthropic	Training	Keeps your pages out of Anthropic's training crawls. Search and user fetches run as separate agents.
Claude-SearchBot	Anthropic	Search	Anthropic says it improves search result quality but does not state what blocking it does. Losing eligibility for Claude's search results is our reading, not a quote.
Claude-User	Anthropic	User-triggered	Fetches a page when a Claude user's request needs it. The consequence of blocking it is not spelled out.
PerplexityBot	Perplexity	Search	Takes you out of the Perplexity search index. Perplexity says it is "not used to crawl content for AI foundation models."
Perplexity-User	Perplexity	User-triggered	Perplexity: "this fetcher generally ignores robots.txt rules." A block here is not a reliable control.
Google-Extended	Google	Training token	Google documents it as cutting both training and grounding for Gemini Apps and Vertex AI. It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Googlebot	Google	Search	Takes you out of Google Search, and with it AI Overviews and AI Mode. They have no token of their own.
bingbot	Microsoft	Search	Takes you out of the Bing index, which Microsoft documents as the backing store for Copilot and Grounding with Bing Search. Its link to ChatGPT is a documented upstream with an unquantified effect.
Meta-WebIndexer	Meta	Search	Meta: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." Blocking it closes that path.
Meta-ExternalFetcher	Meta	User-triggered	Meta documents that it may bypass robots.txt.

Four rows catch people out.

There is no robots.txt token for AI Overviews

Google answers this one directly: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control." You cannot stay in Google Search and block AI Overviews with robots.txt. Block Googlebot and you leave Google Search entirely. The controls that limit what AI Overviews may show from your pages are snippet directives and a Search Console setting, covered in Rendering, noindex and Snippet Controls.

Google-Extended is a token, not a crawler

Nothing fetches your site as Google-Extended. Googlebot does the fetching, and the Google-Extended line tells Google what it may do with the result for Gemini. Read the word grounding in Google's description. Blocking it is not a pure training opt-out, because Google documents it also cutting grounding in Gemini Apps. Decide on it deliberately rather than pasting it in with the rest.

Meta-WebIndexer carries the only positive promise

Across every vendor document in our evidence base, Meta's sentence about Meta-WebIndexer is the only first-party statement that allowing a crawler helps you get cited. Every other vendor describes eligibility, not benefit. Meta also runs Meta-ExternalAgent, which it documents for training and indexing without spelling out what a block costs. Check both, and do not block Meta-WebIndexer by accident.

Some fetchers do not treat robots.txt as binding

OpenAI says robots.txt "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." Meta says Meta-ExternalFetcher may bypass it. Cloudflare calls compliance an open question: "others might be unsure whether these bots even respect robots.txt rules, especially newer or less transparent crawlers." Treat robots.txt as necessary and not sufficient. A hard block lives in your firewall, which is the next lesson.

How a crawler reads your file

Most robots.txt errors are not in what the owner wanted. They are in how the file is parsed. Crawlers follow the robots.txt standard, RFC 9309, and two of its rules surprise almost everyone.

Rule 1: the most specific group wins, and a named group replaces *

A robots.txt is a set of groups. A group starts with one or more User-agent lines and holds the Allow and Disallow rules beneath them. A crawler looks for the group that names its own token, ignoring case. If one exists, the crawler uses it and ignores the * group completely. Only a crawler with no group of its own falls back to *. If two groups name the same bot, their rules are combined.

This is where well-meant edits break. Say your file has User-agent: * with Disallow: /app/, and you add a group for OAI-SearchBot that contains only Allow: /. OAI-SearchBot now reads its own group and nothing else, so the /app/ rule no longer applies to it. A named group does not add to the * rules. It replaces them. Repeat every shared rule inside each named group.

The reverse trap is a file that opens with User-agent: * and Disallow: /, then names Googlebot and bingbot to let them in. Every AI search bot you did not name falls back to * and is blocked.

Rule 2: the longest matching path wins

Inside its group, a crawler does not stop at the first rule that matches. It uses the rule whose path matches the most characters, and on a tie, Allow wins. So Disallow: / followed by Allow: /blog still lets the crawler read /blog. A checker that stops at the first Disallow reports a false block. In paths, * matches any run of characters and a trailing $ anchors the end. An empty Disallow: blocks nothing.

Two more details. Several User-agent lines in a row form one group, so you can give several bots the same rules once. And a robots.txt covers only the host that serves it: docs.yourdomain.com and www.yourdomain.com each need their own.

A file that opts out of training and stays in AI search

Here is what most people meant when they pasted a blanket block: keep content out of training sets, stay eligible for every AI search index, and keep private paths private for everyone.

# Training: opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Gemini. Google documents this also cuts grounding in Gemini Apps.
# Keep it or remove it as a deliberate decision.
User-agent: Google-Extended
Disallow: /

# Search crawlers: stay eligible for AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: Googlebot
User-agent: bingbot
Disallow: /app/
Disallow: /checkout/

# User-triggered fetchers
User-agent: ChatGPT-User
User-agent: Claude-User
Disallow: /app/
Disallow: /checkout/

# Everyone else
User-agent: *
Disallow: /app/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

Notice what it does not do. It never gives the search bots a bare Allow: /, because that group would replace * and drop the private-path rules; it repeats them instead. It leaves out Perplexity-User, because Perplexity says that fetcher generally ignores the file anyway. And it is a statement of intent for bots that comply, not a lock.

Many CMSs and hosting platforms generate robots.txt for you and can override a file you upload. After you deploy, fetch the live file, because that is the only version any crawler sees.

What a robots.txt fix can and cannot do

Be clear about the size of this lever before you report it to anyone.

It is a gate, not a boost. Allowing OAI-SearchBot buys eligibility for ChatGPT search. It does not buy retrieval for any given query, a ranking, or a citation. No markup, wording or file makes a page more eligible than eligible. What robots.txt can do is make a page ineligible, which is why ten minutes on it is worth it.

It only reaches answers built from live retrieval. When a model answers from training data alone, nothing you change this quarter reaches that answer. Opening search crawlers changes what live answers can draw from. Remember that when you read your own numbers: 99Visibility queries ChatGPT and Claude through their APIs without web search, so those columns show what the models already know and will not reflect an OAI-SearchBot fix. Perplexity is queried with web search, and Gemini with Google Search grounding available. So do not wait for a score to tell you the fix worked. Verify it directly: fetch the live file, and check each bot.

It is half of a two-part gate. OpenAI's eligibility rule joins robots.txt to your firewall with the word "and." A bot that robots.txt allows and your CDN refuses is still locked out.

The 99Visibility Crawler Access check reads your robots.txt per agent with the matching rules above, separates training bots from retrieval bots, and shows each vendor's documented consequence beside the verdict, labeled as quoted or inferred.

Common mistakes

• Pasting a "block all AI" snippet. These lists almost always include OAI-SearchBot and PerplexityBot. You leave AI search while believing you only left training.
• Adding a named group with only Allow: /. The named group replaces *, so your private-path rules stop applying to that bot.
• Blocking * and naming only Googlebot and bingbot. Every AI search bot you did not name falls back to * and is locked out.
• Trying to block AI Overviews in robots.txt. There is no token. Blocking Googlebot takes you out of Google Search entirely.
• Treating robots.txt as a lock. User-triggered fetchers may ignore it. Hard blocks belong in the firewall.
• Checking the file in your repo instead of on the live domain. Platforms rewrite robots.txt, and staging files reach production. Read what the domain serves.