AI Crawler List (October 2026): Every Major Bot, What It Does, and How to Control It
An AI crawler is a bot an AI company sends to fetch web pages. AI crawlers do three different jobs: collecting training data, building a search index, and fetching a page because a user asked. This AI crawler list covers the major bots the vendors document as of October 2026: what each one does, whether it follows robots.txt, and whether you can check it against published IP ranges.
Every row comes from the vendor's own page. If a vendor publishes nothing about a bot, the bot is not on the list.
Three jobs, three different decisions
| Job | What it does | What blocking it costs you |
|---|---|---|
| Training | Collects pages that may train future models | Your content stays out of future training. OpenAI, Google and Apple say their search is unaffected |
| Search | Builds the index an AI assistant searches when it answers | You can drop out of that vendor's AI search answers |
| User-triggered fetch | Opens a page because a user asked about it right now | The assistant can't read your page for that user. Several vendors say robots.txt may not apply |
Two entries are not crawlers at all. Google-Extended and Applebot-Extended are robots.txt tokens. They fetch nothing; they tell Google and Apple how they may use what their main crawlers already collected.
The AI crawler list, by vendor
In the robots.txt column, "Yes" means the vendor says the bot follows it. Nobody outside the vendor has measured that.
OpenAI
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| OAI-SearchBot | Search: ChatGPT search | Yes | Published |
| GPTBot | Training | Yes | Published |
| ChatGPT-User | User-triggered fetch | "may not apply" | Published |
| OAI-AdsBot | Checks pages submitted as ChatGPT ads | Visits ad landing pages only | Published |
OpenAI: "Each setting is independent of the others." Blocking GPTBot does not remove you from ChatGPT search. The details are in GPTBot vs OAI-SearchBot, and OAI-AdsBot matters only if you run ChatGPT ads.
Anthropic
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| ClaudeBot | Training | Yes | One list for all three |
| Claude-SearchBot | Search quality | Yes | |
| Claude-User | User-triggered fetch | Yes |
Of the vendors here that run a user-triggered bot, Anthropic is the only one that doesn't carve it out of robots.txt: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." It also warns that blocking its IPs "impedes our ability to read your robots.txt file." More in what ClaudeBot is.
Perplexity
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| PerplexityBot | Search: "not used to crawl content for AI foundation models" | Yes | Published |
| Perplexity-User | User-triggered fetch | "generally ignores robots.txt rules" | Published |
Source: Perplexity's crawler page.
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| Googlebot | Search, including AI Overviews and AI Mode | Yes, "when crawling automatically" | Published |
| Google-Extended | Token: Gemini training and grounding | Token only, no crawler | None needed |
| Google-Agent | User-triggered: agents acting on a user's request | "generally ignore robots.txt rules" | Published |
There is no separate token for AI Overviews or AI Mode. Google: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control." Google-Extended "does not impact a site's inclusion in Google Search", but it also covers grounding in Gemini Apps, so it is not a training-only switch. See how to turn off AI Overviews for the controls Google does offer.
Apple
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| Applebot | Search in Spotlight, Siri and Safari. Its data may also train Apple's models | Yes | Published |
| Applebot-Extended | Token: opt out of training Apple's foundation models | "does not crawl webpages" | None needed |
If your robots.txt doesn't name Applebot but does name Googlebot, Apple says Applebot "will follow Googlebot instructions." To keep content out of AI answers in Siri and Search, Apple points to the nosnippet meta tag, not robots.txt.
Meta
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| Meta-ExternalAgent | Training AI models, or "improving products by indexing content directly" | Yes | Not on Meta's crawler page |
| Meta-WebIndexer | Search: Meta AI | Yes | |
| Meta-ExternalFetcher | User-triggered fetch | "may bypass robots.txt rules" |
Meta says it directly: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."
Amazon
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| Amazonbot | Improves products; "may be used to train Amazon AI models" | Yes | Published |
| Amzn-SearchBot | Search, such as Alexa. Not used for training | Yes | Published |
| Amzn-User | User-triggered fetch for Alexa | "may not follow all robots.txt directives" | Published |
Amazon says changes take about 24 hours, but also that its bots may use a cached copy of your robots.txt "from the last 30 days."
Common Crawl
| User-agent | Job | robots.txt | IP ranges |
|---|---|---|---|
| CCBot | Builds an open web archive anyone can access | Yes | Published |
Source: Common Crawl's CCBot page.
Left off: ByteDance's Bytespider. We could not find a ByteDance page documenting it, so we can't say what it does or whether it follows robots.txt.
Allow or block: decide by job, not by vendor
- Search crawlers: keep them open unless you want out. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot, Applebot and Googlebot feed each vendor's search answers. Allowing one makes you eligible. It does not rank you.
- Training crawlers: a business decision. OpenAI claims no search benefit for allowing GPTBot, and nobody has measured whether blocking a training crawler changes what future models know about your brand.
- User-triggered fetchers: robots.txt is not the control. OpenAI, Perplexity, Google, Meta and Amazon all say theirs may not follow it. To stop one, you need a firewall rule.
- Tokens: read the scope. Applebot-Extended covers training. Google-Extended covers training and Gemini grounding.
To opt out of training and stay in search:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /Add Google-Extended only if losing Gemini grounding is acceptable. Bots without a group of their own fall back to your User-agent: * rules, though Applebot and Amzn-SearchBot first look for rules you gave other search bots. On WordPress, see how to edit your WordPress robots.txt.
How to check an AI crawler can reach your site
- Read your robots.txt per bot. Under RFC 9309, a crawler obeys the group that names it and uses the
*group only "if no matching group exists." - Request a page as each crawler. Send the vendor's user-agent string and look for a 200. A 403 or a challenge page means something in front of your site blocks the bot, whatever robots.txt says.
- Check your logs against the published IP ranges. User-agents can be faked. Google warns "The HTTP user agent string can be spoofed," and Common Crawl is "aware of crawlers falsely identifying themselves as CCBot."
- Check your CDN's bot settings. A default can change without you touching anything, as Cloudflare's AI crawler default change showed.
Step 2 has a limit: your test does not come from the vendor's IP ranges, so a firewall that checks IPs can treat it differently from the real bot.
99Visibility runs steps 1 and 2 for you. It requests your pages as each AI crawler, compares the result with your robots.txt, and flags every place the two disagree. It also audits what five AI platforms say about your brand. Your first audit is free, with no credit card, and covers ChatGPT and Google AI Overviews. See the plans for all five platforms. Inside the dashboard, Agent V reads your audit and your site and drafts the robots.txt lines for you to paste. It never edits your site itself.
FAQ
What is ClaudeBot?
Anthropic's training crawler. It collects web content "that could potentially contribute to" training Anthropic's models. Claude-SearchBot, Anthropic's search crawler, is a separate bot with its own robots.txt group.
What is PerplexityBot?
Perplexity's search crawler, "designed to surface and link websites in search results on Perplexity." Perplexity recommends allowing it if you want to appear in those results.
What is Google-Extended?
A robots.txt token, not a crawler. It controls whether content Google crawls may train Gemini models and ground answers in Gemini Apps. It "does not impact a site's inclusion in Google Search."
Find out which AI crawlers can reach your site. Run a free audit: no credit card needed.
Want more of our research in Google Search and AI Mode? Add us as a preferred source.
Can AI crawlers reach your site?
Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.
Start Free Audit2 of 3 retrieval crawlers can reach you
robots.txt allows 1 crawler your firewall blocks
Related Articles
ClaudeBot: What It Is, What It Collects, and How to Allow or Block It
ClaudeBot is Anthropic's training crawler. Its two siblings, Claude-SearchBot and Claude-User, handle search and live fetches. What each one does, the robots.txt to allow or block each, and how to verify real Anthropic traffic by IP.
GPTBot vs OAI-SearchBot vs ChatGPT-User: Which OpenAI Crawler to Allow
GPTBot collects training data. OAI-SearchBot decides whether ChatGPT search can show your site. ChatGPT-User fetches pages live, and robots.txt may not apply to it. What OpenAI says each one does, a decision table by goal, and the firewall rule robots.txt cannot fix.
WordPress robots.txt: Where It Is, How to Edit It, and the AI Crawler Rules Worth Adding
Where WordPress keeps its robots.txt, how to edit it with Yoast, Rank Math, FTP or code, what "Discourage search engines" does now, which AI crawler rules to add, and why your firewall can overrule the file.