ClaudeBot: What It Is, What It Collects, and How to Allow or Block It
ClaudeBot is the web crawler Anthropic uses to collect public web content that may be used to train Claude. It has two siblings with different jobs. Claude-SearchBot indexes pages for Claude's search results. Claude-User fetches a page when a person asks Claude a question. Each one answers to its own group in your robots.txt.
That split is the whole decision: you can stay out of training and still let Claude reach you when someone asks about you. Below is what each bot does, the robots.txt for each choice, and how to check that your server agrees. Everything here is as of October 2026.
This page is about Anthropic's web crawler, not an AI assistant with a similar-sounding name.
What is ClaudeBot?
Anthropic's crawler page, last updated 7 April 2026, describes it: "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."
The system cards are plainer. The Claude Opus 5.5 system card (22 September 2026): "We use a general-purpose web crawler called ClaudeBot to obtain training data from public websites."
ClaudeBot is a training crawler. Anthropic's documentation attaches no search consequence to it.
What ClaudeBot collects, and what it skips
Public pages. The same system card says: "We do not access password-protected pages or those that require sign-in or CAPTCHA verification." Anthropic's crawler page adds that its bots "respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs for the sites we crawl.)"
Anything behind a login or a CAPTCHA is out of reach by Anthropic's own rules. So is any page your firewall answers with a challenge, which matters most for the two siblings.
ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic runs three robots, and says why: so site owners can "enable those you want to access your content and limit those you don't."
| Bot | Job, per Anthropic | What blocking it does, per Anthropic |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | "Signals that the site's future materials should be excluded from our AI model training datasets" |
| Claude-SearchBot | "Navigates the web to improve search result quality for users" | Stops Anthropic indexing your content for search, which "may reduce your site's visibility and accuracy in user search results" |
| Claude-User | Fetches a page when a person asks Claude a question | Stops Claude retrieving your content for a user's query, which "may reduce your site's visibility for user-directed web search" |
What blocking ClaudeBot does not do
- It does not pull your content out of models already trained. Anthropic's wording is "future materials".
- It does not block Claude-SearchBot or Claude-User. They are separate robots with separate robots.txt groups.
- It has no documented effect on Claude's search answers, in either direction. Anthropic does not claim that allowing ClaudeBot helps you appear in Claude, and we know of no published measurement of whether training data changes what a model says about a brand.
"Allow ClaudeBot so Claude knows about you" is not a claim anyone can back with evidence today. Blocking it is a decision about training. Treat it as one.
OpenAI splits its crawlers the same way, with one difference in how its live-fetch agent treats robots.txt: see GPTBot vs OAI-SearchBot.
The ClaudeBot user agent
Anthropic's crawler page names the three agents: ClaudeBot, Claude-SearchBot and Claude-User. Those names are what your robots.txt matches. As of October 2026 the page does not print a full user-agent string for any of them, so match on the name and confirm identity by IP.
Older guides also list names such as anthropic-ai and Claude-Web. Neither appears on Anthropic's current page. If your robots.txt names only those, it does not name any of the three bots Anthropic documents today.
How to allow or block ClaudeBot in robots.txt
Anthropic says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." The file goes at the root of each host, and Anthropic is explicit that this is per subdomain: "Please do this for every subdomain that you wish to opt out from."
Out of training, still reachable for Claude's answers
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Block all three
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /By Anthropic's own description, blocking the two siblings may reduce your visibility in Claude's search results and live answers.
Slow it down instead
Anthropic supports what it calls "the non-standard Crawl-delay extension to robots.txt", and gives this example:
User-agent: ClaudeBot
Crawl-delay: 1Watch for the blanket rule. A User-agent: * group with Disallow: /, or a copied "block every AI bot" list, can switch off Claude-SearchBot and Claude-User when all you meant to stop was training. Name each bot in its own group. For the bots other vendors run, see our AI crawler list.
Why blocking by IP is the wrong tool
Firewalling Anthropic's IP ranges looks like the stronger option. Anthropic warns against it: "Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."
The same mechanism catches sites that never meant to block anything. If a firewall rule challenges Claude-SearchBot, nobody solves the CAPTCHA: Anthropic says its bots will not try. Your robots.txt says allow, your firewall says no, and the firewall wins. A CDN default can do this without you touching a setting; see Cloudflare's AI crawler default.
How to tell real ClaudeBot traffic from spoofed traffic
Anyone can put "ClaudeBot" in a user-agent header. The name in your logs proves nothing.
Anthropic publishes the check: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic." The list is at claude.com/crawling/bots.json. As of October 2026 it holds 38 IPv4 prefixes and no IPv6 ranges.
- Pull requests naming ClaudeBot, Claude-SearchBot or Claude-User from your server or CDN logs.
- Check each source IP against a fresh copy of bots.json. The file carries a creation date and changes.
- Treat a match as Anthropic. Treat a miss as unverified, and handle it like any other unknown bot.
Anthropic's page offers the IP list as its verification method. It does not document a reverse-DNS check.
How to check that Claude can reach your site
Your robots.txt is what you ask for. Your server's response is what happens.
- Read your live robots.txt for the three names, and for any wildcard group that catches them.
- Request a page with each name as the user agent. A 200 passes; a 403 or a challenge page is a block. From a terminal:
curl -s -o /dev/null -w "%{http_code}" -A "Claude-SearchBot" https://www.example.com/ - Request /robots.txt the same way. If the crawler cannot read the file, it cannot read your permissions.
- Check your logs for the real thing. A test from your own machine only exercises rules keyed on the user agent. IP-based rules show up only in what your server returned to Anthropic's listed addresses.
- Check the page source, not the rendered page. In Vercel's December 2024 analysis of crawler traffic on its network, Claude's crawlers fetched JavaScript files but did not run them: "They can't read client-side rendered content." Anthropic's page says nothing about JavaScript, so put your key facts in the raw HTML.
By hand, that is slow. 99Visibility checks whether AI crawlers can reach your site and compares what your robots.txt says with what your firewall actually returns, alongside an audit of what five AI platforms say about your brand. Your first audit is free, with no credit card, and covers ChatGPT and Google AI Overviews. See the plans for all five platforms. Inside the dashboard, Agent V reads your audit and your site and drafts the robots.txt lines for you to paste. It never edits your site itself.
FAQ
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train Claude. Claude-SearchBot (search indexing) and Claude-User (live fetches) are separate bots.
What is the ClaudeBot user agent?
The name to match is ClaudeBot. As of October 2026, Anthropic's crawler page lists the agent names but no full user-agent string. Verify the source IP against claude.com/crawling/bots.json rather than trusting the header.
Does blocking ClaudeBot remove my site from Claude?
Not from Claude's search, according to Anthropic's documentation. Blocking ClaudeBot signals that your future content should be excluded from training. Search indexing and live fetches use Claude-SearchBot and Claude-User, which follow their own robots.txt groups.
See whether AI crawlers can reach you. Run a free audit: no credit card.
Want more of our research in Google Search and AI Mode? Add us as a preferred source.
Can AI crawlers reach your site?
Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.
Start Free Audit2 of 3 retrieval crawlers can reach you
robots.txt allows 1 crawler your firewall blocks
Related Articles
GPTBot vs OAI-SearchBot vs ChatGPT-User: Which OpenAI Crawler to Allow
GPTBot collects training data. OAI-SearchBot decides whether ChatGPT search can show your site. ChatGPT-User fetches pages live, and robots.txt may not apply to it. What OpenAI says each one does, a decision table by goal, and the firewall rule robots.txt cannot fix.
AI Crawler List (October 2026): Every Major Bot, What It Does, and How to Control It
Every major AI crawler as of October 2026, from the vendors' own pages: what each bot does (training, search or a user-triggered fetch), whether it follows robots.txt, whether its IP ranges are published, and how to decide what to allow.
Should I Block AI Crawlers on Cloudflare? A Per-Bot Decision Guide
Block training crawlers if you object to training; keep search and agent crawlers open to stay in AI answers. What Cloudflare changed on September 15, 2026, per crawler, and how to check yours.