Should I Block AI Crawlers on Cloudflare? A Per-Bot Decision Guide

The short answer to "should I block AI crawlers on Cloudflare?" is: block the training crawlers if you do not want your content used to build future models, and keep the search and agent crawlers open if you want ChatGPT, Claude, Perplexity and Google to find, fetch and cite your pages. Since September 15, 2026 there is one trap: setting Training to Block on Cloudflare now also blocks Googlebot, Bingbot and Applebot, so the setting that keeps search and refuses training is Disallow AI Training.

We first published this on September 13, two days before the change, working from Cloudflare's July announcement. This version, updated October 16, 2026, describes what Cloudflare actually shipped, adds a per-crawler decision table and the exact dashboard steps, and reports the crawler data we captured the day before.

What changed on Cloudflare on September 15, 2026

Cloudflare's September 15 announcement on mixed-use crawlers made four changes:

1. Block now reaches Googlebot, Bingbot and Applebot. Cloudflare: "Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training."
2. A new Training option, Disallow AI Training. It writes a no-training preference into your robots.txt and blocks training-only crawlers at the edge, while those three mixed-use crawlers stay allowed for search.
3. "Block AI Bots" is deprecated in favor of three separate controls: Search, Training and Agent.
4. Managed robots.txt becomes Bot Preference Sync, which writes rules at the top of your robots.txt to match those three controls.

New domains: two presets, chosen by whether you run ads

New domains are now asked whether they monetize with ads and offered one of two presets, which can be changed at any time.

Setting	Site without ads	Site with ads
Search	Allow	Allow
Training	Allow	Disallow AI Training
Agent	Allow	Block on pages with ads
Bot Preference Sync	On	On

"Block on pages with ads" relies on Cloudflare's own automated ad detection, per its AI bot policy documentation. You do not pick the pages it covers.

Existing domains: your old setting was translated

If your site was already on Cloudflare, the new controls were filled in from your legacy Block AI Bots setting. Cloudflare's summary: "Your current settings carry over on their own."

Your old Block AI Bots setting	Search	Training	Agent
Off	Allow	Allow	Allow
Block	Allow	Disallow AI Training	Block on pages with ads
Block on pages with ads	Allow	Disallow AI Training	Block on pages with ads

Read the last two rows. The old setting only covered training crawlers, yet if you had it on, your Agent control is now Block on pages with ads, so chat fetchers such as ChatGPT-User are refused on any page Cloudflare detects as showing ads. Domains that had already set the granular controls kept their Search and Agent choices, and a Training block became Disallow AI Training.

What did not happen, and a correction

Cloudflare's July 1 press release said the new defaults would also apply to "all existing free customers that have not changed their settings by September 15, 2026." Its later posts describe something different: existing domains were migrated from their own legacy setting, and a domain that never turned on Block AI Bots maps to Allow on all three controls.

We found no Cloudflare statement that free-tier sites were switched to blocking. The September 13 version of this article repeated the press-release scope; Cloudflare's later posts do not support it, so we removed it. The reliable answer for your site is in your own dashboard.

Training, search and agent crawlers: what blocking each one costs you

Cloudflare sorts AI traffic by what a bot does. Its AI bot definitions: Search "collects or indexes your content so it can answer questions about it later," Agent is "automated activity acting in real time on a person's behalf," and Training "crawls your content to train or fine-tune a model."

Blocking training crawlers

This is the lowest-risk block for AI search visibility, but not free of cost. OpenAI's crawler documentation says "Each setting is independent of the others," so disallowing GPTBot does not remove you from ChatGPT search. Anthropic's crawler page says restricting ClaudeBot means "the site's future materials should be excluded from our AI model training datasets."

Disallow AI Training also writes a Google-Extended disallow, which Google says covers Gemini training and grounding in Gemini Apps and Vertex AI (not Google Search). What nobody can tell you is whether training opt-outs change what future models say about your brand when they answer without searching: no vendor and no study has measured that, in either direction. Cloudflare reports that 17% of sites on its network "choose to enable some mechanism to block training."

Blocking search crawlers

This removes you from AI answers that cite the web. OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." That crawler is one of the few documented inputs to how ChatGPT decides which brands to recommend.

Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." Almost nobody does this on purpose: "less than 1% of Cloudflare sites choose to block Search bots."

Blocking agent crawlers

Agents fetch a page because a person asked about it right now. Block them and the assistant cannot open your page mid-conversation, so it answers from its index, its memory or someone else's page. Anthropic says disabling Claude-User "may reduce your site's visibility for user-directed web search."

Agents are also where robots.txt is weakest. Perplexity's bot documentation says its user fetcher "generally ignores robots.txt rules" (see how Perplexity cites sources), and OpenAI says of ChatGPT-User that "robots.txt rules may not apply." For Perplexity-User and ChatGPT-User, the Cloudflare setting is the control that reliably works; Claude-User, by contrast, follows robots.txt.

Per-crawler decision table

Cloudflare assigns categories itself. Where it has not named a crawler publicly, the category below follows the operator's stated purpose, so confirm it in your dashboard.

Crawler	What its operator says it does	Cloudflare control	If you block it
GPTBot (OpenAI)	Crawls content "that may be used in training"	Training	Out of future OpenAI training data. ChatGPT search unaffected.
OAI-SearchBot (OpenAI)	Surfaces websites in ChatGPT's search features	Search	Not shown in ChatGPT search answers, except as navigational links.
ChatGPT-User (OpenAI)	Visits a page when a user asks a question	Agent (Cloudflare's own example)	ChatGPT cannot open your page for that user. Only the edge block reliably stops it.
ClaudeBot (Anthropic)	Collects content that could contribute to training	Training	Excluded from future Anthropic training sets. Claude search unaffected.
Claude-SearchBot (Anthropic)	Improves search result quality	Search	Lower visibility and accuracy in Claude's search results.
Claude-User (Anthropic)	Fetches pages when a Claude user asks	Agent	Claude cannot retrieve your page for that question.
PerplexityBot	Builds Perplexity search results; "not used to crawl content for AI foundation models"	Search	Out of Perplexity's search results.
Perplexity-User	Fetches a page a user asked about; "generally ignores robots.txt rules"	Agent	Only an edge block stops it.
Googlebot	Crawls for Google Search, including AI Overviews and AI Mode	Mixed: Search and Training	Block removes you from Google Search entirely. Use Disallow AI Training instead.
Google-Extended	A robots.txt token, not a crawler: Gemini training, and grounding in Gemini Apps and Vertex AI	Written by Disallow AI Training	Google: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Bingbot (Microsoft)	Builds the Bing index behind Bing and Copilot	Mixed: Search and Training	Block removes you from Bing and Copilot. Disallow AI Training does not signal Bing yet; Microsoft's support is targeted for early 2027.
Applebot / Applebot-Extended	Applebot powers Spotlight, Siri and Safari search; Applebot-Extended is the training opt-out token	Mixed / written by Disallow AI Training	Blocking Applebot removes you from Apple's search features. Disallowing Applebot-Extended only opts out of training.
Meta-WebIndexer	Improves Meta AI search; allowing it "helps us cite and link to your content in Meta AI's responses"	Search (by stated purpose)	You lose the one crawler whose operator says allowing it helps citation.
Meta-ExternalAgent	Crawls "for use cases such as training foundation AI models"	Training	Out of Meta model training.
Amazonbot, CCBot, Bytespider	Amazonbot "may be used to train Amazon AI models"; CCBot feeds Common Crawl's open repository; ByteDance publishes no page we could open	Training (all on Cloudflare's managed training list)	Out of those datasets. None is documented as feeding an AI search product; Amazon runs a separate Amzn-SearchBot for search.

Table sources: Google's common crawlers page, Apple's Applebot page, Meta's crawler page, Amazon's Amazonbot page, Common Crawl's CCBot page, and the OpenAI, Anthropic, Perplexity and Cloudflare pages linked above.

Which setup fits your site

Four common positions and what each costs. "Disallow" means Disallow AI Training.

Your goal	Search	Training	Agent	What you give up
Be found and cited in AI answers (most SaaS, ecommerce, local and B2B sites)	Allow	Allow, or Disallow if you object to training	Allow	Nothing in ChatGPT, Claude or Perplexity search. Disallow also opts out of future training and, via Google-Extended, Gemini grounding; the effect on what models say is unmeasured.
Ad-funded publisher (Cloudflare's preset)	Allow	Disallow	Block on pages with ads	Assistants cannot open your ad pages for a user. You stay in search, Google included.
Keep classic search, refuse AI search and agents	Allow, then block OAI-SearchBot, Claude-SearchBot and PerplexityBot one by one in AI Crawl Control	Disallow	Block	Out of ChatGPT search, Claude and Perplexity answers. Google's AI features still use Googlebot.
Opt out of everything	Block	Block	Block	Since September 15, also out of Google, Bing and Apple search.

Note the third row: Cloudflare cannot separate Google's AI features from Google Search, because Googlebot feeds both. Google's guide to AI features names its own controls instead: nosnippet, data-nosnippet, max-snippet and noindex. Search Console also has a generative AI control that is on by default and inherits from parent properties.

How to check and change your Cloudflare AI crawler settings

In the dashboard

1. See the three controls. Go to Security, then Settings, filter by Bot traffic, and open Configure AI bot policies. Note what Search, Training and Agent say today.
2. Fix a Training block you did not mean. Block, or Block on pages with ads, now refuses Googlebot, Bingbot and Applebot too (on ad pages, for the second). To stay in Google, Bing and Apple search, switch Training to Disallow AI Training.
3. Decide on Bot Preference Sync. With it on, Cloudflare adds rules to the top of your robots.txt that match these controls. Its launch post says it "will not directly read from individual custom rules," so hand-written firewall rules are not reflected.
4. Review each crawler. In AI Crawl Control, the Crawlers tab has an Action column to allow or block one crawler at a time, and a Robots.txt violations column showing which crawlers ignored your file.
5. Read the status codes. The Metrics tab breaks requests down by crawler and status code. On the Free plan it covers only the past 24 hours, per Cloudflare's AI Crawl Control setup guide.
6. Look for rules that override your choice. An AI Crawl Control block is enforced as a WAF custom rule, and Cloudflare's crawler management docs note failed requests "may come from any rule or response error." Check your other security rules for anything matching bot user agents.

From outside, in two minutes

1. Read your live robots.txt at yourdomain.com/robots.txt. A section starting # BEGIN Cloudflare Bot Preference Sync or # BEGIN Cloudflare Managed content was written by Cloudflare, not by you.
2. Request your page as a crawler. This terminal command asks your site for its homepage while introducing itself as OpenAI's search crawler, with the user agent OpenAI publishes:

curl -I -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.example.com/

A 200 means the page was served; a 403, a 429 or a challenge page means it was refused. Swap in the user agents from Anthropic's and Perplexity's crawler pages to test theirs.

One limit: your request does not come from OpenAI's IP addresses, and Cloudflare verifies crawlers partly by "a published IP list with a stable user-agent," per its verified bots page. A refusal tells you a rule is keyed to that crawler's name, but a pass does not prove the real crawler gets in. The Metrics tab, which sees the real traffic, is the ground truth.

Why robots.txt cannot tell you whether AI can reach you

Your robots.txt states a preference; your Cloudflare settings and firewall rules enforce one. Cloudflare's documentation team showed the gap on its own site: in November 2025, as much as 80% of AI crawls were hitting unfinished preview sites. A robots.txt disallow cut that to 20%, and a security rule cut it to 0%.

The failure we see more often runs the other way: robots.txt allows a crawler and the edge refuses it. OpenAI's ChatGPT search help page states eligibility as two conditions: "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." Google's AI features guide says crawling must be allowed "in robots.txt, and by any CDN or hosting infrastructure."

Nothing in your analytics shows this failure. The page loads in your browser, Search Console looks normal, and the crawler gets a 403.

What we measured the day before the change

On September 14, 2026, we ran a crawler check on every brand website in the 99Visibility database, 43 sites, reading robots.txt per crawler and then requesting each homepage as each crawler and as a normal browser. A disagreement means robots.txt allowed the crawler, the crawler was refused, and the browser was not. The same kind of check runs inside our AI visibility audit.

Edge in front of the site	Sites a browser could reach	Sites refusing a crawler their robots.txt allowed
Cloudflare	19	8 (42%)
Vercel	6	0
Other or none detected	9	1

What the eight Cloudflare cases looked like:

• 1 site refused all seven of the AI search and agent crawlers named here: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and DuckAssistBot. Its robots.txt allowed all seven and a browser got in. If the real crawlers met the same rule, that site could not appear in ChatGPT search, and nothing on the site says so.
• 5 sites refused only Bytespider, ByteDance's crawler.
• 2 sites challenged our Googlebot and Bingbot requests. We count these as inconclusive: our request carried Google's crawler name but not Google's IP addresses, and refusing a fake Googlebot is sensible security.

Two of the 43 sites blocked AI crawlers in robots.txt, and both listed exactly the eight training crawlers on Cloudflare's managed robots.txt list. Neither blocked a crawler whose only documented job is search, though both disallowed Meta-ExternalAgent and Amazonbot, which our check counts as dual-use. One of them is the site above: its robots.txt welcomed AI search, and its edge refused it.

The limits, plainly: 43 sites is a small sample of our users' brands, not the web, and the requests came from one data-center connection with each crawler's user agent but not its IP addresses. This is also the before picture only. We have not published a re-run since the change, so this article makes no claim about what September 15 did to these sites.

Pay per crawl: charging instead of blocking

Cloudflare also lets a site charge AI crawlers per request. A crawler either signals that it will pay and gets the page, or receives 402 Payment Required with your price. As of October 2026, Cloudflare lists pay per crawl as a closed beta you apply for, and a WAF or Bot Management block overrides the charge.

Whatever you choose, check the result from outside afterwards. Crawler access sits at the top of our GEO checklist because every other fix assumes the crawler got in.

Frequently asked questions

Does Cloudflare block AI crawlers by default?

Not on every site. Since September 15, 2026, new domains that say they run ads start with Training set to Disallow AI Training and Agent set to Block on pages with ads, while new domains without ads start with everything allowed. Existing domains kept the effect of their old setting, so a site that never turned on Block AI Bots allows all three categories.

Will blocking AI crawlers on Cloudflare hurt my Google rankings?

Disallow AI Training does not: it uses Google-Extended, which Google says "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Setting Training or Search to Block, or to Block on pages with ads, is different. Since September 15 that stops Googlebot itself, taking the affected pages out of Google Search, AI Overviews and AI Mode together.

Is ChatGPT a web crawler?

ChatGPT is an assistant, but OpenAI runs three separate bots for it. GPTBot collects training data, OAI-SearchBot builds ChatGPT's search results, and ChatGPT-User opens pages when a user asks. Each can be allowed or blocked on its own, in robots.txt and on Cloudflare.

How do I stop AI from crawling my website?

Disallow the crawler in robots.txt, then enforce it at your CDN or firewall, because robots.txt is voluntary and some user-triggered fetchers say they may not follow it. On Cloudflare, AI Crawl Control blocks a named crawler with one setting. Decide per crawler: blocking a training bot and blocking a search bot have opposite effects on whether AI answers can cite you.

Can I see which AI crawlers Cloudflare is blocking on my site?

Yes. AI Crawl Control's Crawlers tab lists each AI crawler with its requests and robots.txt violations, and the Metrics tab breaks requests down by status code. On the Free plan the window is the last 24 hours.

To see the same picture from outside, sign up free and run the 99Visibility crawler access check: it reads your robots.txt rules for 18 AI and search crawlers, requests your page as those crawlers, names the edge that refused them, and flags every place your robots.txt and your firewall disagree.