Search Passes. Agents and Training Do Not: Cloudflare Changes the Default on September 15

On 2026-09-15, Cloudflare changes what happens to AI crawlers by default. Search crawlers stay allowed. Training and Agent crawlers get blocked on pages that display ads — for all new domains, for new sites added to existing accounts, and for all existing free-tier customers.

Nobody emails you about this. Your traffic looks the same, your rankings look the same, and the page loads perfectly in your browser — because your browser is not GPTBot.

What actually changes on September 15

Cloudflare splits bot traffic into three categories and treats them differently. This is the part most coverage skipped, and it is the part that decides whether you are affected.

Category	What it does	Default after 2026-09-15
Search	Crawls to index your pages so they can be returned in answers later	Allowed
Agent	Fetches your page live, on behalf of a user asking right now	Blocked on ad-displaying pages
Training	Collects content to train or fine-tune models	Blocked on ad-displaying pages

Cloudflare's own wording: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."

Two details matter more than the headline. First, Cloudflare decides which bot sits in which category — not you, and not the AI vendor. Second, crawlers that do more than one job at once get treated as the stricter category. A bot that indexes for search and collects training data is blocked on ad pages.

Who this hits without touching anything

1. New domains added to Cloudflare after the 15th.
2. New sites added to an account that already exists.
3. Every existing free-tier customer. This is the big one. A large share of small-business sites sit on Cloudflare's free tier.

If you are on a paid plan with settings you configured yourself, your existing configuration is what governs. Check it anyway — the point of this article is that nobody can see this from outside.

The failure nobody is looking for: robots.txt says allow, the CDN says 403

Here is the mechanism that makes this worse than a settings change.

Getting crawled is two gates, not one. OpenAI states the requirement plainly: "To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses."

Read the conjunction. It is and. Your robots.txt is the first gate. Your CDN is the second. A permissive robots.txt behind a blocking WAF is a closed door with a welcome mat in front of it.

This is the single most under-measured thing in AI visibility right now. Every GEO tool on the market will read your robots.txt, because reading robots.txt is easy. Almost none of them request your page as the crawler and compare the answer. The gap between what robots.txt promises and what your edge actually returns is invisible unless somebody tests both.

And the failure is silent by construction. A 403 to OAI-SearchBot produces no error in your analytics, no warning in Search Console, and no change to what you see when you visit your own site.

Why robots.txt is necessary but not sufficient

Even without a WAF in the way, robots.txt does less than most people are told. Cloudflare — the party with the actual network view — says effectiveness is unresolved: "others might be unsure whether these bots even respect robots.txt rules, especially newer or less transparent crawlers."

Perplexity documents that Perplexity-User generally ignores robots.txt rules. OpenAI says robots.txt may not apply to ChatGPT-User. Treat robots.txt as the stated gate, and the reachability test as the real one.

Check your site before Monday: 5 steps

All five are things you can do today without a developer.

1. Open Cloudflare → AI Crawl Control. Note which category each AI bot sits in, and what its setting will be after the 15th. Do not assume — the categories are assigned by Cloudflare and they change.
2. Decide deliberately, per category. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) is a legitimate business decision. Blocking retrieval bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) removes you from those answers. These are different choices and they deserve different answers.
3. Request your own page as each retrieval bot and confirm you get a 200. If you get a 403 or a challenge page, your robots.txt is fiction.
4. Check that /robots.txt itself is reachable to those agents. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file" — block the crawler at the network layer and it cannot even read your permissions.
5. Repeat on every distinct surface. A rule scoped to a path, a subdomain or a language directory can pass one page and block another.

The one-line version

Your robots.txt states a policy. Your CDN enforces a different one. Only one of them is real, and it is not the file.

What to do if you are blocking on purpose

Blocking is a defensible position, and this article is not an argument against it. Publishers are using exactly this leverage to negotiate licensing deals, which is the stated reason the default is changing.

What is not defensible is blocking by accident, in a category you did not know existed, because a default changed on a date you did not have in your calendar. The distinction to hold onto:

• Training — blocking costs you nothing in today's answers. Your content is not used to train the next model. That is the trade.
• Agent — blocking means that when a user asks an assistant about you right now, it cannot fetch your page to answer. It answers from memory, or from a competitor who let it in.
• Search — still allowed by default. This is the one you almost certainly want open.

Why this date is worth more than the settings change

There is a second reason to care. This is a dated, announced, third-party change to a defined population — which makes it a clean natural experiment, but only for whoever measured the population before the date.

After Monday you can still fix your configuration. What you cannot do is recover what it looked like beforehand. If you want a record of your own pre-change state, the window closes on the 15th.

Frequently Asked Questions

Does this block Google from indexing my site?

No. Traditional search indexing is unaffected. Cloudflare's Search category remains allowed by default, and Googlebot is not part of this change. What changes is AI training and live agent fetching on ad-displaying pages.

I am not on Cloudflare. Am I safe?

You are not affected by this specific default change. You can still be blocking AI crawlers at your own WAF, host or bot-management layer without knowing it. The robots-versus-reachability test is worth running regardless of who fronts your site.

My robots.txt allows all AI crawlers. Is that enough?

No. OpenAI's documented requirement is robots.txt permission and network-level reachability. If your CDN returns a 403 to the crawler's IP range, the robots.txt allow does nothing. Both gates have to be open.

How do I know if my pages count as "displaying ads"?

Cloudflare applies the ad condition. If you run display advertising of any kind on a page, assume it is in scope, and verify in AI Crawl Control rather than reasoning about it from outside.

We built the access check in 99Visibility for exactly this: request your pages as each AI crawler, compare the result against what your robots.txt claims, and report every place the two disagree. Not a guess about what AI "prefers" — a yes or no on whether it can reach you at all.

Run a free audit and get your pre-change baseline before the 15th. For the evidence behind every claim here, see our research page, or read how LLMs prioritise information about brands.