When robots.txt Says Yes and Your Firewall Says No

Your robots.txt allows OAI-SearchBot. You checked it in the last lesson. ChatGPT search still never cites you. There is a second gate, and almost nobody checks it.

Between every crawler and your pages sits something that decides who gets in: a CDN, a web application firewall, a bot-management product, or a host with rules of its own. Those systems decide whether a request receives your page, a 403, a rate limit or a challenge page. They do not read robots.txt. A crawler your robots.txt invites can be turned away at the door, and nothing on any dashboard tells you.

This lesson covers what that failure looks like, why it happens without anyone touching a setting, and how to test for it without fooling yourself.

Eligibility has two halves, joined by "and"

OpenAI states the requirement for ChatGPT search in one sentence.

Read the conjunction. It says and, not or. Robots.txt permission is the first half. Network reachability is the second. A site that passes only the first is not eligible.

The same logic holds for every crawler, whether or not its vendor spells it out. PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Googlebot and bingbot all have to receive your HTML before anything else can happen. A crawler that gets a block page has nothing to index. If the block also covers /robots.txt, the crawler may not even learn what you allowed. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file."

How often does robots.txt allow a bot that the network then refuses? Nobody has published that number. What is documented is the mechanism, by the operators of the crawlers themselves. That is reason enough to spend ten minutes testing.

What a block looks like from the crawler's side

A crawler never sees your firewall dashboard. It sees an HTTP response. These are the ones that lock it out.

• 403 Forbidden. The classic bot-management block. OpenAI's guidance warns that these systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."
• 429 Too Many Requests. A rate limit. A crawler that keeps hitting it gets less of your site, or none of it.
• A JavaScript challenge or CAPTCHA. A page comes back, sometimes with a 200 status, but it is an interstitial asking the visitor to prove it is human. Crawlers are not. OpenAI: "Since OpenAI crawlers are automated systems, these checks may block access."
• 401 or 503 from the edge. Some setups use these for the same purpose. Treat them as blocks when a normal browser gets through.

The challenge is the sneaky one. Your uptime monitor records a 200. You open the site in a browser, the challenge clears itself in a second, and you see your homepage. The crawler sees a page that says nothing about you.

Why it happens when nobody changed anything

Most of these blocks are not decisions. They come from defaults, managed rule sets and bot scores that someone else maintains.

Cloudflare's AI crawler defaults

Cloudflare’s September 15 announcement says existing settings carry over. New ad-supported domains are offered Search: Allow, Training: Disallow AI Training, and Agent: Block on pages with ads. Training Block also blocks mixed-use crawlers such as Googlebot and Bingbot; Disallow AI Training preserves their search access. Check the settings of your own zone.

Cloudflare, not you, classifies each crawler. Review Search, Training and Agent separately, then confirm verified-crawler responses in your own logs. A preset or migration description is not evidence that a particular site lost access.

Bot management and managed rules

Bot-management products score every request. A crawler that fetches quickly, from datacenter IPs, without running JavaScript or keeping cookies, looks a lot like the scrapers those products exist to stop. Managed rule sets are updated by the vendor on its own schedule, so a rule that let OAI-SearchBot through in spring may not in autumn. Custom rules age badly too: a rule someone wrote during a scraping incident can still be blocking legitimate crawlers a year later.

IPv6-only origins

OpenAI publishes its crawler IP ranges as JSON files: searchbot.json, gptbot.json, chatgpt-user.json and adsbot.json. When our research checked them in August 2026, every one listed IPv4 ranges only. If your origin, or the edge path crawlers are routed to, answers only over IPv6, those crawlers cannot reach it, whatever robots.txt says. Rare, and total when it happens.

Let crawlers in without opening the door to everyone

The fix is not to switch off bot protection. A user-agent is a string anyone can type, and scrapers borrow crawler names all the time. The goal is to admit the real crawler and keep refusing impostors.

Start with verified-bot programs

Large CDNs keep lists of verified bots that you can allow as a class. Cloudflare verifies a bot through "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS," plus robots.txt compliance. A verified-bot rule checks identity, not only the name a request gives itself.

Use published IP ranges to verify, not as your only rule

OpenAI publishes its ranges and, in the same breath, warns against IP-only allowlisting. Read the two together: match the user-agent and the source range, and load the ranges from the vendor's live file rather than a copy someone pasted last year. Perplexity, Google and Microsoft publish IP lists for their crawlers too.

Never block /robots.txt

Whatever else you block, let crawlers fetch the file. A bot that cannot read it cannot learn what you allowed, which is Anthropic's warning in practice.

Make the firewall agree with robots.txt

Blocking training crawlers at the edge is a legitimate choice. Blocking search crawlers by accident is not. Put your robots.txt table next to your CDN settings and make every row match.

Test it yourself, and read the result honestly

You can see what your edge does to a crawler's user-agent with curl, which ships with macOS, Linux and current Windows. These three requests ask for your homepage as OAI-SearchBot, as PerplexityBot, and as an ordinary Chrome browser, and print only the status code. The crawler user-agent strings are the ones the vendors publish.

curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://www.example.com/

curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://www.example.com/

curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" https://www.example.com/

The third request is your control. To catch a challenge served with a 200, save a body and search it:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://www.example.com/ -o probe.html
grep -i -o -E "captcha|cf-chl|attention required|verify you are human" probe.html

Now the caveat that decides what those results mean. A request you send with a crawler's user-agent is not the crawler. It comes from your IP, not from OpenAI's or Perplexity's networks. A firewall that verifies bots by IP sees a request that claims to be OAI-SearchBot from the wrong network, and may block it as a fake, which is exactly its job. So read the results as indicative:

• Crawler user-agent and control both get a 200 with your real page. A good sign, not proof. An IP-based rule could still stop the real bot.
• Crawler user-agent blocked, control gets a 200. A strong lead. Either the edge blocks that user-agent outright, or it blocks unverified traffic claiming to be that bot. Your firewall's event log says which rule fired.
• Both blocked. Your edge is blocking your own IP, or the site is down. Fix that before you read anything else.

The ground truth is in your logs. Filter your server or CDN logs for user-agents containing OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Googlebot and bingbot, and look at the status codes those requests received. Check that the source IPs fall inside each vendor's published ranges, so you are reading real crawlers and not impostors. A run of 403s to verified OAI-SearchBot traffic is your answer, whatever curl said.

One rule for every test: send a handful of requests, spaced out, never a burst. Dozens of rapid requests with a spoofed crawler user-agent teach the bot manager to challenge your IP, and then every later test, yours or a colleague's, is polluted.

The finding the Crawler Access check reports

The 99Visibility Crawler Access check runs both halves of the gate together. It evaluates your robots.txt for each agent, then requests your page using each crawler's published user-agent, plus a normal browser request as a control. Each response is classified as a pass, a blocking status such as 403 or 429, or a challenge page, including a challenge served with a 200.

The headline finding is the disagreement: robots.txt allows the agent, or has no rule for it, while the edge blocks the request and the browser control gets through. The control is what separates "your firewall blocks this bot" from "your site was down." When the disagreement hits a retrieval bot, it is a silent visibility failure, and it is the one no other dashboard shows you.

The same honesty applies to the check as to your own curl test. It sends its requests from our servers, not from the crawler vendors' networks. Treat a disagreement as a strong lead to confirm in your firewall log, and a pass as "this user-agent is not refused," not as proof that the real crawler gets in.

Common mistakes

• Testing in a browser and calling it done. Your browser runs JavaScript, passes challenges and does not send a crawler user-agent. It proves nothing about bots.
• Trusting the status code alone. A challenge page can come back with a 200. Check the body.
• Switching off bot protection to fix it. Spoofed crawler user-agents are common. Allow verified bots instead.
• Allowlisting a copied IP list. OpenAI warns against IP-only allowlisting, and ranges change. Match user-agent and range, from the vendor's live file.
• Bursting test requests. Dozens of spoofed requests get your own IP challenged and spoil every later test.
• Blocking /robots.txt along with everything else. A crawler that cannot read the file cannot learn what you allowed.
• Assuming nothing changed. CDN defaults, managed rules and plan tiers change under you. Re-test after any CDN change and after a vendor announces new defaults.