GPTBot vs OAI-SearchBot vs ChatGPT-User: Which OpenAI Crawler to Allow
GPTBot vs OAI-SearchBot is a choice between training and search. GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot decides whether your site can be shown in ChatGPT search answers. A third agent, ChatGPT-User, fetches pages when a person asks ChatGPT something, and OpenAI says robots.txt may not apply to it.
Blocking "OpenAI" as if it were one bot is the mistake. Below is what OpenAI's crawler documentation says each agent does, how the settings interact, the IP lists your firewall needs, and the robots.txt for each goal. Everything here is as of October 2026.
What each OpenAI crawler does
OpenAI's page lists four agents. Three matter for an ordinary website. The fourth only visits ad landing pages.
| Agent | Purpose, per OpenAI | Does robots.txt control it? |
|---|---|---|
| GPTBot | "Used to crawl content that may be used in training our generative AI foundation models" | Yes |
| OAI-SearchBot | "Used to surface websites in search results in ChatGPT's search features" | Yes |
| ChatGPT-User | Visits a page when a user asks ChatGPT or a Custom GPT a question | "Robots.txt rules may not apply" |
| OAI-AdsBot | Checks pages submitted as ChatGPT ads; its data "is not used to train generative AI foundation models" | Not stated; it "only visits pages submitted as ads" |
GPTBot: training only
OpenAI's description: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." That is everything GPTBot controls. OpenAI's page attaches no search benefit to it.
Allowing GPTBot does not get you cited in ChatGPT. Blocking it does not remove you from ChatGPT search. OpenAI's documentation makes neither claim, and the second is contradicted by the independence rule below.
OAI-SearchBot: ChatGPT search
This is the agent that governs search inclusion: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
Allowing it buys eligibility, not placement. OpenAI's help centre says "ChatGPT ranks search results using multiple factors intended to help users find relevant, reliable information. Placement is not guaranteed." Changes are not instant either: "it can take ~24 hours from a site's robots.txt update for our systems to adjust."
ChatGPT-User: live fetches
When someone asks ChatGPT a question, it "may visit a web page with a ChatGPT-User agent." OpenAI is direct about what that means for your robots.txt: "Because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to determine whether content may appear in Search."
So a Disallow line for ChatGPT-User is a request, not a guarantee, and it is the wrong tool for a search opt-out. OpenAI points you to OAI-SearchBot for that. Anthropic handles its equivalent differently and documents a robots.txt control for Claude-User; see ClaudeBot and its two siblings.
The settings are independent
This is the sentence to remember from OpenAI's page: "Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models."
One detail: if you allow both, OpenAI "may use the results from just one crawl for both use cases to avoid duplicative crawling." Seeing only one of the two in your logs does not mean the other is being ignored. Your robots.txt is what records your choice.
Which OpenAI crawler to allow: a decision table
| Your goal | GPTBot | OAI-SearchBot | Firewall or CDN |
|---|---|---|---|
| Stay out of training, appear in ChatGPT search | Disallow | Allow | Allow the searchbot.json ranges |
| Appear in ChatGPT search, no view on training | Your call: no documented search effect | Allow | Allow searchbot.json, plus gptbot.json if GPTBot is allowed |
| Stay out of both | Disallow | Disallow | Nothing to allow. You can still appear as a navigational link |
None of these rows controls ChatGPT-User. If you want to stop live fetches, robots.txt may not do it. The chatgpt-user.json ranges are what a firewall rule would match, and blocking them stops ChatGPT opening your page from those addresses when a user asks about you.
robots.txt examples
Out of training, in ChatGPT search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Out of training and search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /Name each bot in its own group. A User-agent: * group with Disallow: /, or a copied "block every AI bot" list, can switch off OAI-SearchBot when all you meant to stop was training. For the other vendors' bots, see our AI crawler list. On WordPress, start with our guide to WordPress robots.txt.
The firewall trap: robots.txt is half the requirement
OpenAI's help centre gives the full condition for ChatGPT search: "To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses."
The word that matters is and. A robots.txt allow behind a firewall that returns 403 to OAI-SearchBot keeps you out of ChatGPT search, and your page analytics will not show it. A CDN default can do this without you changing a setting; see Cloudflare's AI crawler default.
OpenAI's published IP lists
| Agent | IP list | IPv4 prefixes, October 2026 |
|---|---|---|
| OAI-SearchBot | openai.com/searchbot.json | 39 |
| GPTBot | openai.com/gptbot.json | 18 |
| ChatGPT-User | openai.com/chatgpt-user.json | 234 |
| OAI-AdsBot | openai.com/adsbot.json | 2 |
The lists are not interchangeable. When we compared them in October 2026, an allowlist built from gptbot.json covered 6 of the 39 OAI-SearchBot prefixes. Allowlist each bot from its own file, and re-fetch the files on a schedule: each carries a creation date, and they change.
Verify the IP, not the header
Anyone can send "GPTBot" as a user agent. OpenAI's example string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and OpenAI notes the version number "may change". Match on the name, then check the source IP against that bot's list. When fetching robots.txt, OpenAI may add a robots.txt marker to the string, which helps if your logs omit paths.
Check your site in five steps
- Open your live robots.txt and find the GPTBot and OAI-SearchBot groups, plus any wildcard group that catches them.
- Request a page as OAI-SearchBot and check for a 200. From a terminal:
curl -s -o /dev/null -w "%{http_code}" -A "OAI-SearchBot" https://www.example.com/ - Request /robots.txt the same way. A crawler that cannot read the file cannot read your permissions.
- Check your logs for requests from searchbot.json addresses and the status codes you returned. A test from your own machine cannot exercise IP-based rules.
- Check your CDN or firewall for any rule that blocks or challenges AI crawlers, whatever your robots.txt says.
99Visibility does this for you. It checks whether AI crawlers can reach your site and compares what your robots.txt says with what your firewall actually returns, alongside an audit of what five AI platforms say about your brand. Your first audit is free, with no credit card, and covers ChatGPT and Google AI Overviews. See the plans for all five platforms. Inside the dashboard, Agent V reads your audit and your site and drafts the robots.txt lines for you to paste. It never edits your site itself.
FAQ
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's crawler for ChatGPT search. If you opt out of it, OpenAI says your site will not be shown in ChatGPT search answers, though it can still appear as a navigational link.
Should I block GPTBot?
It is a training decision with no documented effect on ChatGPT search. If you do not want your content used to train OpenAI's models, disallow GPTBot and keep OAI-SearchBot allowed.
Does ChatGPT-User respect robots.txt?
Not reliably. OpenAI says that because ChatGPT-User acts on a user's request, "robots.txt rules may not apply."
See whether ChatGPT's crawlers can reach you. Run a free audit: no credit card.
Want more of our research in Google Search and AI Mode? Add us as a preferred source.
Can AI crawlers reach your site?
Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.
Start Free Audit2 of 3 retrieval crawlers can reach you
robots.txt allows 1 crawler your firewall blocks
Related Articles
ClaudeBot: What It Is, What It Collects, and How to Allow or Block It
ClaudeBot is Anthropic's training crawler. Its two siblings, Claude-SearchBot and Claude-User, handle search and live fetches. What each one does, the robots.txt to allow or block each, and how to verify real Anthropic traffic by IP.
AI Crawler List (October 2026): Every Major Bot, What It Does, and How to Control It
Every major AI crawler as of October 2026, from the vendors' own pages: what each bot does (training, search or a user-triggered fetch), whether it follows robots.txt, whether its IP ranges are published, and how to decide what to allow.
Should I Block AI Crawlers on Cloudflare? A Per-Bot Decision Guide
Block training crawlers if you object to training; keep search and agent crawlers open to stay in AI answers. What Cloudflare changed on September 15, 2026, per crawler, and how to check yours.