Robots.txt Tester: How to Check What Google and AI Crawlers Can Actually Reach
A robots.txt tester answers one question: does this file allow this crawler to fetch this URL? That is half the answer. The other half is whether your firewall or CDN lets the crawler in, and no robots.txt checker can see that.
This guide covers both tests: how to check robots.txt by hand and with Google's tools as of October 2026, then how to request a page as the crawler and read what comes back.
Where to find a site's robots.txt
Add /robots.txt to the root of the host: https://example.com/robots.txt. Google: "A robots.txt file applies only to paths within the protocol, host, and port where it is posted." The file on example.com does not cover shop.example.com, so test each host.
Open the file in a private window. Then note its status code, because that decides what crawlers do:
| robots.txt returns | RFC 9309 | |
|---|---|---|
| 200 | Crawler "MUST follow the parseable rules" | Uses the file as served |
| 4xx | Crawler "MAY access any resources on the server" | Every 4xx except 429 is treated "as if a valid robots.txt file didn't exist" |
| 5xx or network error | Crawler "MUST assume complete disallow" | Stops crawling for 12 hours, then uses the last good copy for the next 30 days |
Under RFC 9309, a 404 opens the whole site and a 500 closes it. Fixes are not instant either: Google "generally caches the contents of robots.txt file for up to 24 hours".
How crawlers read the rules
A crawler uses the groups naming its product token, matched case-insensitively and merged. Only "if no matching group exists" does it fall back to the * group. Google spells out the consequence: "User agent specific groups and global groups (*) are not combined."
Inside the group, the longest matching path wins, not the first line. RFC 9309: "The most specific match is the match that has the most octets." On a tie, Allow wins. If nothing matches, the URL is allowed. For which token belongs to which product, see our AI crawler list.
A robots.txt test by hand: one file, four checks
User-agent: *
Disallow: /private/
Allow: /private/press/
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /blog/| Crawler | URL path | Group | Deciding rule | Result |
|---|---|---|---|---|
| Googlebot | /private/press/kit.pdf | * | Allow: /private/press/ (15 characters beats 9) | Allowed |
| Googlebot | /private/notes | * | Disallow: /private/ | Blocked |
| GPTBot | /blog/guide | GPTBot | Disallow: / | Blocked |
| OAI-SearchBot | /private/notes | OAI-SearchBot | None matches | Allowed |
The last row is the trap. The owner closed /private/ to everyone, then gave OAI-SearchBot its own group to open the blog, and that group replaced the * rules. A rule a named crawler must follow has to sit inside its own group. On WordPress, see our guide to editing your WordPress robots.txt.
Google's robots.txt testing tools in October 2026
The old robots.txt Tester is gone. On 15 November 2023, Google Search Central announced the Search Console robots.txt report and wrote: "As part of this update, we're sunsetting the robots.txt tester." Three tools replace it:
- The robots.txt report (Search Console, Settings). It "shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered", and lets you request a recrawl. Domain-level properties only.
- URL Inspection. Google's tool for one URL: "you can test the availability of the URL with the URL inspection tool." Read the "Crawl allowed?" field.
- Google's open-source robots.txt library: Googlebot's production code, slightly modified, with a program for "testing a single URL and user-agent against a robots.txt". A developer has to build it.
The report and URL Inspection show Google's view only. For AI crawlers, use the hand method above.
The test robots.txt can't do: request the page as the crawler
RFC 9309 is blunt: "These rules are not a form of access authorization." Your firewall, CDN and bot protection decide who gets in.
OpenAI names both gates. For ChatGPT search, "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." Protection services, OpenAI warns, "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." Defaults change too: see Cloudflare's AI crawler default change.
How to run it
- Request the page as a browser. This is your control.
- Request it again with the crawler's user-agent string, copied from the vendor. OpenAI publishes OAI-SearchBot's, noting "the version number may change".
- Request /robots.txt the same way. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file".
- Compare the codes, then open both saved pages.
curl -s -o browser.html -w "%{http_code}" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" https://example.com/pricing
curl -s -o crawler.html -w "%{http_code}" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/pricingNo terminal? In Chrome DevTools' Network conditions panel, untick "Use browser default", paste the crawler's string and reload.
Read the status code, not a pass/fail label
| The crawler gets | What it means |
|---|---|
| 200 and the real page | Passed |
| 200 and a "verify you are human" page | A challenge. OpenAI: such checks "may block access even if the crawler successfully passes earlier layers" |
| 401 | The request "lacks valid authentication credentials" |
| 403 | The server "understood the request but refuses to fulfill it" |
| 429 | The client "has sent too many requests in a given amount of time" |
| 503 | "A temporary overload or scheduled maintenance". If the control got a 200, it was aimed at the crawler |
| Another 4xx, such as 418 | Still a refusal. Cloudflare lets a site pick its block code "in the range 400-499" |
Any code the crawler gets and the browser does not is a finding. A checker that counts only a fixed list of codes as blocks will call a 418 reachable. Read the number yourself.
Check the control first
If the browser request fails too, the test measured nothing. The site is down or blocking your network, and every crawler result is noise.
What a test request can and can't prove
Your request carries the crawler's name but your IP address. OpenAI publishes its crawler IP ranges, and Google documents how to verify requests from troublemakers "claiming to be from Google". A firewall that checks addresses can treat your test differently from the real crawler, in either direction.
So a block means a rule matched your request: find it in your CDN or firewall and check what it keys on. A pass does not clear an IP-based rule. Send a few requests, not a burst, or bot protection may block your own IP.
Robots.txt test checklist
| Check | How to run it | What a failure looks like |
|---|---|---|
| File on each host | Open /robots.txt per subdomain and protocol | A 5xx, or a host with no file |
| Google can read it | Search Console robots.txt report | Not Fetched, or errors under Issues |
| Right group | Find the crawler's group, else * | A named group drops the * rules |
| Key URLs | Longest match by hand; URL Inspection for Googlebot | A Disallow wins on a page you need |
| Crawler reaches robots.txt and pages | Request both with the crawler's user agent, beside a browser control | 401, 403, 418, 429, 503, a challenge, or any code the control did not get |
| Retest after changes | Wait a day; request a recrawl | Old rules still applied |
Run both tests on the pages that matter
99Visibility checks whether AI crawlers can reach your site, comparing what your robots.txt allows with what each crawler actually gets back, and audits what 5 AI platforms say about your brand. No automated check recognises every way a site can refuse a crawler, so when a result surprises you, request the page yourself and read the code. Your first audit is free, with no credit card, and covers ChatGPT and Google AI Overviews. See the plans for all five platforms. Inside the dashboard, Agent V reads your audit and your site and drafts the robots.txt lines for you to paste. It never edits your site itself.
FAQ
How do I check the robots.txt file on a website?
Add /robots.txt to the protocol and host, then open it: https://example.com/robots.txt. Each subdomain and protocol has its own file.
Does Google still have a robots.txt tester?
Not the old one, retired in November 2023. Use the Search Console robots.txt report, URL Inspection, and Google's open-source robots.txt library.
If robots.txt allows a crawler, can it reach my site?
Not necessarily. Your CDN or firewall can still answer with a 403, a challenge or another refusal code. Request the page as the crawler and compare it with a browser request.
See what crawlers actually get from your site. Run a free audit: no credit card.
Want more of our research in Google Search and AI Mode? Add us as a preferred source.
Can AI crawlers reach your site?
Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.
Start Free Audit2 of 3 retrieval crawlers can reach you
robots.txt allows 1 crawler your firewall blocks
Related Articles
AI Crawler List (October 2026): Every Major Bot, What It Does, and How to Control It
Every major AI crawler as of October 2026, from the vendors' own pages: what each bot does (training, search or a user-triggered fetch), whether it follows robots.txt, whether its IP ranges are published, and how to decide what to allow.
WordPress robots.txt: Where It Is, How to Edit It, and the AI Crawler Rules Worth Adding
Where WordPress keeps its robots.txt, how to edit it with Yoast, Rank Math, FTP or code, what "Discourage search engines" does now, which AI crawler rules to add, and why your firewall can overrule the file.
Should I Block AI Crawlers on Cloudflare? A Per-Bot Decision Guide
Block training crawlers if you object to training; keep search and agent crawlers open to stay in AI answers. What Cloudflare changed on September 15, 2026, per crawler, and how to check yours.