WordPress robots.txt: Where It Is, How to Edit It, and the AI Crawler Rules Worth Adding

Your WordPress robots.txt exists even if nobody ever created it. This guide covers the robots.txt WordPress generates by default, where to edit it, the rules worth adding for AI crawlers as of October 2026, and the check most guides skip: whether your firewall agrees with the file.

Robots.txt is a request, not a lock. RFC 9309, the standard that defines it, says so in one line: "These rules are not a form of access authorization."

Where is robots.txt in WordPress?

Type your domain followed by /robots.txt into a browser. Google says the file "must be located at the root of the site host to which it applies", and a site can have only one.

What you see there comes from one of two places.

The virtual file WordPress writes for you

If there is no robots.txt file on your server, WordPress builds one on each request with its do_robots() function. On a public site installed at the root, the output looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The two rules keep crawlers out of the dashboard and leave the AJAX endpoint open. WordPress has added the Sitemap line since version 5.5, and only when the site is public.

A physical file overrides it

Yoast's documentation puts it plainly: "WordPress generates a virtual robots.txt file if the site root does not contain a physical file." Upload a real one and it wins.

	Virtual file	Physical file
What it is	Generated by WordPress on each request	A robots.txt file on your server
When it is served	Only when no physical file exists	Always, once it sits in the site root
Edit it with	Rank Math, or the robots_txt filter in code	Yoast's file editor, FTP or your host's file manager

How to edit robots.txt in WordPress

Pick one method. If a physical file exists, edits made to the virtual file never show up.

Yoast SEO

Go to Yoast SEO → Tools → File editor, then create or edit the file. Yoast notes the menu "will not appear if your WordPress install has disabled file editing."

Rank Math

Turn on Advanced Mode, then go to Rank Math SEO → General Settings → Edit robots.txt. Rank Math edits the virtual file, so its guide says you'll need to delete any physical robots.txt from your site root first.

FTP or your host's file manager

Create a plain-text file named robots.txt and upload it to the folder your domain serves from.

In code

Developers can append rules with the robots_txt filter, available since WordPress 3.0. It changes the virtual file only:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
    return $output;
}, 10, 2 );

What "Discourage search engines" does now

The checkbox under Settings → Reading → Search engine visibility no longer touches robots.txt. Since WordPress 5.3 it adds a meta tag to your pages instead: <meta name='robots' content='noindex,nofollow' />. The do_robots() changelog confirms the old "Disallow: /" line was removed in 5.3.

• Crawlers can still fetch every page. WordPress: "Neither of these options blocks access to your site — it is up to search engines to honor your request."
• The Sitemap line disappears, because WordPress adds it only to public sites.
• You drop out of AI Overviews too. To be a supporting link in AI Overviews or AI Mode, Google says "a page must be indexed and eligible to be shown in Google Search with a snippet". A noindexed page fails that.

If you built the site on a staging copy, check this box after launch.

AI crawler rules for your WordPress robots.txt

Each AI vendor runs separate bots for separate jobs, and each bot reads its own robots.txt group. This is what the vendors say, as of October 2026. For more vendors, see our AI crawler list.

User-agent	Vendor	Job	What the vendor says
OAI-SearchBot	OpenAI	Search	Opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links"
GPTBot	OpenAI	Training	Content "should not be used in training generative AI foundation models"
ChatGPT-User	OpenAI	User-triggered fetch	"robots.txt rules may not apply"
ClaudeBot	Anthropic	Training	Restricting it signals that future materials "should be excluded from our AI model training datasets"
Claude-SearchBot	Anthropic	Search	Disabling it "may reduce your site's visibility and accuracy in user search results"
Claude-User	Anthropic	User-triggered fetch	Disabling it stops Claude retrieving your content when a user asks
PerplexityBot	Perplexity	Search	"not used to crawl content for AI foundation models"
Google-Extended	Google	A token, not a crawler: Gemini training and grounding	"does not impact a site's inclusion in Google Search"

Sources: OpenAI, Anthropic, Perplexity, Google.

Two rows need a closer read.

• Blocking a training bot does not remove you from that vendor's search. OpenAI: "Each setting is independent of the others." You can allow OAI-SearchBot and disallow GPTBot. The full comparison is in GPTBot vs OAI-SearchBot.
• Google-Extended is not training-only. Google says it also covers grounding in Gemini Apps and on Vertex AI. It does not touch AI Overviews either: Google says "robots.txt directives for Googlebot is the control." See how to turn off AI Overviews for the controls Google does offer.

Example: opt out of training, stay in search

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://example.com/wp-sitemap.xml

OAI-SearchBot, Claude-SearchBot and PerplexityBot have no group of their own here, so they follow the * group. Whether to block training is a business decision. OpenAI claims no search benefit for allowing GPTBot, and nobody has measured what blocking it does to what future models know about you.

Watch out: a named group replaces the * group

RFC 9309 says crawlers use the * group only "if no matching group exists." Google puts it more bluntly: "Only one group is valid for a particular crawler." Add a group for OAI-SearchBot and it stops reading your * rules, so copy across any rule you still want.

How to test your WordPress robots.txt

1. Open it in a private window. Confirm you see the version you edited, not a physical file you forgot about.
2. Check the status code is 200. Under RFC 9309, a 4xx means a crawler "MAY access any resources," and a 5xx means it "MUST assume complete disallow." So a robots.txt that throws a server error locks out compliant crawlers.
3. Use Search Console's robots.txt report. It shows the robots.txt files Google found for your top 20 hosts, when it last crawled them, and any errors. It works on domain-level properties only.
4. Wait a day before judging. OpenAI says search changes take about 24 hours, Perplexity says up to 24 hours, and Google generally caches the file for up to 24 hours.

The trap: robots.txt says allow, your firewall says 403

Robots.txt is the first gate. Whatever sits in front of WordPress is the second: your CDN, your host's firewall, a security plugin. Any of them can answer a crawler with a 403 or a challenge page while robots.txt says allow.

The vendors say both gates count. OpenAI recommends "allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges." Google's AI features guide lists "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure."

Nothing in WordPress shows you this. The page loads in your browser and the crawler gets a 403. If you're on Cloudflare, read Cloudflare's AI crawler default change.

To check by hand, request a page with a crawler's user-agent:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://example.com/

A 200 is a pass. A 403 or a challenge page is a block. One limit: your test does not come from Perplexity's published IP ranges, so a firewall that checks IPs can treat it differently from the real bot. Your server logs show what the real bots got.

99Visibility runs this check for you. It requests your pages as each AI crawler, compares the answer with what your robots.txt says, and flags every place the two disagree. It also audits what five AI platforms say about your brand. Your first audit is free, with no credit card, and covers ChatGPT and Google AI Overviews. See the plans for all five platforms. Inside the dashboard, Agent V reads your audit and your site and drafts the robots.txt lines for you to paste. It never edits your site itself.

FAQ

Where is the robots.txt file in WordPress?

At yourdomain.com/robots.txt. If your site root has no physical file, WordPress generates a virtual one, so you won't find it over FTP. Upload a real file to override it.

How do I fix "Blocked by robots.txt" in WordPress?

Find the Disallow rule that matches the URL and remove it, or add a longer Allow rule for that path: Google uses "the most specific rule based on the length of the rule path." Then request a recrawl in Search Console's robots.txt report.

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. GPTBot is OpenAI's training crawler. OAI-SearchBot is the one that decides whether you appear in ChatGPT search answers.

See which AI crawlers can reach your WordPress site. Run a free audit: no credit card needed.