PerplexityBot: Allow, Block and Verify the Right Requests
PerplexityBot is Perplexity's search crawler. Perplexity says it surfaces and links websites in search results, rather than collecting content for foundation-model training. Allowing it is an access decision; it does not guarantee that a particular page will be cited.
The practical question is not only whether your robots file says Allow. You also need to know who made the request and what your server returned. This guide separates those checks, using Perplexity's crawler documentation, checked on 15 November 2026.
PerplexityBot and Perplexity-User are different
Perplexity documents PerplexityBot for search discovery and Perplexity-User for user-requested page fetches. It says the latter generally ignores robots.txt because the fetch is user initiated. Neither is described as a foundation-model training crawler. Treat these as the operator's documented behaviour, not an independent audit of every request.
Before changing access, write down your goal: appearing in search, limiting a particular path, or refusing unwanted requests. A rule aimed at one activity should not silently become a decision about the other.
Choose a robots.txt rule deliberately
Under the Robots Exclusion Protocol, a matching agent group takes precedence over a wildcard fallback. A new named group can therefore change which shared restrictions apply. Review your entire file, not an isolated snippet.
For example, this illustrative policy allows public pages while excluding an account area:
User-agent: PerplexityBot
Allow: /
Disallow: /account/
To request that PerplexityBot avoid the whole site instead:
User-agent: PerplexityBot
Disallow: /
Choose the policy you intend; do not paste both examples. Robots rules are not authentication. Keep account data behind actual access controls. The RFC explicitly warns against treating the file as a security mechanism.
Verify the incoming request before granting an exception
Perplexity's WAF guidance combines the claimed agent with that agent's published IP ranges. PerplexityBot and Perplexity-User have separate lists. Use the current official endpoints rather than copying addresses from an old article.
Our recommended investigation record is deliberately small:
| Record | Why you need it |
|---|---|
| UTC timestamp, host and path | Connect the event to the rule and release in force |
| Claimed user agent | Identify what the request claims to be |
| Source address from your trusted edge | Compare with the appropriate vendor list |
| Edge status and returned content | Distinguish a page from a block or challenge |
| Verification result and list-check date | Keep confirmed and unconfirmed events separate |
Do not allowlist every request containing the word Perplexity. If an address cannot be verified, label the event unverified and investigate it. Avoid turning uncertainty into a blanket firewall bypass.
Diagnose a block in order
- Check the exact URL. Use the page customers need AI to read, not only the homepage.
- Evaluate its robots policy. Review the named group and any shared restrictions it replaces.
- Inspect the edge event. Find the rule responsible for a refusal or challenge before changing security settings.
- Inspect the body. A successful status carrying a challenge or empty shell is not evidence that the facts arrived.
- Retest and save both observations. Keep the configuration change and resulting response together.
A request you send with a copied crawler user agent tests your own network path. It does not become a real PerplexityBot request. Use that probe to locate a possible disagreement, then corroborate with verified incoming events. A crawler access check is a starting point for this investigation.
What if access works but Perplexity still does not cite us?
Record that outcome honestly: access works, citation not observed. Save the prompt, date, country, answer and cited URLs before making another change. For the broader discovery question, read how Perplexity finds and cites sources; this article owns the narrower access and verification task.
Method and limits: vendor documentation and protocol review, 15 November 2026. No customer logs were accessed, no firewall settings were changed, and no crawl-compliance or citation-rate experiment was run.
Can AI crawlers reach your site?
Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.
Start for Free2 of 3 retrieval crawlers can reach you
robots.txt allows 1 crawler your firewall blocks
Related Articles
Perplexity SEO: How Perplexity Finds and Cites Sources
Perplexity searches the web for its answers and cites sources by number. What its own docs say decides access, which crawler ignores robots.txt, and what to check.
AI Crawler List (October 2026): Every Major Bot, What It Does, and How to Control It
Every major AI crawler as of October 2026, from the vendors' own pages: what each bot does (training, search or a user-triggered fetch), whether it follows robots.txt, whether its IP ranges are published, and how to decide what to allow.

GPTBot vs OAI-SearchBot vs ChatGPT-User: Which OpenAI Crawler to Allow
GPTBot collects training data. OAI-SearchBot decides whether ChatGPT search can show your site. ChatGPT-User fetches pages live, and robots.txt may not apply to it. What OpenAI says each one does, a decision table by goal, and the firewall rule robots.txt cannot fix.


