Back to Blog
technical#PerplexityBot#Perplexity#AI Crawlers#robots.txt

PerplexityBot: Allow, Block and Verify the Right Requests

By Liwaa BeainiNovember 15, 20263 min read
99Visibility article cover for “PerplexityBot: Allow, Block and Verify the Right Requests” — 99Visibility

PerplexityBot is Perplexity's search crawler. Perplexity says it surfaces and links websites in search results, rather than collecting content for foundation-model training. Allowing it is an access decision; it does not guarantee that a particular page will be cited.

The practical question is not only whether your robots file says Allow. You also need to know who made the request and what your server returned. This guide separates those checks, using Perplexity's crawler documentation, checked on 15 November 2026.

PerplexityBot and Perplexity-User are different

Perplexity documents PerplexityBot for search discovery and Perplexity-User for user-requested page fetches. It says the latter generally ignores robots.txt because the fetch is user initiated. Neither is described as a foundation-model training crawler. Treat these as the operator's documented behaviour, not an independent audit of every request.

Before changing access, write down your goal: appearing in search, limiting a particular path, or refusing unwanted requests. A rule aimed at one activity should not silently become a decision about the other.

Choose a robots.txt rule deliberately

Under the Robots Exclusion Protocol, a matching agent group takes precedence over a wildcard fallback. A new named group can therefore change which shared restrictions apply. Review your entire file, not an isolated snippet.

For example, this illustrative policy allows public pages while excluding an account area:

User-agent: PerplexityBot
Allow: /
Disallow: /account/

To request that PerplexityBot avoid the whole site instead:

User-agent: PerplexityBot
Disallow: /

Choose the policy you intend; do not paste both examples. Robots rules are not authentication. Keep account data behind actual access controls. The RFC explicitly warns against treating the file as a security mechanism.

Verify the incoming request before granting an exception

Perplexity's WAF guidance combines the claimed agent with that agent's published IP ranges. PerplexityBot and Perplexity-User have separate lists. Use the current official endpoints rather than copying addresses from an old article.

Our recommended investigation record is deliberately small:

RecordWhy you need it
UTC timestamp, host and pathConnect the event to the rule and release in force
Claimed user agentIdentify what the request claims to be
Source address from your trusted edgeCompare with the appropriate vendor list
Edge status and returned contentDistinguish a page from a block or challenge
Verification result and list-check dateKeep confirmed and unconfirmed events separate

Do not allowlist every request containing the word Perplexity. If an address cannot be verified, label the event unverified and investigate it. Avoid turning uncertainty into a blanket firewall bypass.

Diagnose a block in order

  1. Check the exact URL. Use the page customers need AI to read, not only the homepage.
  2. Evaluate its robots policy. Review the named group and any shared restrictions it replaces.
  3. Inspect the edge event. Find the rule responsible for a refusal or challenge before changing security settings.
  4. Inspect the body. A successful status carrying a challenge or empty shell is not evidence that the facts arrived.
  5. Retest and save both observations. Keep the configuration change and resulting response together.

A request you send with a copied crawler user agent tests your own network path. It does not become a real PerplexityBot request. Use that probe to locate a possible disagreement, then corroborate with verified incoming events. A crawler access check is a starting point for this investigation.

What if access works but Perplexity still does not cite us?

Record that outcome honestly: access works, citation not observed. Save the prompt, date, country, answer and cited URLs before making another change. For the broader discovery question, read how Perplexity finds and cites sources; this article owns the narrower access and verification task.

Method and limits: vendor documentation and protocol review, 15 November 2026. No customer logs were accessed, no firewall settings were changed, and no crawl-compliance or citation-rate experiment was run.

Share

Want more of our research in Google Search and AI Mode? Add us as a preferred source.

Add 99Visibility as a preferred source on Google

Can AI crawlers reach your site?

Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.

Start for Free

Related Articles