PerplexityBot blocked by robots.txt
Perplexity is citation heavy. Blocking PerplexityBot directly removes you from its answer engine.
Your robots.txt disallows PerplexityBot, either under an explicit group or via a catch-all. PerplexityBot honors robots.txt, so Perplexity's index cannot add or refresh your pages, which removes them from the pool of sources Perplexity can cite.
Perplexity answers often surface with inline citations back to the pages they reused. For sites that want AI-driven referral traffic, being absent from Perplexity's index means not being cited at all, even for queries where your page would be the best answer.
- A generic "block AI" robots.txt template added PerplexityBot along with GPTBot and ClaudeBot.
- A wildcard Disallow in the catch-all group sweeps up Perplexity along with every other bot.
- The site was previously under a "no AI" policy that was never revisited.
- A CDN bot rule blocks PerplexityBot at the edge.
- Open AI Crawler Checker and paste a representative URL.
- Find the PerplexityBot row in the matrix.
- Check which group matched and whether the Disallow is intentional.
- Spot-check that the page returns 200 with real text so Perplexity would have something to cite if allowed.
- 1
Add an explicit Perplexity group
Add User-agent: PerplexityBot with Allow: / or Disallow: /, matching your intent. Explicit groups beat catch-alls.
- 2
Decide AI citations as a separate policy
If you are blocking training crawlers (GPTBot, ClaudeBot, Google-Extended), consider whether you also want to block PerplexityBot. They serve different goals.
- 3
Check the CDN
Audit CDN bot rules. If PerplexityBot is returning 403 at the edge, robots.txt changes will not help.
- 4
Re-run AI Crawler Checker
Verify the new policy is in effect. The matrix should reflect your intent.
Does PerplexityBot follow robots.txt?
Yes. Perplexity's stated policy is that PerplexityBot honors robots.txt directives, including Disallow. That is also why robots.txt is an effective way to opt out if that is what you want.
How do I test PerplexityBot?
Use the AI Crawler Checker. It resolves a given URL against your robots.txt from PerplexityBot's perspective and shows allowed or blocked in the matrix.
Should publishers allow it?
If you care about AI referral traffic, allowing PerplexityBot gives you a path to being cited. If you would rather not be used in AI answers at all, disallowing it is a legitimate choice, just know it has a visibility cost.
Ready to diagnose your URL?
AI Crawler Checker runs the exact checks discussed above.