Quick answer: Perplexity runs its own search index. Perplexity describes it as covering hundreds of billions of webpages and says it’s the same infrastructure that powers its public answer engine. None of Perplexity’s own documentation names Google or Bing. That doesn’t settle it, though: in an active federal lawsuit, Reddit alleges Perplexity pulled Google search results through a scraping service called SerpApi, and in 2026 a judge let those claims proceed. The more useful detail for site owners is that Perplexity sends two different bots to your site, and by its own documentation, one of them generally ignores robots.txt.
So far in this series, we’ve followed two paper trails. Claude’s led to Brave, named plainly on Anthropic’s vendor list. ChatGPT’s led to Bing, reached through a privacy link OpenAI never quite explains. Perplexity’s trail is the most tangled of the three. Perplexity says it built its own search engine. A lawsuit says it also borrowed Google’s..
The paper trail
Exhibit A: The infrastructure deal. On April 15, 2025, Vespa.ai announced that Perplexity was bringing its search function in-house, using Vespa’s platform to scale the retrieval system that pulls from the web, internal databases and users’ files. Perplexity’s CEO, Aravind Srinivas, put it this way in the release: “The recipe: 1. Solve Search. 2. Use it to solve everything else.”
Exhibit B: The index, in Perplexity’s own words. When Perplexity opened its Search API to developers in September 2025, it said the API runs on the same infrastructure that powers its public answer engine. It described an index of hundreds of billions of webpages, broken into fine-grained sub-document units so the most relevant passage can be ranked rather than the whole page. It said its systems process tens of thousands of index update requests every second. The post names no outside search provider.
Exhibit C: The help center. Perplexity’s help center says it searches the internet in real time, then has a language model summarize what it finds into a cited answer. It names the models it uses. It doesn’t name any third-party search engine.
Exhibit D: The bot list. Perplexity publishes two user agents, each with its own IP range file. PerplexityBot surfaces and links sites in search results. Perplexity-User fetches a page live when a person’s question calls for it. The docs add that this second fetcher “generally ignores robots.txt rules,” on the reasoning that a human asked for the page.
Exhibit E: The lawsuit. In October 2025, Reddit sued Perplexity, SerpApi and two proxy companies in federal court in New York. Reddit alleges they scraped its content through Google’s search results. Reddit says it planted a post that only Google’s crawler could see, and the post surfaced in Perplexity’s answers within hours. Perplexity says it doesn’t train foundation models and that it cites Reddit threads the way people share links. In 2026, the court let Reddit’s core claims proceed. That’s an allegation that survived a motion to dismiss, not a verdict.
Put together: Perplexity running its own index is confirmed. Perplexity running only its own index is contested in court.
The two gates, and why robots.txt only guards one of them
The ChatGPT post described two gates: being found and being read. Perplexity has the same two gates, but different locks.
Gate one: being indexed. PerplexityBot decides whether your page enters Perplexity’s index. Perplexity’s docs recommend allowing it in robots.txt so your site appears in search results.
Gate two: being fetched at answer time. When someone asks Perplexity a question, Perplexity-User may visit your page live to build the answer. Perplexity’s own August 2025 blog post says that for current information, the AI goes to the relevant websites rather than pulling from a database. By Perplexity’s documentation, robots.txt generally doesn’t stop this bot. A firewall rule can block traffic that identifies itself as Perplexity-User, but Cloudflare alleges that when it did exactly that, Perplexity’s traffic came back under a generic browser identity. Perplexity disputes this.
The practical point most guides miss: on Perplexity, robots.txt works as a visibility switch, not a privacy wall. If you want to be cited, robots.txt plus a properly configured firewall is how you let both bots in. If you want to be left out, robots.txt alone won’t do it.
The controversy you should know about
On August 4, 2025, Cloudflare published a report accusing Perplexity of stealth crawling. Cloudflare said customers had blocked Perplexity in robots.txt and at the firewall but were still seeing their content in Perplexity’s answers. To test it, Cloudflare set up brand-new, unindexed domains, disallowed all crawling in robots.txt, and blocked Perplexity’s declared bots with firewall rules. Perplexity still answered detailed questions about those sites. Cloudflare said that when the declared Perplexity-User agent was blocked, traffic switched to a generic Chrome-on-Mac user agent, from IPs outside Perplexity’s published ranges. It counted 20–25 million daily requests from the declared agent and 3–6 million from the disguised one. Cloudflare then removed Perplexity from its verified-bot list and added the stealth crawler’s signatures to its managed AI-crawler blocking rule, which is available to all customers, including free plans.
Perplexity responded the same day. It argued that a user-driven assistant fetching a page for one person is not a crawler. It also said Cloudflare had wrongly attributed traffic from BrowserBase, a third-party cloud browser service, to Perplexity. By Perplexity’s account, its own BrowserBase use accounts for fewer than 45,000 requests a day.
You don’t need to pick a side. You need to know one thing: if your site sits behind Cloudflare with AI-crawler blocking switched on, you may be blocking Perplexity without having decided to.
Perplexity’s bots, and which one matters
(Keep your existing bots table. The one change: if it links to the crawler docs, update the link to https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Confirmed vs. assumed
Claim |
Status |
Source |
|---|---|---|
Perplexity runs its own search index |
Confirmed |
|
That index powers Perplexity's public answers |
Confirmed, as described by Perplexity |
|
The index covers hundreds of billions of pages |
Confirmed, as described by Perplexity |
|
Perplexity brought search in-house on Vespa.ai's platform |
Confirmed |
|
Perplexity-User generally ignores robots.txt |
Confirmed, in Perplexity's own docs |
|
Perplexity pulls results from Google |
Alleged in court, not proven |
|
Perplexity used undeclared crawlers to evade blocks |
Disputed: Cloudflare alleges it; Perplexity denies it |
|
Blocking PerplexityBot removes you from results entirely |
Not confirmed |
The ten-minute check
Open yourdomain.com/robots.txt. Make sure PerplexityBot isn’t disallowed, either by name or by a wildcard “block all AI” rule.
Check your firewall or CDN. Perplexity recommends allowlisting both bots by user agent and by the IP ranges in its two JSON files, and updating those rules automatically as the ranges change.
If you’re on Cloudflare, check your AI-crawler settings. Cloudflare’s managed AI-crawler blocking rule now includes signatures for the traffic it attributed to Perplexity, so it can catch Perplexity even if your robots.txt allows it.
Watch your server logs. After any change, look for both PerplexityBot and Perplexity-User. Perplexity notes that firewall changes take time to propagate.
Give it a day. Perplexity says robots.txt changes can take up to 24 hours to take effect.
Ask Perplexity yourself. Type a question your best page answers, and see whose site gets cited.
Front-load the answer. Perplexity says its index scores pages in fine-grained, sub-document units rather than as whole pages. Our recommendation: put a clean, self-contained answer near the top of the page, so the passage worth citing is easy to find.
FAQ
Does Perplexity use Google to search the web?
Perplexity's own documentation describes its own index and never names Google or Bing. But in an active federal lawsuit, Reddit alleges Perplexity pulled Google search results through a scraping service called SerpApi, and in 2026 a court let those claims proceed. Perplexity disputes the case. The honest answer: its own index, confirmed; Google, alleged.
If I block PerplexityBot in robots.txt, will Perplexity stop reading my site?
Not necessarily. Robots.txt governs PerplexityBot, the indexing bot. Perplexity-User, which fetches pages live when a user asks, generally ignores robots.txt according to Perplexity's own docs. A firewall can block traffic that identifies itself as Perplexity-User, though Cloudflare alleges Perplexity has gotten around such blocks. Perplexity denies it.
Is Perplexity ranking the same as Google ranking?
No. Perplexity describes its own index and its own ranking, which scores fine-grained passages of each page against the query. A strong Google position doesn't guarantee a Perplexity citation.
Should I block Perplexity because of the Cloudflare report?
That depends on what you want. Our view: if you sell services and want AI assistants to recommend you, blocking Perplexity takes you out of one more place your clients are asking. If your content is your product, the report is worth reading in full before you decide.
Not sure whether your firewall is quietly turning Perplexity away, or whether your robots.txt says what you think it says? That’s part of what we check. Get your free consultation, and we’ll show you exactly where you stand.
Sources (verified, primary)
Perplexity
Cloudflare
Litigation
