When you ask Claude to recommend a business, compare products or research a particular subject, where does it actually get its information from?
It’s a question that matters increasingly to SEOs and marketing teams trying to understand why their businesses appear in some AI-generated answers but not others.
With traditional SEO, we’re accustomed to thinking about Google’s crawlers, its search index and the ranking systems that determine which websites appear for a particular query. AI search introduces another layer of complexity because the information presented to users can come from a combination of existing model knowledge and external retrieval systems.
What’s particularly interesting about Claude is that its developer, Anthropic, has publicly disclosed several companies involved in its web-search infrastructure.
According to the Anthropic Trust Center, these include Brave Search, TurboPuffer and, most recently, Parallel Web Systems.
All three are classified as web-search subprocessors, but they provide different technologies, and their exact roles within Claude aren’t fully public.
At Peaco Marketing, we’ve been investigating these relationships to understand what they mean for search visibility. Here’s what the official documentation tells us about how Claude discovers and retrieves information, along with the implications for businesses investing in SEO and Generative Engine Optimisation (GEO).
How Does Claude Search the Web?
Before looking at the individual providers, it’s worth understanding that Claude doesn’t necessarily search the internet every time somebody asks a question.
Like other large language models, Claude can generate responses using knowledge acquired during training. If you ask it to explain a well-established concept, it may be able to provide an answer without retrieving any new information.
However, when a question requires current information, Claude can use web-search tools to retrieve external sources.
Anthropic’s official web-search documentation describes a process where Claude determines when a search is necessary, executes searches through the available tool and receives results that it can use to construct an answer.
Importantly, Claude can perform multiple searches during the same request.
For example, someone asking Claude to recommend a local marketing agency might trigger searches relating to agencies in the relevant location, their services or other information needed to answer the question.
However, that doesn’t mean we can see every search query Claude uses or know precisely how it selects the businesses that appear in its final recommendation.
Claude’s Search Process
Based on Anthropic’s published API documentation, we can describe a simplified search process:
- The user asks a question. Claude interprets the request and determines whether external information is needed.
- Claude initiates a search. Its web-search tool retrieves information relevant to the query.
- Results are supplied to Claude. Depending on the tool version, results may be filtered before reaching the model’s context window.
- Claude generates an answer. The model uses the available information to respond, potentially including citations to retrieved sources.
This is an important distinction from a conventional search engine results page.
In Google Search, the user is generally presented with a selection of websites to choose from. With Claude, the model can interpret information from several sources and incorporate it into a single answer.
For businesses, the challenge isn’t simply appearing somewhere in a search index. Their information needs to be discoverable, relevant to the user’s question and useful enough to contribute to the eventual answer.
Anthropic’s Web-Search Providers: What We Know
One of the most useful sources for understanding Claude’s infrastructure is Anthropic’s Trust Center.
Companies operating AI services often rely on external providers for infrastructure, security and data processing. These relationships are disclosed through subprocessor lists, which identify organisations processing information on the company’s behalf.
Anthropic’s list explicitly identifies three providers under the category of web search.
| Provider | Anthropic’s classification | Product coverage |
|---|---|---|
| Brave Search | Web Search | All products |
| TurboPuffer | Web Search | All products except Claude for Government |
| Parallel Web Systems | Web Search | All products except Claude for Government |
Source: Anthropic Trust Center, Subprocessors, checked October 2026.
These disclosures are significant because they establish a direct relationship between Anthropic and all three providers.
However, there’s an important limitation.
A subprocessor listing doesn’t reveal exactly how a service is implemented, how frequently it’s used or what role it plays in ranking and selecting individual results.
For example, the fact that TurboPuffer is listed as a web-search provider doesn’t automatically mean it operates the same type of public web index as Brave.
Understanding what each company actually does gives us a better idea of how the pieces might fit together.
Brave Search: An Independent Search Index Used by Anthropic
Brave Search is arguably the easiest provider to understand because it’s already a publicly accessible search engine.
Unlike many search products that depend on results licensed from larger search providers, Brave operates its own independent web index.
Brave’s official search index documentation confirms that it uses its own crawler to discover and index web pages, with results served from its independent index.
This matters because websites don’t necessarily have identical visibility across Google, Bing and Brave.
A business might rank well in Google while appearing much less prominently in Brave’s results. Equally, pages can be discovered and indexed at different times because the search engines operate separate crawling and indexing systems.
How Brave Makes Information Available to AI Systems
Brave offers more than traditional search results.
Its Search API documentation describes several products, including conventional web search and an LLM Context API designed specifically for AI applications.
Traditional web search generally returns ranked websites with information such as titles, URLs and snippets.
The LLM Context API is designed to return relevant passages from web pages in a format suitable for language models. Rather than requiring an AI application to retrieve and process every page independently, Brave can provide extracted content associated with the original sources.
This distinction is important because modern AI search systems often need usable information rather than simply a list of links.
For example, if someone asks about the services offered by a particular company, an AI application might benefit from receiving relevant passages explaining those services instead of just the company’s homepage URL.
However, we need to be careful about the conclusion.
Brave’s documentation confirms that these capabilities exist, while Anthropic confirms Brave’s role as a web-search subprocessor. It doesn’t establish that every Claude search uses Brave’s LLM Context API or disclose precisely how Anthropic configures its retrieval systems.
What Does This Mean for SEO?
The immediate implication is that Google visibility and Claude visibility shouldn’t be treated as interchangeable.
If you’re investigating why a website isn’t being discovered in Claude, checking whether its important pages appear in Brave Search may provide useful diagnostic information.
It’s not proof of how Claude selected its sources, but it can help identify discrepancies between independent search indexes.
It’s also worth noting that Brave doesn’t provide the same webmaster reporting environment as Google Search Console. Although Brave has documented processes for requesting that certain pages be refreshed or removed, that isn’t equivalent to having a comprehensive URL inspection and indexing management platform.
For now, we’d treat Brave visibility as one part of a wider Claude GEO investigation rather than a standalone optimisation strategy.
TurboPuffer: The Search Database Behind Part of Anthropic’s Infrastructure
TurboPuffer is particularly interesting because, unlike Brave, it isn’t a consumer-facing search engine.
It’s a search database designed to store, retrieve and rank large collections of information efficiently.
According to TurboPuffer’s technical documentation, the platform supports both vector search and full-text search, including hybrid approaches that combine the two.
To understand why this matters, it’s worth looking at how these techniques differ.
Full-Text Search vs Vector Search
Traditional full-text search can identify documents containing particular words or phrases and rank them according to their relevance.
TurboPuffer supports BM25, a widely used information retrieval algorithm that considers factors such as term frequency and document length.
Vector search works differently.
Instead of relying exclusively on the exact words used in a query, vector search represents information numerically, allowing systems to retrieve content based on semantic similarity.
Imagine somebody searching for “software to manage customer relationships”.
A full-text search system might find pages containing those words, while a semantic search system could potentially identify relevant information about CRM platforms even where the wording differs.
Combining these approaches is known as hybrid search.
TurboPuffer’s documentation explains how hybrid retrieval can produce candidates using both search methods, which can then be combined or reranked to improve relevance.
Why Would This Be Useful for Claude?
Large language models need access to relevant information without necessarily processing an entire collection of documents.
A search database capable of retrieving information efficiently could support many different retrieval workflows, including AI search.
This makes TurboPuffer’s role as an Anthropic web-search subprocessor particularly interesting.
However, we don’t currently have enough public information to say exactly what Anthropic stores in TurboPuffer or how its systems interact with the database.
It would be reasonable to describe TurboPuffer as technology capable of supporting sophisticated retrieval infrastructure. It wouldn’t be reasonable to claim that Claude uses a particular TurboPuffer ranking configuration to select websites without supporting evidence.
For SEOs, the distinction matters because the existence of vector search doesn’t mean conventional keywords have suddenly become irrelevant.
TurboPuffer’s own documentation demonstrates that modern retrieval systems can combine exact term matching with semantic similarity.
That reinforces the importance of creating content that accurately describes your services and provides useful context, rather than abandoning established SEO practices in favour of vague claims about semantic optimisation.
Parallel Web Systems: Anthropic’s Newer Web-Search Provider
Perhaps the most interesting recent development is the addition of Parallel Web Systems.
On 14 October 2026, Anthropic published a Trust Center update identifying Parallel as a new web-search subprocessor for its products, excluding Claude for Government.
This is particularly relevant because Parallel provides infrastructure specifically designed to help AI applications search and retrieve information from the web.
Its products include web-search and extraction APIs, alongside tools designed to support more involved research workflows.
How Does Parallel Retrieve Information?
Parallel’s Anthropic integration documentation explains how developers can connect its search services to Anthropic-powered applications using tool calling.
In that implementation, Claude can determine that a search is needed, call a registered search tool and receive sourced results that it can use to construct an answer.
Parallel also operates its own crawling infrastructure.
Its official crawler documentation identifies a crawler called ShapBot, which helps discover and index websites for Parallel’s web APIs.
According to Parallel, website owners can support discovery by allowing ShapBot through their robots.txt configuration and permitting legitimate requests from its designated IP ranges.
Parallel also provides mechanisms to verify genuine crawler requests, including cryptographically signed HTTP messages and a published IP registry.
These details are particularly useful for developers managing websites behind firewalls or bot-protection systems.
Why Is ShapBot Important for GEO?
For search engines and AI retrieval providers to discover web content independently, they need some way to access it.
If a website blocks a legitimate crawler, that provider may be unable to collect information from the site through its normal indexing process.
Parallel explicitly recommends allowing ShapBot to help maximise visibility in results provided through its web APIs.
However, we shouldn’t confuse being accessible to Parallel with being guaranteed a recommendation in Claude.
Parallel’s subprocessor relationship with Anthropic establishes that it provides web-search services, but it doesn’t reveal how often Claude uses its index or how retrieved information is selected for any particular response.
What we can say is that AI crawler accessibility is becoming a legitimate technical SEO consideration, especially for websites that deliberately restrict automated requests through security services.
This is an area where website owners and technical SEO specialists have something practical to investigate.
Anthropic Also Operates Its Own Crawlers
So far, we’ve discussed three external providers, but Anthropic also operates web crawlers itself.
Its official crawler documentation, updated in April 2026, identifies three bots with different purposes.
| Crawler | Documented purpose |
|---|---|
| ClaudeBot | Collects public web content that may contribute to model training |
| Claude-SearchBot | Crawls web content to improve the relevance and accuracy of search results |
| Claude-User | Retrieves website content in response to user-initiated requests |
This is one of the most important distinctions for website owners.
Not all AI crawlers serve the same purpose, and decisions about permitting or blocking them should reflect what the business is trying to achieve.
ClaudeBot: Model Development
ClaudeBot collects information that may be used to develop Anthropic’s models.
Its purpose is distinct from retrieving a website in response to an immediate user question.
Anthropic documents controls that allow website owners to restrict this crawler.
For organisations with concerns about the use of their content for AI training, it’s therefore important to understand that restricting ClaudeBot is not the same as restricting every form of Claude search access.
Claude-SearchBot: Search Discovery
Claude-SearchBot is more directly relevant to AI search visibility.
Anthropic states that the crawler is used to improve the relevance and accuracy of search responses.
It also warns that blocking Claude-SearchBot prevents its system from indexing that content for search optimisation, potentially reducing visibility or accuracy in search results.
This provides direct evidence that Anthropic has its own search-related crawling activities, separate from its relationships with external web-search providers.
Claude-User: User-Requested Retrieval
Claude-User supports requests initiated by people using Claude.
For example, a user might ask Claude to inspect a particular website or retrieve information from a page.
Anthropic explains that restricting Claude-User prevents that system from accessing content in response to these user-directed requests.
For businesses trying to improve their visibility within Claude, that distinction is important.
You might have a website that’s indexable by conventional search engines but unintentionally blocks the bot used for a particular AI retrieval operation.
Equally, access by Claude-User doesn’t guarantee that your website will be indexed, recommended or cited in unrelated searches.
How Search Results Become Claude Answers
Understanding the providers is useful, but there’s another stage that’s just as important: what happens after information has been retrieved.
One of the biggest misconceptions surrounding GEO is that appearing in a search provider’s index automatically means appearing in an AI-generated answer.
That isn’t how the process should be understood.
A search provider might return several potentially useful sources, but Claude still has to interpret the information and produce a response appropriate to the question.
Anthropic’s web-search documentation provides a particularly interesting detail about how this process has evolved.
With the web_search_20260209 tool version and later, supported Claude models can use dynamic filtering to process search results before they’re added to the model’s context window.
Rather than loading every retrieved result, Claude can filter information to retain material relevant to the task.
The later web_search_20260318 version also introduced additional response-inclusion controls for agentic workflows.
These features reinforce an important point: retrieval and answer generation are separate processes.
A website might be present in a search provider’s index but never make it into the information ultimately used to construct the answer.
Even when its content is retrieved, that doesn’t necessarily mean Claude will mention the business or cite the source.
For SEOs, this makes AI visibility more complicated than tracking conventional keyword rankings.
What Does All This Mean for Claude SEO?
The information we’ve examined tells us a reasonable amount about the technologies involved in Claude’s search infrastructure.
However, it also demonstrates why businesses should be cautious about anyone claiming to know exactly how to rank number one in Claude.
We don’t have a publicly documented, universal ranking algorithm that determines which businesses Claude recommends.
We do have evidence supporting several practical considerations.
1. Crawl Accessibility Matters
Anthropic and Parallel both document crawlers used for search-related purposes.
For technical SEO, this makes it worth investigating whether important pages are accessible to relevant AI crawlers and whether security services are unintentionally blocking legitimate requests.
That doesn’t mean indiscriminately allowing every bot on the internet.
Website owners should make informed decisions about crawler access, verify legitimate requests where necessary and consider their organisation’s security and content policies.
2. Clear, Accessible Website Content Still Matters
AI retrieval systems need usable information.
While the exact extraction and rendering capabilities of every provider aren’t public, ensuring that important content can be accessed without unnecessary technical obstacles is a sensible foundation.
For JavaScript-heavy websites, this may involve reviewing whether essential information is available in the initial HTML response or depends entirely on client-side rendering.
Server-side rendering can help make content more accessible to systems that don’t fully execute JavaScript, although it shouldn’t be presented as a guaranteed Claude ranking advantage.
3. Being Indexed Isn’t the Same as Being Recommended
Perhaps the most important lesson is that indexing, retrieval and citation aren’t interchangeable.
A page can exist in a search index without being selected for a particular query. It can be retrieved without being used in the final answer, and it can contribute information without necessarily producing a prominent brand recommendation.
This means SEO reporting needs to look beyond whether individual URLs are indexed.
Businesses should also investigate whether they’re mentioned in relevant AI responses, whether their websites are cited and how visibility changes across different questions.
4. Don’t Abandon Traditional SEO
It’s tempting to see the arrival of AI search as evidence that everything we’ve learned about SEO needs replacing.
The documentation suggests a more nuanced picture.
Claude’s infrastructure includes technologies associated with conventional full-text retrieval, semantic search, crawling and content extraction.
The fundamentals of making accurate information accessible to search systems remain relevant, even if the way that information reaches the end user is changing.
What We Still Don’t Know About Claude Search
For all the information Anthropic and its providers have published, there are still some important unanswered questions.
We don’t know precisely how Anthropic distributes different search requests between Brave, TurboPuffer and Parallel, or whether those systems contribute to the same retrieval workflows.
We also don’t know the exact ranking and filtering signals used when selecting sources for particular Claude responses.
There’s insufficient public evidence to establish that being indexed by all three providers produces better Claude visibility than being accessible through only one.
Likewise, there isn’t enough evidence to claim that implementing llms.txt, adding additional structured data or using a particular heading format will directly improve Claude recommendations.
Some of these techniques may be useful in specific circumstances, but usefulness isn’t the same as a demonstrated ranking effect.
This is where independent testing becomes particularly valuable.
For example, researchers could compare cited websites across Claude responses and traditional search results, examine crawler requests in server logs or assess how accurately different page formats are extracted.
These investigations won’t necessarily reveal Anthropic’s internal ranking systems, but they can help establish which website characteristics are associated with successful retrieval.
Final Thoughts
Claude’s search infrastructure is considerably more interesting than a simple connection to one search engine.
Anthropic publicly identifies Brave Search, TurboPuffer and Parallel Web Systems as web-search subprocessors. Each provides different search technologies, while Anthropic also operates crawlers for model development, search discovery and user-directed retrieval.
The addition of Parallel in October 2026 demonstrates how quickly this infrastructure is evolving, and why businesses shouldn’t assume that optimising for one search engine automatically provides complete coverage across AI platforms.
For marketers and SEOs, the opportunity is to understand these systems without confusing technical capabilities with proven ranking factors.
At Peaco Marketing, we’re particularly interested in how traditional SEO foundations interact with these newer retrieval systems. Our technical SEO services cover the crawlability, rendering and indexing considerations that underpin search visibility, whether the end user is searching through Google or an AI assistant.
The next question isn’t simply how Claude finds information, but how businesses can make their content more accessible and useful to the systems involved.
That’s where the practical work of Claude GEO begins.