Additional commentary by Emily Lyons
Welcome back to AI Pulse, our blog series in which we share insights on the state of the AI bots and the AI agent landscape.
If you missed any of the previous blog posts in the AI Pulse series, be sure to check them out:
This post’s topic: AI search crawlers. We’re diving into how they behave, who they target, and what trends we’re seeing.
The AI search crawler dilemma
Since we began tracking AI bots and agents, one category has always been tricky to define: AI search activity. Sometimes AI behaves like a crawler, other times like a fetcher. Most AI vendors now assign distinct user agents for AI search, and industry analysts are beginning to treat it as its own class of activity.
AI search is not a major focus for legacy vendors with mature ecosystems. As noted in a previous AI Pulse, their crawling and data collection already flow through existing ingest pipelines — the same ones powering their traditional search and indexing operations.
After months of observing this activity and referencing AI vendor definitions of their distinct user agents for search, Akamai has defined AI search crawlers as AI bots that automatically scan and index websites for AI-powered search experiences.
AI bot activity and AI search crawlers
One thing is clear: The activity of AI search crawlers is on the rise — and it’s happening fast. The bots we classify under this category include Amazon Kendra, iAskBot, Claude-SearchBot, PerplexityBot, and OAI-SearchBot.
Compared with the surge in AI fetcher activity, this growth pattern in AI search crawlers is more volatile — marked by sharper spikes — and shows a slightly higher overall percentage increase (2.7x vs. 2.4x; Figure 1). Both AI search crawler and AI fetcher traffic growth is organic, driven by steady end-user adoption of AI-powered products.
The rhythm of AI search crawler growth
Figure 2 represents the breakdown of AI search crawler growth by individual AI search crawler.
OpenAI (OA-SearchBot) is driving the majority of activity, but it is not alone. AI search and AI fetching are both highly sensitive to organic, user-driven growth: the more people use AI tools, the more bot activity we observe. This stands in contrast to AI training crawlers, which methodically consume the internet on fixed schedules, indexing content one page at a time.
The growth of AI search crawlers, however, seems to happen in distinct steps rather than in a smooth climb. By ignoring short multiday spikes, we can see clear plateaus from June through mid-July, mid-August through early September, and late September through early November 2025.
What’s particularly interesting is that this stepped pattern does not appear to be tied to any single vendor, suggesting a broader, ecosystem-wide trend in how users engage with AI search capabilities.
Breaking down individual AI search crawlers
It’s interesting to observe how AI search crawlers target different customers and to watch their individual trends over time. Let’s break down the traffic we see across the top three AI search crawlers, OAI-SearchBot, PerplexityBot, and Claude-SearchBot.
OAI-SearchBot traffic
OAI-SearchBot, as defined by OpenAI, is a web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time information retrieval in OpenAI's search applications.
We can see that OAI-SearchBot saw notable growth among specific industries (Figure 3). As traffic increased, some industries (such as commerce) were targeted more thoroughly.
PerplexityBot traffic
PerplexityBot, as described by industry analysts, functions as an AI search crawler that indexes web content to power Perplexity’s search results. Like OAI-SearchBot, it exhibits customer-specific targeting behavior, though across a much broader set of domains (Figure 4).
At its peak, PerplexityBot recorded more than 30 million triggers in a single day, compared with OAI-SearchBot’s high of more than 250 million. This difference highlights how Perplexity’s search crawler distributes its activity more widely, while OpenAI’s tends to concentrate traffic more heavily on select targets.
Claude-SearchBot traffic
Claude-SearchBot, as defined by industry analysts, is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.
Unlike its peers, this AI search crawler hasn’t generated much industry discussion. While reviewing AI search crawler activity for this post, however, we noticed a massive spike on November 4, 2025, concentrated among two primary industries but visible across dozens more (Figure 5).
At the time of this writing, we are not sure what the spike is related to or how long it may last, but our team will continue to track this significant traffic increase.
The so what
AI search crawlers are changing how the web gets discovered. What used to be traditional search indexing is now driven by AI agents that scan and learn from sites in real time.
For businesses, this means your content may already be powering AI search results — whether you planned for it or not.
The takeaway? Gain control of your AI search crawler traffic. Know who’s crawling your site, how often, and what they’re taking, so you can stay visible, protected, and optimized for the AI era.
Learn more
To learn more about how to manage and control AI search crawler traffic, contact an Akamai expert.
Tags
