Welcome back! This is the third post in our AI Pulse blog series, which shares insights on the state of AI bots. (If you missed either of the previous posts, you can read AI Pulse: OpenAI’s Wild Bot Behavior After GPT-5 and AI Pulse: AI Bot Mitigation Is Increasing Everywhere now.)
In this blog post, we’re diving into a somewhat personal topic: sleep schedules. Did you know some AI bots actually follow a circadian rhythm? Let’s get into it.
AI bot categorization is a fuzzy task
Across the industry, there’s been a lot of discussion (and confusion!) about how to categorize AI bots based on how they behave. This is largely because AI bots’ behaviors can span different categories, AI vendors use their own categorizations, and user agent labels don’t always match.
At Akamai, we’ve worked to define four core categories for AI bots based on primary behavioral research (Table).
Category name |
Description |
Popular examples of Bot (Operator) |
|---|---|---|
AI training crawlers |
AI training crawlers automatically scan and collect large amounts of data from websites to be used as training data for large language models (LLMs) |
GPTBot (OpenAI) |
AI search crawlers |
AI search crawlers automatically scan and index websites for AI-powered search experiences |
OAI-SearchBot (OpenAI) |
AI fetchers |
AI fetchers grab specific web pages in real time to fulfill specific user requests on AI assistants, such as answering a question or summarizing a webpage via an AI chatbot |
ChatGPT-User (OpenAI) Claude-User (Anthropic) |
AI agents |
AI agents are autonomous software systems that use artificial intelligence to achieve goals and complete tasks for users. |
ChatGPT-Agent (OpenAI) |
The core categories for AI bots based on behavior
AI fetchers behave differently from AI crawlers
In creating this categorization, we realized one distinct truth: AI fetchers behave differently from crawlers. Fetchers are focused on sourcing content on behalf of a user request, while crawlers periodically source content for use in training/updating models.
AI bot fetchers are growing more rapidly than crawlers, indicating that user-driven AI bot requests are driving a significant amount of the total AI bot traffic we observe on Akamai’s network (Figure 1).
Some bots get tired
Since users (humans) need to sleep, our team hypothesized that we’d be able to tell which AI bots are user-driven versus fully automated by observing the natural circadian rhythms in their traffic.
Let’s use the ChatGPT fetcher as an example. Figure 2 shows an intense circadian rhythm, which is slightly offset between regions (a general proxy here for time zone). By taking a closer look, we see that downtimes for each region correspond with the general nighttime for that same region. The Asia-Pacific and Japan (APJ) region generally remains a common outlier when looking for these circadian rhythms.
The same behavior can be observed in Perplexity’s fetcher (Figure 3). We see the natural following of a circadian rhythm that clearly indicates user-driven behavior, though showing a slightly less regional offset than OpenAI.
Oddly, however, Anthropic’s Claude fetcher does not follow a circadian rhythm like the others — proving that our hypothesis is not entirely correct (Figure 4). Indeed, this bot does not sleep, leading us to expect differing and less user-driven behavior from this entity.
Crawlers stay awake
With the general assumption that fetchers are user-driven and crawlers are not, we can expect crawlers to stay awake and always on.
Observing the main crawlers across pure-play AI companies, there are no discernible circadian patterns in traffic. Instead, crawlers exhibit little rhythmic spikes and dips that more closely resemble a cluster restarting or being altered than anything else (Figure 5).
The so what
Although we’ve known these AI crawlers operate within systems not directly driven by humans, there’s still a widespread assumption that anything labeled “AI” functions entirely on its own.
In reality, that’s far from true. Humans remain a major force behind how AI traffic is generated and used. Fetchers (the fastest-growing category of AI bots) reflect human intent and behavior, just expressed through new interfaces. They must be treated as users, too — ones that interact with your business and data in fundamentally different ways.
Stay tuned
Stay tuned for the next AI Pulse to dive into another AI bot insight. To learn more about how to manage your AI bot traffic, contact an expert.
Tags
