Agent analytics is the measurement of AI agents visiting your website: the crawlers that gather pages for AI training and AI search, the fetchers that open a page while an assistant answers someone's question, and the browsers that AI agents drive to complete a task. You read it from your server, CDN or host logs, not from page-tag analytics, because Google Analytics automatically excludes known bots. Agent analytics answers one question well: can AI systems reach my pages, and which ones do they read? It does not tell you whether an answer named you.
The term means three different things, depending on who is asking:
- In contact centres, Observe.AI defines agent analytics as "the process of tracking agent performance in a contact center" (Observe.AI). The agents are the staff answering customers.
- In product teams, Amplitude says AI agent analytics "measures conversation quality, tool usage, and business outcomes for AI agents in production" (Amplitude). The agents are ones you built.
- In AI search, the sense this guide covers, the agents are other companies' AI systems visiting your site. LLM Pulse, OtterlyAI and Profound each sell a product called Agent Analytics in this sense. LLM Pulse puts the reason plainly: "Before an AI model can cite your site, its crawlers have to visit it" (LLM Pulse).
Every crawler description below comes from the operator's own documentation, and every tool description from the vendor's own page, read on 12 November 2026 and linked where it is used. We have not run these tools side by side, so this compares what each vendor says about itself; it is not a test. Crawler names and tool plans change, so check the linked page before you rely on a detail.
TopCited is a GEO platform with SEO included. It does not read server logs or report crawler traffic, so none of the tools compared here is ours, and several compete with us on AI answer monitoring. Where TopCited fits is set out at the end.
Key definitions
| Term | Definition |
|---|---|
| Agent analytics | Measuring which AI crawlers, fetchers and agent-driven browsers request your pages, how often, and what your server returned to them. |
| Training crawler | A crawler that collects content that may be used to train AI models, such as OpenAI's GPTBot and Anthropic's ClaudeBot. |
| Search crawler | A crawler that indexes pages so an AI assistant can find and link them in answers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. |
| User-triggered fetcher | A request made because a user asked something, such as ChatGPT-User, Claude-User and Perplexity-User. It is not an automatic crawl. |
| Agentic browser | A real browser that an AI agent drives to complete a task for a user. Anthropic marks its own with a Claude-Agent token; Google's agents identify as Google-Agent. |
| Robots.txt product token | A name you can use in robots.txt that never visits. Google-Extended is one: it controls how Google may use content it already crawls. |
| Verified request | A request whose IP address is on the list its operator publishes or, for Google, one that passes a reverse and forward DNS check. |
The AI agents that visit your site
The table lists the agents that OpenAI, Anthropic, Perplexity and Google document for site owners. It is not every AI bot on the web; LLM Pulse, for one, says its product tracks "over 40 AI bots grouped by company".
| Agent | Operator | What it does | Follows robots.txt? | How to verify it |
|---|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models | Yes. Disallowing it says your content should not be used for training | IP list at openai.com/gptbot.json |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | Yes. Opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links | IP list at openai.com/searchbot.json |
| ChatGPT-User | OpenAI | May visit a page when a ChatGPT or Custom GPT user asks a question | OpenAI says robots.txt rules may not apply, because a user started the action | IP list at openai.com/chatgpt-user.json |
| ClaudeBot | Anthropic | Collects web content that could contribute to training Anthropic's models | Yes. Anthropic says its bots honour robots.txt | IP list at claude.com/crawling/bots.json |
| Claude-SearchBot | Anthropic | Navigates the web to improve search result quality for Claude users | Yes | IP list at claude.com/crawling/bots.json |
| Claude-User | Anthropic | May access a site when a Claude user asks a question | Yes. Disabling it stops Claude retrieving your content for a user's query | IP list at claude.com/crawling/bots.json |
| Claude-Agent | Anthropic | A browser hosted by Anthropic, driven by a Claude agent to complete a task at a user's direction | Set separately from Anthropic's crawler preferences | IPs listed under Claude-Agent at claude.com/.well-known/sail |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results; not used to crawl content for AI foundation models | Yes | IP list at perplexity.ai/perplexitybot.json |
| Perplexity-User | Perplexity | May visit a page when a Perplexity user asks a question, and link it in the answer | Generally ignores robots.txt, because a user requested the fetch | IP list at perplexity.ai/perplexity-user.json |
| Google-Agent | Used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request | Google's user-triggered fetchers generally ignore robots.txt | Google's user-triggered-agents.json list, or a reverse and forward DNS check | |
| Google-Extended | Not a visitor. A robots.txt token for whether content Google crawls may be used to train future Gemini models and for grounding | It is a robots.txt token only | Nothing to verify: it has no user agent string of its own |
Sources: OpenAI's overview of its crawlers; Anthropic's pages on its crawlers and its hosted browsers; Perplexity's crawler documentation; Google's lists of user-triggered fetchers and common crawlers.
Four things in that table matter more than the rest:
- Each job has its own switch. OpenAI says "Each setting is independent of the others": a site can allow OAI-SearchBot to appear in ChatGPT search answers and disallow GPTBot to keep its content out of training. Anthropic and Perplexity split their crawlers the same way.
- User-triggered fetchers are the closest thing to a live signal. A ChatGPT-User, Claude-User or Perplexity-User request means someone's question led the assistant to open your page. Our guide to monitoring brand references in ChatGPT live covers that signal for ChatGPT.
- Agentic browsers look like browsers. Anthropic says Claude-Agent requests "come from a real browser running on Anthropic's infrastructure" and carry that browser's user agent with a Claude-Agent token added, so it advises you to "match on the Claude-Agent token rather than the full string". The identity schemes are new: Anthropic says the format it uses to publish those addresses "is not yet a standard", and Google says it is "experimenting with the Web Bot Auth protocol" for its agents.
- Google-Extended never appears in a log line. Google says it "doesn't have a separate HTTP request user agent string" and that crawling "is done with existing Google user agent strings". It also says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". If a report shows Google-Extended as a visitor, ask the vendor how it identified those requests.
Why Google Analytics does not show AI crawlers
Google's help page is direct: "In Google Analytics properties, traffic from known bots and spiders is automatically excluded." It adds that "you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded" (Known bot-traffic exclusion).
So agent analytics reads the record your web server, CDN or host writes for every request, whether or not a page script ran. That is why the tools below connect to your logs or your CDN instead of asking you to install a tag. Ahrefs says its Bot Analytics "captures bot traffic that JavaScript-based tools would never see" (Ahrefs Bot Analytics), and Profound says its Agent Analytics "Uses server logs instead of JavaScript trackers" (Profound).
What Google Analytics does show is the other half: the people who click through from an AI answer. Keep both. Crawler requests tell you what AI systems read; referrals tell you what reached a person.
How to set up agent analytics, step by step
- Find your request logs. They live on your web server (Apache or nginx access logs), your CDN (Cloudflare, Amazon CloudFront, Fastly, Akamai, Bunny) or your host (Vercel, Netlify). Keep at least the timestamp, the path, the status code and the user agent, plus the client IP so you can verify each request. LLM Pulse's setup guide, for example, asks CloudFront users to keep the date, time, user agent, URI and status fields.
- Filter on the agent names in the table. Match the token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Agent, PerplexityBot, Perplexity-User, Google-Agent), not the whole string. OpenAI notes that "the version number may change".
- Verify every request before you count it. Anthropic warns that "Anyone can send the same User-Agent header". Check each IP against the operator's published list. For Google, run a reverse DNS lookup on the IP, check that the name is in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP (Verify requests from Google). Fetch the lists again regularly: Anthropic says its agent addresses "may change over time".
- Split the traffic by job. Training crawls, search crawls, user-triggered fetches and agentic-browser sessions mean different things, and Scrunch's Agent Traffic, for one, separates "training, indexing, and retrieval" (Scrunch).
- Check what your server returned. A 403 or a challenge page served to OAI-SearchBot or PerplexityBot means that engine cannot read the page. Perplexity's documentation says that behind a web application firewall "you may need to explicitly allow Perplexity's bots", and that Cloudflare's Bot Fight Mode "can't be skipped with WAF custom rules". Our guide to Perplexity crawler access walks through it.
- Check that robots.txt says what you mean. Allow the search crawlers of the assistants you want to appear in, and decide about training crawlers separately. Perplexity says a change "may take up to 24 hours" to take effect. Anthropic says that blocking its IP addresses instead of using robots.txt "may not work correctly or persistently guarantee an opt-out".
- Watch the pages you care about over time. After you publish or update a page, see whether the search crawlers request it. LLM Pulse calls rising AI crawler traffic "an early signal that models are picking it up". Read it as a sign of discovery, not of citation.
- Put it next to answer monitoring and referrals. Agent analytics tells you a page was read. Whether an answer named you takes a prompt-tracking tool, and whether a person arrived takes your web analytics. Our guide to AI search visibility metrics and KPIs covers the answer-side measures.
What to measure
| Metric | What it tells you | What it cannot tell you |
|---|---|---|
| Verified requests per agent, per day | Which AI systems are reading your site, and the trend | Whether any answer used what they read |
| Pages requested by search crawlers | Which pages AI search can discover | Which of them an answer cites |
| Status codes returned to AI agents | Whether your firewall, CDN or server is blocking them | Why a crawler did not come back |
| User-triggered fetches per page | Which pages assistants open while answering real questions | What the answer said about you |
| Agentic-browser sessions | Where AI agents try to complete tasks on your site | Whether the task succeeded, unless your logs record the outcome |
| Requests that fail verification | How much traffic claims to be an AI agent and is not | Who sent it |
Agent analytics tools compared
| Tool | What it reports | How the data gets in | Availability (as listed 12 Nov 2026) |
|---|---|---|---|
| Cloudflare AI Crawl Control | Which AI services access your content, which crawlers follow your robots.txt, and allow or block rules per crawler. Pay per crawl is in private beta | Built into Cloudflare, for sites behind it | Cloudflare says it works automatically on all Cloudflare plans |
| Ahrefs Bot Analytics | Search engines, AI crawlers, SEO tools and social bots, with one toggle to isolate AI bot traffic; flags crawls wasted on 404s and low-value pages | Cloudflare Logpush or Worker, Vercel, Amazon CloudFront or Fastly. No JavaScript | Free during beta, on all paid Ahrefs plans |
| LLM Pulse Agent Analytics | Over 40 AI bots grouped by company, daily charts per bot, and a Per URL tab with hits, status codes and first and last seen dates | Cloudflare, Bunny CDN, log drains (Vercel, Akamai, Fastly, Netlify, GCP Load Balancer), CloudFront or S3 log buckets, or a log upload | Scale plans and above; Per URL analysis uses monthly credits |
| OtterlyAI Agent Analytics | How AI agents crawl your site, alongside its AI answer tracking | Not stated on its pricing page | Standard (200,000 events a month) and Premium (1,000,000 a month); custom on Enterprise |
| Profound Agent Analytics | When, how often and which AI bots access your content, and how many human visitors convert from AI search; separates real AI bots from spoofed ones | Server-log forwarding from cloud, edge and web platforms, instead of JavaScript trackers | Not stated on the page we read |
| Scrunch Agent Traffic | Visits from AI agents over time, split into training, indexing and retrieval, and the pages agents rely on most | Not stated on the page we read | Part of its Agent Experience Platform (AXP) |
| Screaming Frog Log File Analyser | Crawled URLs, crawl frequency and response codes for search and AI bots, with bot verification | Upload your log files to a desktop app for Windows, macOS or Linux | Free for 1,000 log events; a yearly licence removes the limit |
Sources: Cloudflare AI Crawl Control (formerly AI Audit), Ahrefs Bot Analytics, LLM Pulse Agent Analytics, OtterlyAI pricing, Profound Agent Analytics, Scrunch Agent Traffic and Screaming Frog Log File Analyser. Most of these vendors also sell AI answer tracking; the table covers only their crawler analytics.
Which agent analytics tool to use
Our recommendation for most sites: start with the logs you already have. Filter them on the agent names, verify the IPs, and fix any 403s served to the search crawlers you want. That costs nothing, and it settles the question that matters first, whether AI search can read you at all. Then:
- If your site is behind Cloudflare, open AI Crawl Control. Cloudflare says it works automatically on all plans, and it lets you allow or block each crawler from the same place you see it.
- If you already pay for Ahrefs, try Bot Analytics while it is free in beta.
- If you want crawler data beside AI answer tracking in one product, LLM Pulse, OtterlyAI, Profound and Scrunch each offer both. Check which plan includes the crawler feature: LLM Pulse lists it on Scale and above, and OtterlyAI on Standard and above.
- If you have no CDN integration and no budget, export a day of logs and open it in Screaming Frog's free tier.
What agent analytics cannot tell you
- A visit is not a citation. A ChatGPT-User request shows that ChatGPT opened your page, not what it said. OpenAI notes that ChatGPT-User "is not used to determine whether content may appear in Search".
- No visit is not proof of absence. An assistant can name you on the strength of another site's page, or of material its model was trained on, without requesting anything from you.
- Google's AI features in Search are better measured in Search Console. Its generative AI performance report includes impressions for AI Overviews and AI Mode, by page, country and device (Google).
- More requests are not more visibility. A crawl tells you a page was read, not that an answer used it. OpenAI adds that if a site allows both GPTBot and OAI-SearchBot, it may use the results from one crawl for both purposes.
- User agents can be faked. Count only verified requests. Profound says its product "Differentiates real AI bots from spoofed crawlers", and LLM Pulse says spoofed user agents "are not over-counted"; ask any vendor how it does it.
Where TopCited fits
TopCited is a GEO platform with SEO included, and it does not do agent analytics: it does not read your logs or count crawler visits. It covers the step after it.
- Content monitoring runs your query sets on a schedule you set (cron, with timezone support), and records brand mentions, gap analysis and share of voice. That shows whether the pages AI systems read are turning into answers that name you.
- The CORE optimizer rewrites your existing content and scores it before and after, including keyword coverage, so a page AI systems read but do not cite gets improved rather than reported.
- A GEO audit is available through the form on our site if you want to see where you stand first.
Use agent analytics to confirm that AI systems can read your site, and answer monitoring to learn whether they say your name. For the case for monitoring AI answers at all, see why use AI search monitoring tools.
Frequently asked questions
Agent analytics is the measurement of AI agents visiting your website: training and search crawlers such as GPTBot, ClaudeBot and PerplexityBot, fetchers such as ChatGPT-User that open a page while an assistant answers a question, and agent-driven browsers such as Claude-Agent and Google-Agent. It is read from server, CDN or host logs. The same term also names performance tracking in contact centres and the measurement of AI agents a company builds itself.
Google says traffic from known bots and spiders is automatically excluded from Google Analytics properties, and that you cannot turn the exclusion off or see how much was excluded. To see AI crawlers, use the request logs your server, CDN or host keeps.
Decide per job. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, while disallowing GPTBot signals that content should not be used for training, and the two settings are independent. Perplexity says PerplexityBot is not used to crawl content for AI foundation models. Google says Google-Extended does not affect a site's inclusion in Google Search. Allow the search crawlers of the assistants you want to appear in, and decide about training crawlers separately.
Check the request's IP address against the list its operator publishes. OpenAI, Anthropic and Perplexity each publish JSON lists, and Google publishes IP ranges and also supports a reverse and forward DNS check. Anyone can send a crawler's user agent string, so do not count a request you have not verified.
No. It means a user's question led ChatGPT to open the page. OpenAI says ChatGPT-User is not used to determine whether content may appear in Search, and the visit does not show what the answer said. Pair it with prompt monitoring to see what was said.
They identify AI agents that browse the web to complete tasks for a user. Anthropic says Claude-Agent requests come from a real browser on its infrastructure, carry a Claude-Agent token in the browser's user agent, and come from IP addresses it publishes at claude.com/.well-known/sail. Google says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request, from IP ranges in its user-triggered-agents list.
Cloudflare says AI Crawl Control works on all Cloudflare plans, Screaming Frog's Log File Analyser is free for 1,000 log events, and Ahrefs Bot Analytics is free during its beta on paid Ahrefs plans. Your own server logs, filtered on the agent names and verified against the published IP lists, cost nothing.
No. TopCited does not read server logs or report crawler visits. Its content monitoring runs your query sets on a schedule and records brand mentions, gap analysis and share of voice, and its CORE optimizer rewrites existing content with before and after scores.