Agent Analytics: How to See Which AI Crawlers and Agents Visit Your Website (2026)

Agent analytics for website owners: how to measure the AI crawlers, fetchers and browser agents from OpenAI, Anthropic, Perplexity and Google that visit your site, why Google Analytics leaves them out, how to verify them in your logs, and the tools that report them, as checked on 12 November 2026.

By TopCitedPublished Updated
Back to Blog

Agent analytics is the measurement of AI agents visiting your website: the crawlers that gather pages for AI training and AI search, the fetchers that open a page while an assistant answers someone's question, and the browsers that AI agents drive to complete a task. You read it from your server, CDN or host logs, not from page-tag analytics, because Google Analytics automatically excludes known bots. Agent analytics answers one question well: can AI systems reach my pages, and which ones do they read? It does not tell you whether an answer named you.

The term means three different things, depending on who is asking:

  • In contact centres, Observe.AI defines agent analytics as "the process of tracking agent performance in a contact center" (Observe.AI). The agents are the staff answering customers.
  • In product teams, Amplitude says AI agent analytics "measures conversation quality, tool usage, and business outcomes for AI agents in production" (Amplitude). The agents are ones you built.
  • In AI search, the sense this guide covers, the agents are other companies' AI systems visiting your site. LLM Pulse, OtterlyAI and Profound each sell a product called Agent Analytics in this sense. LLM Pulse puts the reason plainly: "Before an AI model can cite your site, its crawlers have to visit it" (LLM Pulse).

Every crawler description below comes from the operator's own documentation, and every tool description from the vendor's own page, read on 12 November 2026 and linked where it is used. We have not run these tools side by side, so this compares what each vendor says about itself; it is not a test. Crawler names and tool plans change, so check the linked page before you rely on a detail.

TopCited is a GEO platform with SEO included. It does not read server logs or report crawler traffic, so none of the tools compared here is ours, and several compete with us on AI answer monitoring. Where TopCited fits is set out at the end.

Key definitions

TermDefinition
Agent analyticsMeasuring which AI crawlers, fetchers and agent-driven browsers request your pages, how often, and what your server returned to them.
Training crawlerA crawler that collects content that may be used to train AI models, such as OpenAI's GPTBot and Anthropic's ClaudeBot.
Search crawlerA crawler that indexes pages so an AI assistant can find and link them in answers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.
User-triggered fetcherA request made because a user asked something, such as ChatGPT-User, Claude-User and Perplexity-User. It is not an automatic crawl.
Agentic browserA real browser that an AI agent drives to complete a task for a user. Anthropic marks its own with a Claude-Agent token; Google's agents identify as Google-Agent.
Robots.txt product tokenA name you can use in robots.txt that never visits. Google-Extended is one: it controls how Google may use content it already crawls.
Verified requestA request whose IP address is on the list its operator publishes or, for Google, one that passes a reverse and forward DNS check.

The AI agents that visit your site

The table lists the agents that OpenAI, Anthropic, Perplexity and Google document for site owners. It is not every AI bot on the web; LLM Pulse, for one, says its product tracks "over 40 AI bots grouped by company".

AgentOperatorWhat it doesFollows robots.txt?How to verify it
GPTBotOpenAICrawls content that may be used to train OpenAI's foundation modelsYes. Disallowing it says your content should not be used for trainingIP list at openai.com/gptbot.json
OAI-SearchBotOpenAISurfaces websites in ChatGPT's search featuresYes. Opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational linksIP list at openai.com/searchbot.json
ChatGPT-UserOpenAIMay visit a page when a ChatGPT or Custom GPT user asks a questionOpenAI says robots.txt rules may not apply, because a user started the actionIP list at openai.com/chatgpt-user.json
ClaudeBotAnthropicCollects web content that could contribute to training Anthropic's modelsYes. Anthropic says its bots honour robots.txtIP list at claude.com/crawling/bots.json
Claude-SearchBotAnthropicNavigates the web to improve search result quality for Claude usersYesIP list at claude.com/crawling/bots.json
Claude-UserAnthropicMay access a site when a Claude user asks a questionYes. Disabling it stops Claude retrieving your content for a user's queryIP list at claude.com/crawling/bots.json
Claude-AgentAnthropicA browser hosted by Anthropic, driven by a Claude agent to complete a task at a user's directionSet separately from Anthropic's crawler preferencesIPs listed under Claude-Agent at claude.com/.well-known/sail
PerplexityBotPerplexitySurfaces and links websites in Perplexity search results; not used to crawl content for AI foundation modelsYesIP list at perplexity.ai/perplexitybot.json
Perplexity-UserPerplexityMay visit a page when a Perplexity user asks a question, and link it in the answerGenerally ignores robots.txt, because a user requested the fetchIP list at perplexity.ai/perplexity-user.json
Google-AgentGoogleUsed by agents hosted on Google infrastructure to navigate the web and perform actions on a user's requestGoogle's user-triggered fetchers generally ignore robots.txtGoogle's user-triggered-agents.json list, or a reverse and forward DNS check
Google-ExtendedGoogleNot a visitor. A robots.txt token for whether content Google crawls may be used to train future Gemini models and for groundingIt is a robots.txt token onlyNothing to verify: it has no user agent string of its own

Sources: OpenAI's overview of its crawlers; Anthropic's pages on its crawlers and its hosted browsers; Perplexity's crawler documentation; Google's lists of user-triggered fetchers and common crawlers.

Four things in that table matter more than the rest:

  • Each job has its own switch. OpenAI says "Each setting is independent of the others": a site can allow OAI-SearchBot to appear in ChatGPT search answers and disallow GPTBot to keep its content out of training. Anthropic and Perplexity split their crawlers the same way.
  • User-triggered fetchers are the closest thing to a live signal. A ChatGPT-User, Claude-User or Perplexity-User request means someone's question led the assistant to open your page. Our guide to monitoring brand references in ChatGPT live covers that signal for ChatGPT.
  • Agentic browsers look like browsers. Anthropic says Claude-Agent requests "come from a real browser running on Anthropic's infrastructure" and carry that browser's user agent with a Claude-Agent token added, so it advises you to "match on the Claude-Agent token rather than the full string". The identity schemes are new: Anthropic says the format it uses to publish those addresses "is not yet a standard", and Google says it is "experimenting with the Web Bot Auth protocol" for its agents.
  • Google-Extended never appears in a log line. Google says it "doesn't have a separate HTTP request user agent string" and that crawling "is done with existing Google user agent strings". It also says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". If a report shows Google-Extended as a visitor, ask the vendor how it identified those requests.

Why Google Analytics does not show AI crawlers

Google's help page is direct: "In Google Analytics properties, traffic from known bots and spiders is automatically excluded." It adds that "you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded" (Known bot-traffic exclusion).

So agent analytics reads the record your web server, CDN or host writes for every request, whether or not a page script ran. That is why the tools below connect to your logs or your CDN instead of asking you to install a tag. Ahrefs says its Bot Analytics "captures bot traffic that JavaScript-based tools would never see" (Ahrefs Bot Analytics), and Profound says its Agent Analytics "Uses server logs instead of JavaScript trackers" (Profound).

What Google Analytics does show is the other half: the people who click through from an AI answer. Keep both. Crawler requests tell you what AI systems read; referrals tell you what reached a person.

How to set up agent analytics, step by step

  1. Find your request logs. They live on your web server (Apache or nginx access logs), your CDN (Cloudflare, Amazon CloudFront, Fastly, Akamai, Bunny) or your host (Vercel, Netlify). Keep at least the timestamp, the path, the status code and the user agent, plus the client IP so you can verify each request. LLM Pulse's setup guide, for example, asks CloudFront users to keep the date, time, user agent, URI and status fields.
  2. Filter on the agent names in the table. Match the token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Agent, PerplexityBot, Perplexity-User, Google-Agent), not the whole string. OpenAI notes that "the version number may change".
  3. Verify every request before you count it. Anthropic warns that "Anyone can send the same User-Agent header". Check each IP against the operator's published list. For Google, run a reverse DNS lookup on the IP, check that the name is in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP (Verify requests from Google). Fetch the lists again regularly: Anthropic says its agent addresses "may change over time".
  4. Split the traffic by job. Training crawls, search crawls, user-triggered fetches and agentic-browser sessions mean different things, and Scrunch's Agent Traffic, for one, separates "training, indexing, and retrieval" (Scrunch).
  5. Check what your server returned. A 403 or a challenge page served to OAI-SearchBot or PerplexityBot means that engine cannot read the page. Perplexity's documentation says that behind a web application firewall "you may need to explicitly allow Perplexity's bots", and that Cloudflare's Bot Fight Mode "can't be skipped with WAF custom rules". Our guide to Perplexity crawler access walks through it.
  6. Check that robots.txt says what you mean. Allow the search crawlers of the assistants you want to appear in, and decide about training crawlers separately. Perplexity says a change "may take up to 24 hours" to take effect. Anthropic says that blocking its IP addresses instead of using robots.txt "may not work correctly or persistently guarantee an opt-out".
  7. Watch the pages you care about over time. After you publish or update a page, see whether the search crawlers request it. LLM Pulse calls rising AI crawler traffic "an early signal that models are picking it up". Read it as a sign of discovery, not of citation.
  8. Put it next to answer monitoring and referrals. Agent analytics tells you a page was read. Whether an answer named you takes a prompt-tracking tool, and whether a person arrived takes your web analytics. Our guide to AI search visibility metrics and KPIs covers the answer-side measures.

What to measure

MetricWhat it tells youWhat it cannot tell you
Verified requests per agent, per dayWhich AI systems are reading your site, and the trendWhether any answer used what they read
Pages requested by search crawlersWhich pages AI search can discoverWhich of them an answer cites
Status codes returned to AI agentsWhether your firewall, CDN or server is blocking themWhy a crawler did not come back
User-triggered fetches per pageWhich pages assistants open while answering real questionsWhat the answer said about you
Agentic-browser sessionsWhere AI agents try to complete tasks on your siteWhether the task succeeded, unless your logs record the outcome
Requests that fail verificationHow much traffic claims to be an AI agent and is notWho sent it

Agent analytics tools compared

ToolWhat it reportsHow the data gets inAvailability (as listed 12 Nov 2026)
Cloudflare AI Crawl ControlWhich AI services access your content, which crawlers follow your robots.txt, and allow or block rules per crawler. Pay per crawl is in private betaBuilt into Cloudflare, for sites behind itCloudflare says it works automatically on all Cloudflare plans
Ahrefs Bot AnalyticsSearch engines, AI crawlers, SEO tools and social bots, with one toggle to isolate AI bot traffic; flags crawls wasted on 404s and low-value pagesCloudflare Logpush or Worker, Vercel, Amazon CloudFront or Fastly. No JavaScriptFree during beta, on all paid Ahrefs plans
LLM Pulse Agent AnalyticsOver 40 AI bots grouped by company, daily charts per bot, and a Per URL tab with hits, status codes and first and last seen datesCloudflare, Bunny CDN, log drains (Vercel, Akamai, Fastly, Netlify, GCP Load Balancer), CloudFront or S3 log buckets, or a log uploadScale plans and above; Per URL analysis uses monthly credits
OtterlyAI Agent AnalyticsHow AI agents crawl your site, alongside its AI answer trackingNot stated on its pricing pageStandard (200,000 events a month) and Premium (1,000,000 a month); custom on Enterprise
Profound Agent AnalyticsWhen, how often and which AI bots access your content, and how many human visitors convert from AI search; separates real AI bots from spoofed onesServer-log forwarding from cloud, edge and web platforms, instead of JavaScript trackersNot stated on the page we read
Scrunch Agent TrafficVisits from AI agents over time, split into training, indexing and retrieval, and the pages agents rely on mostNot stated on the page we readPart of its Agent Experience Platform (AXP)
Screaming Frog Log File AnalyserCrawled URLs, crawl frequency and response codes for search and AI bots, with bot verificationUpload your log files to a desktop app for Windows, macOS or LinuxFree for 1,000 log events; a yearly licence removes the limit

Sources: Cloudflare AI Crawl Control (formerly AI Audit), Ahrefs Bot Analytics, LLM Pulse Agent Analytics, OtterlyAI pricing, Profound Agent Analytics, Scrunch Agent Traffic and Screaming Frog Log File Analyser. Most of these vendors also sell AI answer tracking; the table covers only their crawler analytics.

Which agent analytics tool to use

Our recommendation for most sites: start with the logs you already have. Filter them on the agent names, verify the IPs, and fix any 403s served to the search crawlers you want. That costs nothing, and it settles the question that matters first, whether AI search can read you at all. Then:

  • If your site is behind Cloudflare, open AI Crawl Control. Cloudflare says it works automatically on all plans, and it lets you allow or block each crawler from the same place you see it.
  • If you already pay for Ahrefs, try Bot Analytics while it is free in beta.
  • If you want crawler data beside AI answer tracking in one product, LLM Pulse, OtterlyAI, Profound and Scrunch each offer both. Check which plan includes the crawler feature: LLM Pulse lists it on Scale and above, and OtterlyAI on Standard and above.
  • If you have no CDN integration and no budget, export a day of logs and open it in Screaming Frog's free tier.

What agent analytics cannot tell you

  • A visit is not a citation. A ChatGPT-User request shows that ChatGPT opened your page, not what it said. OpenAI notes that ChatGPT-User "is not used to determine whether content may appear in Search".
  • No visit is not proof of absence. An assistant can name you on the strength of another site's page, or of material its model was trained on, without requesting anything from you.
  • Google's AI features in Search are better measured in Search Console. Its generative AI performance report includes impressions for AI Overviews and AI Mode, by page, country and device (Google).
  • More requests are not more visibility. A crawl tells you a page was read, not that an answer used it. OpenAI adds that if a site allows both GPTBot and OAI-SearchBot, it may use the results from one crawl for both purposes.
  • User agents can be faked. Count only verified requests. Profound says its product "Differentiates real AI bots from spoofed crawlers", and LLM Pulse says spoofed user agents "are not over-counted"; ask any vendor how it does it.

Where TopCited fits

TopCited is a GEO platform with SEO included, and it does not do agent analytics: it does not read your logs or count crawler visits. It covers the step after it.

  • Content monitoring runs your query sets on a schedule you set (cron, with timezone support), and records brand mentions, gap analysis and share of voice. That shows whether the pages AI systems read are turning into answers that name you.
  • The CORE optimizer rewrites your existing content and scores it before and after, including keyword coverage, so a page AI systems read but do not cite gets improved rather than reported.
  • A GEO audit is available through the form on our site if you want to see where you stand first.

Use agent analytics to confirm that AI systems can read your site, and answer monitoring to learn whether they say your name. For the case for monitoring AI answers at all, see why use AI search monitoring tools.

FAQ

Frequently asked questions

Agent analytics is the measurement of AI agents visiting your website: training and search crawlers such as GPTBot, ClaudeBot and PerplexityBot, fetchers such as ChatGPT-User that open a page while an assistant answers a question, and agent-driven browsers such as Claude-Agent and Google-Agent. It is read from server, CDN or host logs. The same term also names performance tracking in contact centres and the measurement of AI agents a company builds itself.

Google says traffic from known bots and spiders is automatically excluded from Google Analytics properties, and that you cannot turn the exclusion off or see how much was excluded. To see AI crawlers, use the request logs your server, CDN or host keeps.

Decide per job. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, while disallowing GPTBot signals that content should not be used for training, and the two settings are independent. Perplexity says PerplexityBot is not used to crawl content for AI foundation models. Google says Google-Extended does not affect a site's inclusion in Google Search. Allow the search crawlers of the assistants you want to appear in, and decide about training crawlers separately.

Check the request's IP address against the list its operator publishes. OpenAI, Anthropic and Perplexity each publish JSON lists, and Google publishes IP ranges and also supports a reverse and forward DNS check. Anyone can send a crawler's user agent string, so do not count a request you have not verified.

No. It means a user's question led ChatGPT to open the page. OpenAI says ChatGPT-User is not used to determine whether content may appear in Search, and the visit does not show what the answer said. Pair it with prompt monitoring to see what was said.

They identify AI agents that browse the web to complete tasks for a user. Anthropic says Claude-Agent requests come from a real browser on its infrastructure, carry a Claude-Agent token in the browser's user agent, and come from IP addresses it publishes at claude.com/.well-known/sail. Google says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request, from IP ranges in its user-triggered-agents list.

Cloudflare says AI Crawl Control works on all Cloudflare plans, Screaming Frog's Log File Analyser is free for 1,000 log events, and Ahrefs Bot Analytics is free during its beta on paid Ahrefs plans. Your own server logs, filtered on the agent names and verified against the published IP lists, cost nothing.

No. TopCited does not read server logs or report crawler visits. Its content monitoring runs your query sets on a schedule and records brand mentions, gap analysis and share of voice, and its CORE optimizer rewrites existing content with before and after scores.