We Logged 30 Days of AI Bot Traffic. 71% of robots.txt Hits Couldn’t Be Verified

Our /robots.txt file took 13,263 bot-identified hits in 30 days. We could verify only 3,867 as coming from the operators those requests claimed to represent. If an AI visibility report had counted every hit, it would have called a lot of unverified traffic progress.

I have spent enough late nights mining search terms to distrust any chart that goes up and to the right without explaining how it counted. A user-agent string is easy to claim. We counted a request as verified only when the user agent and source IP matched the operator’s published range, the check described in this bot-traffic detection overview. By that standard, 71% of the bot-identified hits to our robots.txt could not be verified. Unverified does not automatically mean spoofed or malicious. It means I cannot use the request as evidence that the named bot visited.

The page-level numbers made the distinction more useful. Three category hubs drew roughly 680 verified fetches combined and no user-triggered page fetches in our log. One dated changelog post, /post/google-ads-updates-2026, drew 611 verified fetches, including 377 live-answer fetches from ChatGPT-User and Claude-User. Those requests show an assistant pulling the page in response to a user, not that the page appeared in the final answer. Even with that limit, the contrast is hard to miss: being crawled and being fetched at answer time are different jobs.

First, separate claimed bots from verified ones

We logged requests at the edge, not in GA4. A hit here is a server request carrying a bot user agent. Verified means its source IP matched the published range for that operator. OpenAI publishes bot information and IP lists; the method applies the corresponding operator check to each bot we counted. Requests we could not match stayed unverified. That group can include spoofs, stale ranges and proxies we could not confirm.

A live-answer fetch is narrower still: a page pull by ChatGPT-User or Claude-User when a user asks for something, rather than a background crawl. OpenAI describes ChatGPT-User as user-initiated. We can identify those requests in the log. We cannot identify a citation from an edge request alone. The assistant may fetch a page without quoting it, naming it or sending a visitor back.

The source for the tables below is our own edge log: one site, groasatelier.com, over 30 days. The path list available for this analysis is truncated. These are observations about our site, not a benchmark for everyone else’s.

I used to tell clients to look at the bot chart in their CDN and feel good when the line moved. I was wrong. The line can be busy without telling you who visited or why. Analytics tags miss crawlers altogether, while AI-referred visits can be difficult to recognise in analytics: in one dataset, GA4 classified 70.6% of 446,405 AI-referred visits as Direct. The edge gives us the requests those reports miss, provided we verify the identity attached to each one.

The first decision is simple: do not call a claimed bot hit a verified visit. Our robots.txt count fell from 13,263 to 3,867 when we applied that test.

The homepage drew the most live-answer fetches

Our homepage recorded 822 live-answer fetches during the period. 773 came from ChatGPT-User; most of the remainder came from Claude-User. That fits the homepage’s job. It is the broadest page about who groas is and what it does, so it is a plausible page for an assistant to open during a brand question. ChatGPT-User fetches pages in response to users, rather than following a background crawl schedule.

I would not call those 822 citations. The log does not show what the assistant said after the fetch. It does tell us which page was repeatedly available to be read at that moment. If your homepage makes readers work to find a plain description of the business, an assistant opening it faces the same problem. Fix the opening copy before building another page that repeats it. That matters especially when an assistant relies on limited page information: one account of ChatGPT citation behaviour describes results containing a title, URL and roughly 200 characters.

A dated changelog drew fetches the hubs did not

The sharper split was between page types. Our post at /post/google-ads-updates-2026 recorded 651 hits, 611 verified fetches and 377 live-answer fetches. Three category hubs drew about 680 verified fetches combined and no live-answer fetches in the available path list. Same site, same period. Very different reasons to visit.

That last column is a request ratio, not a citation conversion rate. A live-answer fetch may not produce a visible mention. I still find the comparison useful: the hubs attracted verified bots, but the dated post attracted user-triggered page pulls.

The post is narrow and dated, with named changes, direct answer paragraphs and lists. The hubs are broad by design; they route a reader to other pages. That gives me a sensible page hypothesis, not proof of what caused every request. Outside our logs, one cited-URL analysis reported more list sections on cited pages and more citations for fast pages. I would not transplant its results onto our site. I would use our own split to decide what to test next: write a page that can answer a specific question in one fetch, rather than expecting a directory to do that work.

The blog index shows why crawl volume is not enough

Our blog index drew 213 verified fetches and 4 live-answer fetches: about 2% of verified fetches. The changelog post’s corresponding ratio was about 62%. Both could be fetched. Only one repeatedly drew user-triggered pulls in this record.

That is not a verdict against indexes. An index helps people and crawlers find articles. It is simply the wrong page to grade as though it were a stand-alone reference answer. Nor does a verified fetch guarantee that an assistant will use what it finds. A fixed set of buyer prompts, rerun across assistants and scored for share of voice and citation rate, addresses the next question: did the brand actually appear?

High fetch volume has never meant proportionate traffic back, either. In Cloudflare’s July 2025 figures, Google Search fetched 5.4 pages per human sent, versus about 1,091 for GPTBot and 38,065 for Anthropic. Those figures describe a different dataset, but they reinforce the measurement problem. A crawler request is a request. It is not a visitor, an answer or revenue. Keep each stage in its own column.

The assistants did not follow the same path

All 1,222 robots.txt checks associated with live-answer activity in our record came from Claude-User. On the homepage, 773 of 822 live-answer fetches came from ChatGPT-User. The log shows different request patterns; it does not, by itself, explain every reason behind them.

The operators’ stated rules provide useful context. Anthropic says its Claude bots honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, which fetches at a user’s request. OpenAI also distinguishes GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user-initiated visits. Its training and search controls can be set independently.

That is why I would not treat a single robots.txt setting as an on/off switch for every kind of AI appearance. Choose your training and search rules deliberately, then check the requests that actually arrive. A rules file states an instruction; an edge log shows what reached the site. Neither one, on its own, shows what an assistant ultimately told a user.

What I would measure and change next

I would start with a short report, not a bot-traffic trophy chart:

1. Verified fetches by path. Keep the full user-agent string, source IP, path and timestamp; match claimed operators against their published IP ranges.
2. User-triggered fetches by path. Tag ChatGPT-User and Claude-User page pulls separately from background crawls. Do not label those requests citations.
3. Observed answers. Rerun a fixed set of buyer prompts across assistants and score whether the brand and pages appear, using the share-of-voice and citation-rate approach described here.

The first two come from edge logs. The third requires looking at answers. That boundary matters more than another decimal place in a crawler chart.

I would also check access at both the robots.txt and server layers. A correct robots.txt does not ensure a CDN or WAF lets a crawler through. One reported sample found major LLM crawlers blocked at the CDN layer on about 27% of B2B SaaS and ecommerce sites despite a correct robots.txt. Test responses by user agent, then confirm what the access log records. If you intend to allow search crawlers, check OAI-SearchBot, ClaudeBot and PerplexityBot in both robots.txt and your bot rules. Do not assume one setting speaks for the other.

Then work on pages, not just permissions. Our changelog is a useful model to test: a date, a narrow subject, a direct answer near the top and facts laid out in a short list. Keep category hubs for browsing. Just stop asking them to perform like stand-alone answers when your own request data says they do not. Make the page worth fetching at answer time, then check whether answers actually use it.

You can do the first pass in-house: pull 30 days of edge requests, verify identities, group paths and compare verified crawls with user-triggered fetches. The harder work is repeating the buyer-prompt check, reviewing what appears and rewriting pages that get crawled but never named. Our logs tell us where to look. They do not let us skip that second measurement.

Most tools could have shown us 13,263 robots.txt hits and called it AI visibility. Verification left 3,867. The page breakdown then showed hundreds of verified hub fetches without a live-answer fetch, beside a changelog repeatedly pulled by ChatGPT-User and Claude-User. I keep the order straight now: claimed, verified, user-triggered, then observed answer. If a report skips a step, I want to know why.

The second picture captures the page decision. The dated changelog drew 377 live-answer fetches; the three category hubs drew none despite roughly 680 verified fetches. That does not prove the changelog was quoted 377 times. It gives us a better candidate to test in actual answers than a hub built to send readers elsewhere.

A one-time edge pull tells you where requests went. Staying visible calls for rerunning buyer prompts across ChatGPT, Claude, Perplexity and AI Overviews, scoring share of voice and citation rate, and revising pages against what appears. That continuing work is what groas runs through its SEO and AI Search sprint, rather than treating a monthly bot-hit chart as the result. For this site and this month, the decision is clear: verify the bots, build pages assistants can use, and count appearances in answers separately from hits.