robots.txt Only Works on the Crawlers That Were Never Your Problem

Here is the number that ended the argument about whether my traffic was real.

Cloudflare, one month, on a music site with about 2.6 million catalogue pages:

Desktop    1,980,000
Mobile         1,050

Real human traffic runs somewhere between half and two thirds mobile. A ratio of 1900 to 1 is not a demographic. It is a fleet.

The fleet did not care about robots.txt

I checked the obvious thing first. The three paths taking the most abuse were already disallowed. They had been for months. The bots hitting them were not confused about the rules, they had never read them.

What they actually were, once I measured instead of guessing:

• 2,215 distinct IPs in the last 5,000 requests, about 2.3 requests each, from roughly 180 countries. A residential proxy pool.
• 99.2% of URLs requested exactly once. Nobody browses like that. That is an enumeration.
• Real headless Chrome. They executed JavaScript, which meant they also fired every client-side call a real visit makes, including the ones that write, and passed the JS challenge without noticing it was there.
• A cache hit rate of 5.75%, because every request was for a page nobody had asked for before.

It was not an attack. Zero login attempts, zero probing for .env or .git or wp-admin, one request in the whole sample that returned a 401. They did not want to break anything. They wanted the catalogue.

You cannot write a line in a text file that stops that. There is no user agent to name. The user agent says Chrome, because it is Chrome.

What I tried, in order of how much it helped

robots.txt did nothing, as covered above. The file is a request, and requests only work on people who read them.

Bot Fight Mode did almost nothing. Twenty requests a second became 18.4. Cloudflare's own bot detection cannot do much about a real browser driven by a real automation stack from a residential address, because there is very little left to detect.

A WAF managed challenge on the worst paths hit its target and missed the point. I put the five heaviest non-catalogue routes behind a challenge and they went to zero. Total traffic did not move. The fleet just went and read the catalogue instead. Path-level defence does not remove pressure, it relocates it.

Turning off prefetch was the actual fix, and it was embarrassing. Forty percent of my traffic was my own framework: of 28,513 requests, 11,564 were Next.js <Link> prefetch. It compounded against me specifically because the scrapers ran a real browser, so every page they read fired a prefetch fanout for every link on it. My performance optimisation was multiplying their load for them.

Making prefetch opt-in instead of default roughly halved the request rate, and took the load on the box down with it. That is the whole story of that month: the largest single lever on bot traffic was a setting in my own app.

There is a footnote to this that I liked less. My view_count column fed the sitemap gate, so the question of which pages Google was shown had quietly been handed to whoever was crawling hardest. The most viewed artist on the site had 29 views. I zeroed the column and 110,660 rows of daily view data with it.

So the file is useless. Except it is the only thing that works.

The crawlers that ignored robots.txt are the ones I could not identify, could not negotiate with, and could not block by name. The crawlers that do read robots.txt are the ones that announce themselves: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, CCBot. They send a real product token. They fetch the file. They obey it.

Which means the file has exactly one audience left, and it is the AI companies. Everything you write in there is a message to them and to nobody else. The scrapers were never going to read it, and now they never will.

That makes the accuracy of those lines matter more, not less. And this is where almost every site I have looked at gets it wrong, because they treat "AI" as a single switch.

It is three switches

The named agents fall into three groups, and blocking one costs something completely different from blocking another.

Training crawlers. GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent. They collect pages to train on. Blocking them costs you nothing you can measure this year or next, unless you believe being in the weights is worth something. This is the easy one, and it is the one everybody argues about.

Search indexes. OAI-SearchBot, PerplexityBot, Claude-SearchBot. These build the corpus the assistants answer from. Blocking one removes you from its results the same way blocking Googlebot removes you from Google. People block these by accident constantly, usually by writing a rule for "OpenAI" and catching two bots when they meant one.

Fetchers acting for a person. ChatGPT-User, Claude-User, Perplexity-User. These fire when somebody pastes your URL into an assistant and asks it to read the page. Blocking one means a reader who deliberately went looking for your content gets an error instead of it.

That third group is the one worth checking today. Almost nobody blocks it on purpose. It gets caught by a blanket Disallow: / under a wildcard, or by a rule aimed at a training crawler that happens to match a sibling token.

Google-Extended is not what most people think it is

The single most common mistake I find. Google-Extended controls whether your content is used for Gemini training and grounding. It has no effect on Google Search ranking or indexing. None.

People block it believing they are opting out of AI answers appearing in search results. They are not. They are opting out of Gemini while leaving Search exactly as it was, which may well be what they want, but almost none of them chose it deliberately.

Applebot-Extended has the same shape: it governs Apple's generative models, while Applebot, which feeds Siri and Spotlight, is a separate token and unaffected.

The rule that blocked humans and exempted bots

One more, because it is the same mistake one layer down and it cost me real users.

I once had a Cloudflare rate limit on a catalogue path: fifty requests in ten seconds, action block. It was built as a cost shield against exactly the fleet described above. The expression was:

(starts_with(http.request.uri.path, "/catalogue/")) and not cf.client.bot

Read that carefully. cf.client.bot matches verified bots. So not cf.client.bot means "everyone except verified bots", which is to say: only humans. The rule exempted Googlebot and caught readers.

It got worse from there. The whole catalogue sits under that one prefix, and Next.js was prefetching every visible link, so opening a single page fired well over a dozen requests beneath it. Fifty was three or four clicks away. The action was a hard block rather than a challenge, so a false positive hit a Cloudflare-branded error page rather than a puzzle.

I found it by browsing my own site and getting Error 1015.

The rule protected nothing, because the scrapers were never verified bots either, and a fake user agent does not become one. It only ever punished the people I built the site for.

The lesson generalises past Cloudflare: a rule written to catch bots, that does not name a bot, catches people. robots.txt at least fails safe, because the crawlers that ignore it were never going to obey anything. A block rule fails the other way.

What to actually do

Read your own file, agent by agent, and decide each of the three groups separately. That is tedious by hand because the matching rules are less obvious than they look: the most specific user-agent group wins even if it is empty, the longest matching path rule wins rather than the first one written, and allow beats disallow on a tie.

So I built the thing that does it: an AI crawler robots.txt checker. Give it a URL and it reports what your file says to fifteen named agents, grouped by what blocking each one actually costs, and it flags the case where you are turning away a fetcher that a human explicitly sent.

And once you are sure the file says what you meant, there is a second question, which is whether anything ever reads it. That one has its own answer, and its own post.