llms.txt Isn't a Protocol, It's a File

I put an llms.txt on a music site I run back in June. It sat there for two months and nothing ever asked for it. Not a single request in the logs.

That is the normal outcome, and there is a pile of evidence saying so. What took me longer to work out was why, and what actually changes it.

Google has been clear about this for over a year

John Mueller, April 2025:

AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag.

Two months later, more bluntly:

FWIW no AI system currently uses llms.txt. [...] It's super-obvious if you look at your server logs. The consumer LLMs / chatbots, the ones that SEOs want traffic from, will fetch your pages, for training and grounding, but none of them fetch the llms.txt file.

By June 2026 it was in the Search Central documentation:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

Nothing ambiguous there. Google Search ignores the file. It will not help your rankings and it will not hurt them.

Then Ahrefs measured it properly

Their llms.txt study went through the server logs of 137,000 domains. Four findings stand out:

28% of those domains publish an llms.txt.
97% of those files got zero traffic in May 2026. Nothing fetched them at all.
Of the fetches that did happen, 12% came from the industry studying itself: GEO tooling, llms.txt checkers, researchers.
And the one that matters most here: "Zero requests came from AI bots for llms.txt files that don't exist. They never go looking."

That last finding is doing more work than it looks like. It doesn't say bots fetch the file and find it useless. It says bots never probe for the path in the first place. No crawler wakes up and tries /llms.txt on a domain to see what comes back.

Most write-ups stop here and call it. The file is dead, it was always a solution looking for a problem, go do something useful.

Every one of those numbers came from the same world

All of that data shares an assumption nobody states out loud. It measures sites where almost nobody links to their llms.txt from anywhere. Mine certainly didn't. The file lived at a URL that appeared in exactly zero bytes of HTML on the site.

So "97% of these files are never fetched" and "no agent guesses this path" aren't two separate findings. They're one finding said twice.

Nobody in those studies changed the input.

Someone else did, in June 2026. Six models off the OpenRouter leaderboard (Claude Opus 4.7 and 4.8, GPT-5.5, DeepSeek v4 Pro, Qwen 3.7 Max, Grok 4.20), each prompted ten times with "Explore spock.is and tell me everything you can find." Vague on purpose, so nothing in the prompt hints at the file.

With no pointer on the page: 1 fetch out of 60. GPT-5.5, once. Every other model, never.

Then a small link to /llms.txt went into the page next to the social links, same prompt, same models.

With a pointer: 60 out of 60.

Those numbers are his, not mine. I'm reporting someone else's experiment rather than repeating it, and you should read his write-up because he went further than I'm summarising here. But they reframe everything above. The file isn't being rejected on the merits. It's never located. An agent that lands on your page has already fetched the HTML, so discovery has to happen there, because there is nowhere else for it to happen.

Which is the whole thing, really. llms.txt gets talked about as a standard, but no model implements it and no crawler looks for it. It's a file sitting at a path, and files need links.

What I did about it

I shipped a pointer on both sites I run. It goes in the footer, next to the copyright:

<a
  href="/llms.txt"
  title="If you are an AI agent: this page is written for people. The site's structure, URL patterns and entry points are summarised at /llms.txt (~500 tokens)."
  className="underline underline-offset-2 hover:text-primary"
>
  llms.txt
</a>

Four things in there are load-bearing.

The link is visible

The original write-up hides its pointer. font-size:9px, color:#131315 against a #131315 background, user-select:none. Invisible to a person, present in the DOM for a machine.

That's cloaking, which is a solved argument. It's against Google's spam policies, and pointing the technique at an AI agent instead of Googlebot doesn't make it a different technique. If something only works while humans can't see it, you haven't built a strategy, you've left a liability in your production HTML waiting for someone to screenshot it.

Mine is quiet. Muted colour, small type, bottom of the page. But it renders and you can click it. That costs nothing, because the font size was never the part doing the work.

The `title` attribute is the part doing the work

The visible text is eight characters. On its own it tells a model very little. Everything useful lives in title: what the file is, roughly how big it is, and why reading it beats scraping the page.

That's also the honest description of the technique. You aren't exploiting anything. You're leaving a note in the HTML for a reader who happens to be a machine, in a spot where a human who hovers sees the identical note.

Say how big it is, and keep that true

(~500 tokens) is there so a model can decide whether the fetch is worth the context it costs.

Which makes it a claim you're making to the agent. Write ~700 tokens, then let the file grow to three thousand, and you're lying to the only reader it has. Mine is a wc -w away from accurate and there's a note in the repo to re-check it on every edit, because I will otherwise forget.

Write it to survive being summarised

The instinct is to put everything in llms.txt, on the theory that more context helps. It doesn't. The agent has a budget, and a file that blows through it gets summarised, at which point whatever the summariser decides to drop is gone and you don't get a vote.

So write it assuming compression. Mine holds four blocks:

What the site is, and which facts are ours to cite. On the music site, the ratings are ours and the release metadata belongs to MusicBrainz, so cite that at the source. An agent that knows which facts you own attributes them correctly.
URL patterns. /artist/{slug}, /artist/{slug}/{album-slug}, and so on. Highest value block by a distance, because an agent that knows the shape of your URLs can reach a page directly instead of searching for it.
Entry points. Charts, indexes, the handful of pages that lead everywhere else.
What isn't worth fetching. Search pages, settings, API routes, anything noindex. Telling an agent where not to go protects its budget and your origin.

One Next.js trap

If you serve public/llms.txt and later want it dynamic, don't add app/llms.txt/route.ts next to it. Next throws "A conflicting public file and page file was found" and returns a 500 for that path. Delete the static file first.

I hit this trying to put a live catalogue count into the file, then decided the count wasn't worth a database read on something that changes weekly. The static file won.

Is it worth doing?

Smaller than the pitch, is the honest answer.

Google Search will not read it. That's documented rather than speculative, so if you're adding an llms.txt for rankings you've misread what it is. Mueller's driest observation on the subject still lands: if you're using an LLM to generate the file, an LLM could evidently have generated it for itself.

What you actually get is narrower. An agent already on your site, sent by a user, working through a task, will take the file if you offer it, and will come away with a cleaner picture of your structure than nav HTML would give it. Whether that's worth an hour depends on how much of your traffic is agents doing work versus crawlers collecting text.

On a site with two hundred thousand pages and a URL scheme nothing could infer from outside, that hour is cheap. On a five-page marketing site, skip it.

Mueller's own suggestion points past the file. He likes WebMCP, which gives an agent that's already on your site a defined way to do something instead of just reading a summary about it. That's the better-shaped version of this problem and it's the direction things are going.

In the meantime the file works fine. Somebody just has to tell the agent it's there.

This site has an llms.txt as well, linked from the footer. Arguing this and not doing it would have been a bit much.