Two Million Pages, and 12,560 Worth Indexing
A catalogue ingest went from 578,000 track pages to two million in a week. Then I counted: 99.99% of them had never been opened by anyone. Here is what that cost, and the order you have to remove them in.
The counter on the homepage of Riffiter, the music site I run, says 2,265,165 songs. I put it there because it looks like an achievement.
For most of this year it was closer to a liability. Two million of those song pages had never been opened by a human being, and Google was crawling every one of them.
The week the catalogue grew 3.5x
On 9 June the catalogue held about 578,000 tracks. On 16 June it held 2,038,269.
Nothing dramatic happened in between. A catalogue growth job and a tag gap-fill job ran the way they were written to run, night after night, and the number went up. I wrote them, they worked, and I stopped looking at the output, because the output was rows in a table and the table was getting bigger, which was the entire point of the job.
Add 304,434 albums, 9,337 artists and 8,725 concert pages, and the site was roughly 2.36 million public URLs. The plan I had written for myself said a hundred thousand.
Then I counted what was actually on them
This is the part I recommend doing early, because it takes one query and it ends several arguments at once.
Of those 2,038,269 track pages, 99.88% had no lyrics. 99.94% had no rating from anyone. And 2,038,015 of them, which is 99.99%, had a view count of exactly zero.
Not low traffic. Zero. Two million pages that no visitor, signed in or not, had ever opened.
I had spent two weeks before this blaming a caching bug and then a view-counter bug for a rising bill. Both theories were wrong, and both were more comfortable than the real answer, which was that the pages existed and something was dutifully fetching all of them.
The gate that let them through was not a gate
Album pages did have an index condition. It was cover_url is not null.
That sounds like quality control if you say it quickly. Cover art came free with the bulk metadata import, so every album had one. The condition passed on essentially the whole catalogue. A filter that nothing fails is not a filter, it is a comment.
What that produced was 337,745 albums in the sitemap whose content was a cover image and a tracklist copied from an upstream source. Google's spam policy on scaled content abuse describes that shape almost exactly. The consequence is not a penalty on the thin page. It is the crawl budget for the whole domain getting quieter.
There was a second cause running alongside it, and it was stupider. A month after launch the site had no search traffic at all, and the sitemap had never been submitted to Search Console. The homepage had been crawled once, on 20 June, and Google had not come back since.
Both things were true, and the order mattered. Fixing only the submission would have handed Google 337,745 stubs and asked it to form an opinion.
Rebuilding the gate out of real signals
The replacement condition asks whether anything has actually happened to the record: has a person rated it, has anyone viewed the page, or is it popular enough upstream to be worth a look on its own.
337,745 albums in the sitemap became 12,560.
Removing 96% of your own indexable pages is not a sentence I enjoyed writing in the commit message. I still think it was the cheapest hour I have spent on this project.
The part that makes it bearable is that promotion is automatic. When a page picks up a rating or a visitor, it clears the condition, and the next revalidation ships it as indexable with its structured data attached. Nothing has to be re-run by hand and nothing is deleted. The catalogue stayed exactly as large as it was.
Track pages, all of them, unconditionally
Track pages got a blunter treatment: index: false, follow: true, with no condition at all.
The earlier version indexed a track if it had lyrics, which felt principled and moved almost nothing, since 99.88% had none. Dropping the condition entirely gave up very little. Individual song titles are long-tail queries with close to no volume, and everything worth reading on a track page also appears on the album page above it.
follow: true is the part worth keeping. The link graph still works, so anything valuable
sitting behind a track page is still reachable. Only the indexing stops.
I also removed the MusicRecording and BreadcrumbList JSON-LD from those pages. Search
engines ignore structured data on a page they have been told not to index, so it was two
million copies of markup that existed to be read by nobody.
The order matters more than the decision
The obvious next move is to put a Disallow in robots.txt and stop the crawling directly. That
is backwards, and it is the most common way this goes wrong.
If the crawler cannot fetch the page, it cannot see the noindex tag. The pages stay in the index, and they stay there for much longer, because you have removed the mechanism by which Google would have learned to drop them.
So the sequence is: ship noindex, wait until the Search Console pages report actually moves those URLs into "Excluded by noindex", and only then close the crawl. On a site with low crawl budget that wait is measured in weeks, not days, which is annoying and not optional.
A noindex tag has to be crawled to do its job. Blocking the crawl is not a stronger version of noindex. It is the thing that prevents it from working.
The other check that belongs in the same commit: confirm the route is not listed in any sitemap. A URL that is noindexed and submitted at the same time is a contradiction you are sending on purpose, and Search Console will tell you about it for months.
The cheapest fix turned out to be an internal link
The change that moved the bill most was one attribute on a link.
Thin track links inside album pages got rel="nofollow". Those pages are noindexed and absent
from every sitemap, so an internal link was the only way a crawler ever found them. Add nofollow
and the crawler stops walking into them, which means the page is never requested, which means it
is never rendered, which means the render is never paid for.
Every other fix in this post reduced what a crawl costs. This one reduced whether the crawl happens. When the link earns real content, it goes back to being followed on its own.
What I would actually change
Not the ingest. Having the data is worth something, both for the product and for the occasional page that turns out to matter, and the whole cost problem is solvable without deleting a single row.
What I would change is the assumption underneath it. A job that writes rows to a database is also, in a framework like this, a job that creates public URLs, and it has no idea that this is what it is doing. Every row it adds is a page some crawler will eventually request. Nothing in the job's design or its logs says so. I found out from an invoice, which I have written about separately, and by then it was two million pages deep.
If you are running an ingest that produces pages, put the index condition next to the ingest, not next to the page template. By the time it is a rendering decision, the pages already exist and something is already crawling them.
The counter on Riffiter still says 2,265,165. I still think it is worth having. It just is not worth indexing, and those turned out to be two very different questions.