Unreviewed Agent Code Put My Database on a Public CDN

curl -r 0-15 https://<the-public-cdn-domain>/backups/<a-dated-filename>.dump

HTTP/2 206
5047 4d50    →  "PGDMP"

That is a PostgreSQL custom-format dump answering an unauthenticated range request. Sixteen of them, going back two weeks. User emails, password hashes, Spotify and Last.fm refresh tokens, private lists. All of it, to anyone who typed the URL.

The window was about two days. I found it with one curl, while checking something else entirely.

I did not write the script that did this, and that is the point of the post rather than an excuse. An agent wrote it, during a migration, at a speed where reading every line felt like the slow path. It did exactly what I asked. What I asked was incomplete, and nothing between the asking and production was going to catch that.

How it got there

There was no bad decision. There were two reasonable ones that never met.

During the migration, the backup script was generated onto the new box. It needed somewhere to put a nightly dump, and rclone was already configured for one R2 bucket, so it used that. Least resistance, and at the time it looked like reuse rather than a choice.

The other half of the problem lived somewhere else entirely: that bucket is served publicly on its own domain, because it holds album art for a music site. That fact is recorded in the Cloudflare dashboard and in an env file. It is not recorded anywhere near the backup script.

Two configurations, both correct in isolation, and nothing in the system where they would ever be compared.

Before reusing a credential or a bucket, the question is not whether you have access. It is what else is already attached to that resource. "I already have access" is how you end up writing secrets into a CDN.

Directory listing was off, which meant nothing

The directory itself returned 404, and for about ten seconds that felt like mitigation.

The filenames carried a date and a time. The date increments in one direction. The hour was one of two values. Enumerating the entire set is a loop, not an attack.

Obscurity is not worthless, and I will come back to that, but a predictable naming scheme converts "listing is disabled" into decoration.

Fixing it, and what the fix could not be

The obvious move is a separate private bucket. I could not make one: the R2 token on that box was scoped to that one bucket and nothing else, so creating another returned 403. Fixing the token properly meant a dashboard round trip, and the dumps were public while I did it.

So I moved the objects instead. All sixteen, server-side, to a random 24-byte prefix inside the same bucket. No data movement, no download, no window where they existed in two places.

I considered a WAF rule blocking /backups/ and rejected it. Moving the object is transport-independent. A rule closes one path; the custom domain, the r2.dev URL and anything else pointed at that bucket all stop working at once when the object is not where the path says it is.

Then the part I nearly missed. After the move, one URL still returned 206. The object was gone from the origin and alive in Cloudflare's edge cache. Moving a file does not unpublish it. A file-level purge did, and I verified all five known URLs returned 404 while the album art next to them still returned 200.

Last, the script now reads the prefix from R2_BACKUP_PREFIX in the environment and exits 1 if it is missing, rather than falling back to a default. A silent fallback to a predictable path is the same bug wearing a different hat, and I had already stashed the prefix in a throwaway file on the box before noticing that was one too.

Did anything read them

Cloudflare Analytics shows no requests to /backups/ other than mine. The window was about two days, the URL was never published anywhere, and the names, while enumerable, were never enumerated.

On that basis I did not rotate tokens, did not invalidate anyone's Spotify or Last.fm connection, and did not force a password reset.

That is a judgement call and I want to be honest about its edges. Cloudflare's free tier samples analytics and its retention window is short, so "no requests" is strong evidence rather than proof. If the site had had more users, or the window had been longer, or the data had included anything I could not have re-derived, I would have paid the cost and rotated. I can defend the decision. I would not call it obviously correct.

The fix I actually wanted, in the end

The first move stopped the bleeding without being right. The dumps were no longer at a guessable path, but they were still inside a bucket with a public domain attached, and the only thing protecting them was that nobody knew the prefix. That is obscurity, and I wrote at the time that it was not the fix.

Later the same night it got the real one: a separate bucket, no domain in front of it, its own token scoped to nothing else. The dumps live there now and there is no public route to them at all, rather than a public route nobody has guessed.

I am keeping the intermediate step in this post because the ordering is the useful part. The move-and-purge was the right first action even though it was not the answer, and the answer took another hour I did not have while the site was leaking.

Backups do not live in the infrastructure that serves the data. If a bucket has a public domain, nothing private can be in that bucket, regardless of the path.

The actual failure was upstream of the script

Here is what bothers me most, a couple of weeks later.

The backup chain had been discussed for hours that week. Does the backup run. Did it run last night. How large is the dump. Can we restore from it, and how long does the restore take. All good questions and all of them answered.

Nobody asked where it went. And everything the answer needed was already visible: the bucket name was in the script, the public domain was in an env file, and the two were maybe forty characters apart in two files I had both touched. One curl confirmed it, once the question existed.

The question existed only because I eventually read the script.

That is the uncomfortable version of this incident. When I review what an agent writes, I catch things like this. Not sometimes: the mismatch is obvious the moment two lines are in front of the same pair of eyes. The failure is not that the model wrote something insecure. It is that I had stopped putting the lines in front of my eyes, because the whole appeal of working this way is that you do not have to.

Migrations are the worst case for it. The output is scripts and unit files rather than application code, and those do not land in the repository. That night, five separate pieces of configuration on that box existed nowhere else: the systemd unit, a Postgres config fragment, a cache pruning script, a disk alarm, and the backup script. Every one of them was generated, applied and left there.

Nobody reviewed a single one, because review is a thing that happens to pull requests and these were things on a machine. Four of them cost me a night of debugging. One published the database.

I do not have a satisfying process fix, and I distrust posts that end with one. What I changed is small enough that I might actually keep doing it: anything an agent writes that touches data or credentials gets read before it runs, even when it is a shell script, and especially when it is going somewhere a diff will never see.

The narrower version, which is the one that would have saved me here: when something writes data somewhere, say out loud where it goes, then look at what else is pointed at that place. Thirty seconds. It was the only step missing.