Every Defect Was a Believable Number

A price pipeline I run was pulling quotes from Yahoo in batches of twenty, sequentially. The catalogue had grown to 11,206 symbols, which works out to 561 chained batches on every fifteen-minute tick. It was not going to survive the next thousand rows.

Alpaca's snapshot endpoint takes the same 11,164 symbols in six requests. That is the whole reason the migration happened, and it is the number that makes "we moved to Alpaca" a decision rather than a preference.

The migration landed clean. No incident, no rollback, no bad price in front of a user. Agents wrote most of it, including the parts I would have got wrong, and I want to be precise about what actually made that true, because it was not the code.

The part that went well is the boring part

The plan named two alternatives and killed each in a line. Point the existing scheduler at the new function: rejected, because that function scans the table directly instead of going through the view that knows about market hours and chronic non-responders, so one change would have quietly cancelled three earlier decisions. Backfill the old symbol column for the whole catalogue: rejected, because that puts every row back on the twenty-at-a-time path the migration exists to escape.

The chosen version changed the price job and the staleness alarm in the same migration. That detail is the one I would keep above all the others. The alarm's job is to shout when a price source stops updating. If the alarm keeps testing the old definition of "priceable" while the job uses the new one, then the day the new branch silently stops, nothing shouts. An alarm that tests a different question than the one you changed is worse than no alarm, because you believe it.

None of that is exotic. It is the kind of care that gets skipped at 1am, and it did not get skipped, and the reason is that it was written down as a plan before anything was written as code.

Then every bug came back as a number

Four defects survived into the first working version. Not one of them raised an exception in the shape that matters, and all four are inherent to how a market data API has to work rather than anything wrong with this one.

The API has no index product, so it sells you a stock instead

The catalogue has index rows. DAX, ES, that family. The endpoint resolves a ticker against equities and ETFs, so it answers.

DAX came back as the Global X DAX Germany ETF at around $40. The actual index was near 24,000. ES came back as Eversource Energy, a utility.

HTTP 200. A float. A timestamp. Three orders of magnitude wrong on one, a completely different company on the other. Nothing in the response says "I could not find an index so here is something that shares its ticker", because from the endpoint's point of view nothing went wrong. It was asked for a symbol and it found a symbol.

Index rows were left on the old provider on purpose. That is a one-line decision that only exists because somebody typed DAX into the new endpoint and looked at the answer.

Everything is priced in USD, including the things that are not

Twelve of the ETFs in the catalogue trade in other currencies. Six in KRW, four in EUR, two in GBP.

Sent to the new endpoint, they would have come back with a USD figure and no complaint. Not a converted price, and not an error either. A number in the wrong unit, written into a column that does not carry a unit, rendered next to a currency symbol chosen from a different column. The filter that keeps them on the old path checks the currency column at run time rather than naming symbols, because a hardcoded list is a thing that drifts as the catalogue grows.

The free tier is a different tape with the same field name

Live prices on the free tier come from IEX. Requesting the consolidated feed returns 403. The field is called the same thing either way.

For liquid symbols the difference is invisible. For thin ones the last trade can lag, because IEX is one venue and the consolidated tape is all of them. The measurement that made this concrete: Apple on 28 July showed 52 million shares on the consolidated tape and 1.87 million on IEX. Same day, same instrument, same field name, one line of code apart.

Historical daily bars are consolidated, so the chart and the live quote are not necessarily drawn from the same universe. That is a fine trade at this stage and a paid tier fixes it. It is not fine to discover it from a support ticket.

One bad symbol takes the entire batch with it

This is the one that does throw, and its blast radius is still silent.

Send eleven thousand symbols and include one the endpoint does not recognise, and the request returns 400. The whole request. To its credit, the error message names the offender: invalid symbol: 2222. If you do not read that message, strip the symbol and retry, you lose everything in the batch.

First run: 7,034 of 11,165 symbols priced. After adding the retry: 10,970.

The 4,131 rows in between did not show up as failures anywhere a person would look. They kept their previous price and their previous timestamp, and a price from earlier today looks exactly like a price from now unless you are counting. Eight known bad tickers currently trip it, including a couple of numeric ones from exchanges that use them.

What caught all four was counting, not testing

Every one of these was found by asking the database or the API a question with a number for an answer.

How many ETFs does this provider actually cover? 105 checked, 89 priced, 8 invalid symbols, 8 accepted but returning nothing. What does this endpoint say DAX is? What is the volume difference between the two feeds on a day I can look up? How many rows did the batch actually write, against how many I sent?

None of those is a unit test, and a unit test would not have caught any of them, because in every case the code did what it was written to do. The failure was in an assumption about the world on the other side of the network, and assumptions about the world are checked by measurement.

When an integration can return a wrong value as easily as an error, the acceptance criterion is a count, not a green suite. That is the whole method, and it is the part that scales to however fast an agent writes.

Two things that were only visible because someone had explained themselves

A scheduler that reports success before the response exists. One of these jobs was being rejected on every run while the scheduler's own history table said succeeded. Those are two different events: the scheduler had queued the request successfully, which is all it claims to know. The HTTP status came back later, to nobody. If a job's health is read from the queue's log rather than from the effect the job was supposed to have, the log will keep saying yes for months. This was the fifth or sixth instance of the same class in this codebase.

A comment that carried a reason, and therefore expired loudly. The gate that decides when US symbols are worth fetching used to be 09:00 to 16:30 ET, and it carried a note saying why: our source is a delayed quote endpoint.

Moving to Alpaca made that sentence false on the day of the migration, because the new source reports the extended session. The comment is what made the gate visible as stale. 09:00-16:30 ET on its own would have stayed there looking perfectly correct, forever.

The measurement that followed: at 06:35 UTC with the market closed, twenty liquid symbols came back with post-close prints. MU at 16:59 ET, up 0.52%. TSLA 16:47, INTC 16:27, PLTR 16:02, BA 16:00. Fifteen of the twenty were still stamped 15:59, which is IEX not having after-hours pressure on them. Partial coverage, but the closed window was a certain loss, and the movement it was losing is the expensive kind, because earnings reactions happen after the bell.

If a gate's reason lives in the source it reads from, changing the source is the moment to re-read the gate. Nothing prompts you to do that except a sentence somebody left behind.

What I would tell someone doing the same swap

The provider is good, and none of the four defects above are its fault. An endpoint that resolves tickers against equities will resolve an index ticker to an equity. A free tier fed by one venue is a free tier fed by one venue. Both are documented. They only become defects when they meet a catalogue that assumed otherwise.

The agent part worked, and worked better than I would have alone, particularly on the plan: alternatives named and rejected with reasons, alarm and job changed together, filters reading from the database rather than from a list that would rot. I got a migration I would have taken a weekend to write, in an evening, and it was correct.

It was correct because before it shipped, somebody counted 105, then 89, then 7,034, then 10,970, and typed DAX into an endpoint to see what came back. That work is not faster with an agent. It is the same work it always was, and it is now the part of the job that is left.