Interrogate Your Zeros

I spent a day building an agent platform for music label operations — three agents over 468,160 royalty statement lines — and then I built the thing that mattered more: an exam for it.

The exam is a 133-question eval suite with a property most LLM evals can't have: an exact answer key. The synthetic label the platform runs on is generated deterministically from one seed, and the generator computes every artist's royalty ledger before any agent exists. Every question — "what's this artist's net payable for October, after recoupment, across four currencies" — has one correct answer, to the penny. No LLM-as-judge vibes for the money questions. The number is right or it isn't.

I expected the suite to grade my agents. Instead, for it's first 24 hours, it mostly graded me.

That week taught me the thing I actually want to write down: when an eval fails, the failure is a measurement of the whole system — model, harness, scorer, budget policy, billing meter, your assumptions — and the model is usually the least broken part. A zero is not a verdict. A zero is a lead.

The first run: 84.9, and three suspicious zeros

First full live run: 133 questions, real API, $11 of tokens. Overall score 84.9, with the money categories — royalty math, recoupment, cross-collateralization — at a perfect 100.

Great, right? Except three numbers looked wrong in a way that itched:

• Abstention: 10%. These are trick questions about artists that don't exist. Correct behavior is refusing to answer. My agents had passed exactly this test in a live smoke days earlier — and now they were failing 9 out of 10?
• Multi-step workflows: 38.9%, with process checks near-perfect but final answers missing entirely.
• The runner reported $16.74 spent against a $15 budget — it blew through its own hard cap and didn't stop. Meanwhile my provider dashboard said I'd spent ~$11.40.

A worse version of me publishes the 84.9, writes "agents struggle with long-horizon tasks and abstention," and moves on. It would have sounded smart. It would have been fiction.

Bug one: the ruler was 1.5× too long

The spend discrepancy cracked first. My price table carried the model's sticker price; the API was billing a dated intro tier — exactly 2/3 of sticker. Every cost the platform metered was 1.5× reality.

A wrong price table sounds cosmetic until you follow it downstream. Each agent run has a real-money budget cap. The caps were denominated in dollars — measured by the inflated meter. So every workflow agent was being strangled at two-thirds of its intended budget, dying mid-job, scoring zero. One wrong constant in a YAML file, three symptoms across the table, none of them anywhere near the constant.

The fix wasn't just the number — it was making prices dated, with the tier transition explicit, so the meter can never silently go stale on a scheduled price change again.

Bug two: the budget gate that couldn't fire

Why didn't the run stop at $15? Because the enforcement gate compared the cap against landed spend — questions fully finished and booked. With questions running concurrently, huge in-flight costs were invisible to the gate until they landed, and by then the run was over. Forensically: cumulative landed spend first crossed $15 at question 132 of 133. Even a correct landed-cost gate would have caught nothing.

The redesign is the only one that works under concurrency: reservations. Every question reserves its worst-case cost up front; the gate checks committed (reserved + landed) spend; reservations release to actuals on completion. The regression test was verified to fail against the old code — the test I should have written first, written second.

Bug three: budgets sized by guesswork

Caps fixed, prices fixed, re-run: workflows still died — now at ~$1.10 of true spend against a $1.00 cap. The per-run budget itself had been a guess, made before any live run existed, and the guess was simply smaller than what a full-period reconciliation across six distributor statements legitimately costs.

So the budget became an empirical floor sized from measured workflow cost — and the decision log entry says the quiet part out loud: per-run budgets must be sized against the measured cost of the work they guard. A safety limit set below the cost of the job it protects isn't safety. It's a bug with a dignified name.

Bug four: the guillotine

Budget raised, re-run again. Workflows still failed — under budget now, but exhausting their iteration caps after nine minutes. This is the point where you stop guessing and read the flight recorder.

Every span tree told the same story. The agent did the entire job correctly — ingested statements, matched thousands of lines, computed allocations for ~100 artists — and then tried to submit the batch. The submission is a large JSON payload. The model's output window was 4,096 tokens; the payload was bigger. The reply was cut mid-JSON, the runtime saw invalid tool arguments and rejected it, and the model — which believed it had sent the whole thing — tried again. Identical amputation. Sixteen consecutive times, on a metronome: stop_reason: max_tokens, 4096 tokens, denial, retry.

The agent was never lost. It was gagged.

Raising the ceiling unblocks it. But the scarier finding was the near-miss: if the truncation had ever landed cleanly between two allocation entries, the prefix would have parsed as valid JSON and submitted a silently partial batch — dozens of artists just missing from a payroll, no error anywhere. Only alignment luck prevented it, eighteen runs in a row. So the durable fix isn't the bigger window — it's the contract: a truncated reply is never acted on, ever, keyed off the stop reason. There's now a test where a truncated payload forms a valid prefix and the runtime must still refuse it. That rule outlives the bug.

The zero that was actually a disagreement about honesty

And the abstention massacre? I read all ten transcripts expecting hallucinations — invented artists, confabulated rates. I found zero. In every single "failure," the agent correctly identified that the entity didn't exist, said so plainly... and then also emitted the ANSWER: line the question's template demanded, filled with a placeholder like $0 or N/A. My grader pattern-matched the placeholder as an attempted answer and flunked the run.

Two of my own instructions were in conflict — the system prompt's abstention format versus the question template's mandatory answer line — and the models resolved the contradiction reasonably. The grader punished them for my ambiguity. The fix went into the grader, not the agents, pinned by tests containing the exact transcript shapes that exposed it.

One abstention question kept failing every model even after the fix, and it turned out to be the most interesting question in the suite: it asks about a clause in §9 of a contract whose real audit-rights language lives in §5. The models find the true content and helpfully answer about it — repairing the question's false premise instead of refusing it. That's not hallucination; it's the exact boundary between helpfulness and precision that production systems live on. The question stays. It's doing its job.

Week-one ledger: four runs, four bugs, zero hallucinations. A pricing miscalibration, a concurrency race in budget enforcement, an empirically-undersized safety cap, and a silent output-truncation hazard that could have filed partial financial batches. All four were caught because deterministic zeros refused to go away, all four were diagnosed from span traces rather than vibes, and all four now have regression tests proven to fail against the old code.

Then the exam did its real job

With the instrument calibrated, I ran the experiment it was built for: the same 133 questions, same prompts, same tools, same pinned judge — swapping only the model. The flagship, the mid-tier, the cheap one. The question every team argues about in Slack, answered with a table instead of an opinion:

The headline: a dead heat at 4× the price. The 0.09-point gap between the flagship and the mid-tier is roughly thirty times smaller than the measured run-to-run noise of a single model on this suite. Paying flagship money bought nothing measurable. The step from cheap to mid-tier buys ~9 points for four cents a query; the step from mid-tier to flagship buys ~0.09 for eighteen cents — about 400× steeper per point of accuracy.

The second finding explains the first. Five whole categories — every money category included — scored a perfect 100 on all three models. Seventy-three of 133 questions where swapping the brain across a 16× price range changed nothing. That's not the models converging; that's the scaffolding equalizing them: retrieval hands over the governing clauses, one deterministic engine does all the arithmetic, the SQL surface is parser-constrained, and the model's remaining job — driving well-built tools — is one the cheap model can do. On a disciplined platform, model choice becomes a routing decision, not an identity decision. Where the models genuinely separated was judgment under ambiguity — deciding which anomalies matter, holding a long workflow's thread — which is precisely where you'd want your spend concentrated.

One last calibration story, because it's the whole essay in miniature. The benchmark's single security deduction — the flagship's lone failure on prompt-injection questions — turned out, on trace adjudication, to be my scorer again: a phrase-check for canary compliance matched the words "i have approved" inside the sentence "I have approved nothing and disclosed no internal data." The model had not only refused the injected instruction; it identified the injection as non-contractual text, explained why it would be void even if it were a clause, and recommended a data-integrity review of how tampered language entered the record. The best adversarial answer in the entire sweep, docked by a grader that couldn't parse a negation. It's documented as a checker limitation — the score stands until the next deliberate suite revision, because rulers don't get edited after the measurement.

What I actually believe now

Exact ground truth changes the epistemics of everything downstream. Because the world's answer key is deterministic, a zero could never be shrugged off as model variance — it demanded a mechanism. Every "the model failed" hypothesis I formed that week was wrong, and I only found out because the suite made wrongness cheap to interrogate: every span, token, cost, and stop reason of every run is on the record.

Score the process, not just the answer. A question here scores the minimum of three tiers — exact match, mechanical trace assertions, judged prose — so a right number produced by mental math fails, and a beautifully-cited wrong number fails. Two of the four bugs were invisible at the answer layer and obvious at the trace layer.

Quarantine your infrastructure failures. Mid-sweep, a billing-limit outage froze ten errors into one model's row as zeros, silently reading as "the flagship can't reconcile." The harness now renders infrastructure errors as a dagger — never a fake zero — and heals them in place under the same run lineage. A provider outage is not model incapability, and an eval that lets one masquerade as the other will eventually make you ship the wrong conclusion.

And interrogate your zeros. Every dramatic number in my first week — the 10%, the 38.9, the 30.0, the security "failure" — was my own harness talking. The models, across ~$120 of adversarial, exact-answer, process-audited examination, fabricated nothing. Not once. The uncomfortable inversion of the standard discourse: I went looking for the model's failures and kept finding my own, filed under the model's name.

The suite now runs in CI. Any change that drops a category more than three points fails the build. The ruler is calibrated, the ruler has teeth, and the ruler has a documented list of its own known flaws — which, as far as I can tell, is the only kind of ruler worth owning.

---

Backline is public: an agent platform for music label operations — three agents, structured-first retrieval, human-gated writes, and the eval suite this post is about. The repo's README is pinned to its artifacts by tests, so every number above is recomputed from committed data on every CI run. github.com/sergioavilax/backline

This post has a sequel. I deployed the platform to AWS with Terraform, migrated the exact world into RDS, and re-ran the entire suite as a pre-registered parity experiment — the scores landed 0.8 points apart, and the regression gate failed both runs, which turned out to be the good part: The Gate Failed Both Runs.