Aug 13, 2026 · 7 min read

The Cache That Lies Politely

This is the third post in an accidental trilogy about measuring the things AI systems usually just assert. The first found zeros — retrieval over near-identical contracts scoring MRR 0.000 without structural scoping, not degraded, zero. The second found reds — an eval suite whose first week caught four bugs in its own harness and none in its agents. This one is about a feature I built, recommended everywhere, that turned out to lie. Politely. With a straight face. At a similarity of 0.9995.

The feature everyone ships

Semantic caching is one of the first optimizations anyone bolts onto an LLM gateway. The pitch writes itself: users ask the same things in different words, embeddings put paraphrases near each other, so serve the cached answer when a new question lands close enough to an old one. Pick a cosine threshold — 0.90 is the number you will find in most vendor docs — and watch your token bill drop.

I built it into Headroom, an LLM gateway and control plane: virtual keys, per-tenant budgets enforced on atomic DynamoDB conditional writes, provider failover with a circuit breaker, per-request cost attribution, and — of course — exact and semantic response caching. The gateway half is solid, ordinary infrastructure. This post is about what happened when I pointed the platform's measurement discipline at its own most-recommended feature.

The reason almost nobody measures semantic cache correctness — as opposed to hit rate, which everyone measures, because hit rate is the number that flatters — is that measuring correctness needs a large question set with exact ground truth. You have to know, for every question, what the right answer actually is.

I had one. My previous project, Backline, generates a synthetic record label — 468,160 royalty statement lines, 385 contracts — with every answer computed before any agent runs. Its eval suite carries 130 questions with answers exact to the penny.

So: seed all 130 into the cache. Generate paraphrases — three per question, each one passed through mechanical checks and an operator review that ended up catching two systematic generation failures the mechanical checks missed. Embed 520 probes once. Then replay the cache's admission decision offline at every threshold from 0.70 to 0.99, so the whole curve is a property of the data rather than of a lucky run.

What the curve said

Two numbers, both from the same corpus at the same industry-default threshold of 0.90:

The cache served 389 of 390 genuine paraphrases from cache, 382 of them correctly. If you only measured hit rate on paraphrase traffic — which is what a pilot does — this feature looks spectacular.
The cache also answered 98 of 130 never-before-seen questions from a neighbouring entry. 92 of those answers were provably wrong. Wrong as in: the answer key says a different number, and the cache returned someone else's answer with full confidence and zero indication anything unusual had happened.

Both halves are the finding. A control surface that looks excellent on the traffic you tested it with, and poisons the traffic you did not, is not a safe control surface. It is a demo that has not met production yet.

"So raise the threshold"

That was my first thought too. It does not work, and the shape of why it does not work is the actual lesson.

The closest wrong answer in the corpus scores 0.999539 — two reconciliation questions differing in a single period token, 2026-02 against 2026-04, with entirely different answers. The furthest correct paraphrase scores 0.889850. The bands overlap. There is no cosine value with wrongness on one side and rightness on the other — the experiment defined a "safe threshold" by rule before drawing the curve, and that threshold does not exist anywhere in 0.70 → 0.99.

Here is the part I want to be precise about, because it would be easy to walk away with the wrong conclusion: the embedding model is not broken. It is doing exactly its job. "Reconcile the February statement" and "reconcile the April statement" are nearly identical sentences — semantically, structurally, in every way an embedding is built to notice. The model correctly reports that seven characters differ and that those seven characters are a negligible fraction of the text.

The failure is mine, and everyone's who ships this: using a similarity score as an admission decision, in a domain where the characters the embedding correctly calls negligible are the entire answer. Any application that templates its prompts — same question shape, different entity, different date, different account — has manufactured exactly this trap for itself. Which is to say: most production LLM applications.

What would actually work

Not a better threshold. The repo's position, held in its decision log, is that admission needs a signal that does attend to the tokens that change answers: entity-and-parameter extraction in front of the similarity check, exact matching on the extracted slots, semantic matching on the residue. Cheap versions of that exist — the point of the finding is not that caching is hopeless, it is that cosine-over-the-whole-string is the wrong control surface for templated traffic, and no amount of tuning the one dial it offers changes that.

In the meantime, Headroom ships the honest version: the semantic cache exists, works, and is delivered with its measured safety curve instead of a recommended threshold — plus a per-request cache_disposition on the ledger so an operator can see exactly what the cache did to every request after the fact.

The discipline underneath

A finding like this is only worth publishing if the reader can check it, so the repo is built so that the reader — and the CI — always can. Every number in this post is in the README, and every number in the README is recomputed from a committed artifact by a test on every pull request: the curve values from the experiment results, the 93.7-vs-93.3 parity verdict from the analysis file, the passthrough overhead percentiles from live ledger rows, the cost table from the billing capture. A claim that stops following from its evidence turns the build red. One test even tightens itself: while the billing screenshot was missing, the cost table was required to say "pending" — and the day the capture landed, the same test demanded the number instead.

The same discipline priced the whole adventure. The gateway ran on Docker Compose, then ECS Fargate, then EKS — same code, three runtimes, three different first-contact bugs, each one committed with its story — and both cloud deployments were applied by hand, measured live, and destroyed the same day, down to per-service empty checks. Total cloud bill: $3.56, against $20–25 budgeted before any of it existed. (The undershoot is a fourteen-hour cluster window, not efficiency, and a test fails if any document tries to claim otherwise.)

The trilogy, closed

Three projects, three measurements the industry mostly argues about by vibes:

Zeros — unscoped semantic retrieval over templated documents: MRR 0.000. Structure first, or nothing.
Reds — an unforgiving eval suite's first week: four harness bugs, zero agent hallucinations. The instrument needs testing more than the agents do.
The cache that lies politely — at the default threshold, one hit in five is a confident wrong answer, and no threshold fixes it, because the dial being tuned does not attend to the tokens that change answers.

The common thread is not that these systems are bad. It is that every one of these numbers was cheap to measure and load-bearing to know — and that the measured version of each story is more useful, and more interesting, than the asserted one.

The repo is github.com/sergioavilax/headroom. A cold clone reaches a working keyless demo in one command. The curve, the corpus, the probes, and every judgment call from H-000 to H-102 are committed. Check my numbers — the tests already do.