How I Rebuilt My LLM Eval Stack From the Floor Up, Part 1: The Deterministic Checks

This is the first post in a series on how I rebuilt my eval stack after a judge-only setup cost me a five-figure bill and still missed the obvious bug. Part 1 is the floor: the deterministic checks that catch most of what production sees for the cost of running your test suite.

The bill arrived a week after we shipped continuous evaluation. A judge-model call at a few hundred milliseconds and a fraction of a cent sounds cheap right up until you multiply it by a million traces a day, and then it is not cheap. The first month landed around forty thousand dollars. What actually changed my mind, though, was not the number. It was that the judge had sailed past a bug a parser would have caught for free: a refund agent quoting an amount off by two orders of magnitude because a tool-call schema had drifted three releases earlier. The judge read the polite prose around the broken field and gave it a 0.89 helpfulness score.

That was when I rebuilt the eval stack from the floor up, and the one-line lesson is this: schema validity, regex, citation validity, and structural assertions catch most of what production actually sees, for the cost of running your test suite. This first post is the floor. Later parts cover the classifier layer, the judge layer, and how the three fit into one cascade.

Why a judge-only eval stack failed me

The setup was a support agent on a retrieval index, five rubrics on every trace, all of them judge calls against a pinned frontier model at temperature zero. Helpfulness, faithfulness, refusal calibration, tone, policy compliance. The dashboard was clean and the CI gate was green, and then a user posted a screenshot of the agent offering a 4,200 dollar refund on a 42 dollar product, which the customer happily accepted and finance later flagged.

I pulled the trace. The judge had scored that response 0.89 on helpfulness. The retrieved context had the correct amount. The tool call had gone out with the wrong number because of a unit-confusion bug in the prompt template, and the field type in the tool schema did not match what the agent passed. A schema validator we did not have would have caught the mismatch in microseconds.

That one bug taught me three things I should have known already.

• A judge reads prose, not structure. The response was fluent and structurally wrong, and the judge had no opinion on whether the tool-call field matched the retrieved amount, because it was scoring the user-facing text and never parsed the call. Semantic evaluation cannot do structural validation.
• The judge was running everywhere, including the roughly 80 percent of requests where any failure would have been a missing field, a malformed link, or a banned phrase that a regex catches for nothing.
• Pinning the judge by model name is not pinning anything. The vendor pushed an upgrade, faithfulness scores shifted several points overnight, and CI started failing for reasons that had nothing to do with our code.

The fix was not a smarter judge. It was a cheaper floor underneath it.

The four families of deterministic checks I shipped

A check is deterministic if running it twice on the same input returns the same score forever, with no external model in the loop. Four families covered most of what I needed.

• Schema and structural validation. JSON schema, and parsers for code, SQL, and XML. This catches malformed tool-call arguments, broken structured outputs, and syntax errors in generated code. It was the first family I added and the one that paid back fastest, since the refund bug would have died right here. It runs in well under a millisecond, costs nothing, and never drifts. The one maintenance cost is keeping the schema files in sync with the actual tool signatures.
• Regex and contains checks. Required substrings, forbidden patterns, and format gates: compliance disclosures, PII patterns, banned names, a required opening tag. Also sub-millisecond, with one operational cost, which is pattern rot, where a prompt changes and a stale regex silently passes output it should fail. The fix is to treat the pattern like any other test artifact, versioned and exercised against a small fixture of known-bad inputs every release.
• Structural assertions. Exact match on canonical answers, function-name match on tool calls, parameter-type validation, length contracts, and normalized equality with whitespace and casing stripped. These are honest on yes-or-no, classification, and constrained-format tasks. The single highest-signal check for agents is validating the call’s parameters against the tool’s declared schema, because most agent failures are not a wrong final answer, they are the wrong tool called with the wrong arguments.
• Citation validity. A retrieval answer can pass a judge’s faithfulness check and still drop the source it was supposed to cite. Checking that the answer attributes claims to retrieved chunks, and that the cited spans actually exist in the context, is a deterministic operation on the answer plus the context. For a RAG system this is the one deterministic check I would ship before any judge call.

There is a fifth family, lexical overlap like BLEU and ROUGE and embedding similarity, but its useful range is narrow. It works on translation and extractive summarization, where the reference and the candidate share short, similar surface forms, and it falls apart on open-ended chat and anything with paraphrase tolerance. I run it for the tasks it was built for and gate nothing else on it.

How the checks stack into a pyramid

The four families settle into a three-layer shape, with one rule holding it together: every layer runs only on what the cheaper layer below it could not decide.

• The deterministic floor runs on every request: schema, regex, function-call validation, citation presence, exact match where it applies, length contracts. Sub-millisecond, zero API cost. In my own audit across three agents, this floor caught roughly 60 percent of the failures the judge would have caught, for nothing.
• The classifier layer runs on whatever passes the floor: small fine-tuned models for sharp safety targets like toxicity, PII, and prompt injection, and entailment models for faithfulness against retrieved context. Single-digit to low-double-digit milliseconds, a small fraction of judge cost, and the same answer on the same input every time.
• The judge layer runs on a residual sample of what survives the first two: the open-ended rubrics like helpfulness, tone, and long-form faithfulness where claim extraction matters. It is orders of magnitude more expensive per call, so it runs on a 5-to-10 percent production sample plus every case in the CI regression suite.

The discipline is to reach for the cheapest layer that gives the right answer. Gate only on the floor and you ship semantic regressions, because word-overlap on a chatbot passes deploys that drop user satisfaction. Gate only on the judge and you ship a five-figure bill and a flaky CI gate.

How the floor gates the judge

The wrong framing is deterministic checks versus the judge. The right one is that the deterministic checks gate the judge, and three patterns held up for me at volume.

• Fail fast. If the response fails schema validation, or a citation-required rubric has no citations, or a refusal-required scenario matches an answer regex, the eval fails outright and the judge never runs. The floor is thousands of times cheaper than a frontier judge call, so it goes first.
• Augment, then judge. A classifier produces a score and per-claim reasoning and hands that to the judge as evidence, so the judge starts from grounded reasoning and only pays the frontier cost when the classifier signal is ambiguous. On faithfulness this cut most of my judge cost with no measurable drop in detection.
• Judge off the hot path. The deterministic checks run inline on the user-facing request, and the judge runs asynchronously as a score attached to the trace afterward. The user never waits for the judge, the dashboard still gets the judge score, and CI runs both.

Where the checks run in production

The floor is not a notebook artifact, it runs in three places against the same definitions.

• On every request, inline. Schema validation on structured outputs, citation presence on retrieval answers, function-call validation on tool calls, and compliance regex on every output, all at the gateway hop rather than inside application code.
• On every commit, in CI. The same deterministic suite plus the classifier layer plus the judge against a pinned golden set, with the deterministic checks gating the judge, and the patterns, schema versions, canonical answers, judge model, and rubric text all pinned in version control so re-running on the same input gives the same score.
• On every production span, sampled. A 5-to-10 percent sample runs through the same definitions that ran in CI, and failures route to an annotation queue and get clustered so a repeated failure becomes one issue instead of a hundred alerts.

The thing that compounds is not any single check. It is the contract: one metric definition, pinned and versioned, producing the same number on the same input whether it runs at the gateway, in CI, or on the sample.

What I would tell myself before building

• Build the floor before the judge. Judge-only eval is fast to ship and slow to operate, and the bill is the first warning, the drift is the second, and the one that took me longest to hear is that a judge cannot do structural validation at all.
• Match the check to the failure mode. Schema for structure, regex for compliance, parameter validation for agents, citation validity for retrieval, and lexical overlap only for translation and extractive summarization. Anything open-ended belongs in the classifier or the judge.
• Pin the contract, not the implementation. The artifact that ages well is the metric definition in version control, reviewed and replayable, because a check with a stale schema fails in the wrong direction and a pinned definition fails for a reason you can fix.

The boring part is the four families of checks. Here is the thing nobody warns you about, and the reason this is a series and not one post: the floor works beautifully until your traffic outgrows a single machine. A million traces a day does not fit in one process, and the moment you try to run this suite at that volume, a new set of problems shows up that has nothing to do with the checks themselves and everything to do with how you run them.

That is Part 2, and it is coming soon: how I took this eval suite distributed, what broke when I did, and the runner design that finally made it hold at production scale. If you have ever watched an eval job that ran fine on your laptop fall over on real traffic, that one is for you.

Until then, tell me which check earned its place first for you. For me it was schema validation, because it is the one that would have caught the bug that started all of this. See you in Part 2.