My CI evals were green. A regression still paged me at 3 AM.

The eval that finally caught a bad regression for me was not the one in CI. CI had run twelve hours earlier, against a frozen test set, and passed. The regression was sitting in live production traffic that my test set never covered. Nobody was scoring that traffic, so nobody knew until users did.

So I built a worker pool that scores live traffic outside the app, separate from CI entirely. It sounded simple. It was not, and the reason it was hard is the actual lesson here.

This is a distributed system, not a bigger eval

The moment you move eval out of the request path and onto its own workers reading from a queue, you have built a distributed system. And it fails in all the boring, classic distributed-systems ways, none of which your CI eval ever taught you to expect.

I learned there are four properties this pipeline has to have, and every one I skipped came back to bite me.

• Async. The app never waits for the evaluator. It emits a trace and moves on. If the eval queue is down, the app keeps serving users, it does not stall on a score.
• Idempotent. The same trace must never get scored twice. If a worker crashes and the queue redelivers the message, a second scoring run cannot produce a second charge and a second number.
• Observable. You can answer, at any moment, where a given trace is: waiting, being scored, done, or failed. And what the whole thing is costing you.
• Recoverable. A worker restart picks up exactly where it left off. Nothing in flight gets silently dropped.

Miss any one and the pipeline either loses data, double-charges you, hides failures, or cannot restart cleanly. I managed to hit three of those before I got it right.

The three ways it actually broke

These are not hypotheticals. Each one cost me real money or real trust.

The double-score. A worker pulled a trace, called the judge, and crashed right before saving the result. The queue redelivered it, a second worker scored the exact same trace, and now my judge bill was doubling and my dashboard had two different scores for one trace. The fix was not better retry logic. It was making the whole thing idempotent, so re-running a trace is a safe no-op instead of a second charge.

The silent gap. A worker pool restart dropped everything that was in flight. Traces from a twenty-minute window just never got scored. The rolling average smoothed right over the hole, so it looked fine, and I only found the gap weeks later. The fix was durable queue state plus a small replay tool, so a gap can actually be filled instead of quietly living in the data forever.

The runaway judge. I added a new rubric that timed out on a chunk of traffic, each timeout retried three times, and the bill tripled overnight. There was no dashboard for cost per rubric because I had treated pipeline monitoring as an afterthought. The fix was watching the eval pipeline like any other production service, with cost and error rate per rubric, not just the scores it produces.

What I would tell myself before starting

Build the boring parts first. An idempotency key so a redelivered trace is safe. A dashboard that shows where every trace is and what it costs. Durable queue state and a replay tool, on day one, before you need them. Treat the eval pipeline as core infrastructure, the same way you treat your main message queue, because that is exactly what it is.

The mental shift that fixed it for me was small. I stopped thinking of this as “a bigger version of my CI eval” and started thinking of it as a distributed system that happens to run judges. The day I did that, the failures stopped being surprises and started being things I had already designed for.

If you want the deeper version, with the actual queue-and-worker architecture and how each of the four properties gets enforced in code, this piece lays out the whole blueprint.

If you run eval on live traffic, I am curious which of the four bit you first. For me it was idempotency, and I found out the expensive way, on the invoice.