Why Your Agent Passes Every Eval and Still Fails in Production

A curated eval set is a snapshot. Production is a river. Six ways the snapshot ages the day it ships, and the loop that stops the offline-pass-production-fail gap from being inevitable.

A support agent passes its CI suite, 47 scenarios across four rubrics, and ships. In production it starts quoting refund amounts off by an order of magnitude, contradicting itself across turns, and on one trace handing a user another customer’s order. Pull the failing conversations and every one still passes the per-turn faithfulness rubric. Every one failed the user. The eval said ship, and production had already lost the user’s trust.

The instinct was to blame the rubric. The truer read is that the rubric was fine in March and aging in April, because production had been moving the whole time. The eval set is a snapshot, production is a river, and evals do not ship, traces do. Every release ages the eval set the day it lands, so without a way to feed new failure modes back into the offline set, the offline-pass-production-fail gap is mathematical, not accidental.

What I believe now, after enough of these incidents: the right unit of evaluation is the production trace, not the curated test case. Offline evals are a necessary regression gate and never a sufficient ship gate, because they measure against a world that stopped existing the day they froze. Six drift modes age every eval set, and they run on different clocks.

Drift mode What the offline eval sees What production actually does Dataset drift All curated cases pass New user intents the set never had Tool-API drift The mock returns the same shape The vendor changed schema or error codes Prompt drift Rubric written for v3, frozen The prompt is on v17, rubric still grades v3 Retrieval-corpus drift Index frozen at build time Index doubled, same query, new chunks User-distribution drift Hand-authored test inputs Real traffic looks nothing like the set Agent-step compounding Per-step success 95 percent Eight steps multiply to 66 percent

Why the eval set ages the day it ships

When you freeze an eval set you freeze a hypothesis: what users will do, how tools will behave, which prompt will be in production, which chunks the retriever will surface. Months later every one of those has moved a different distance, and the rubric that scored a clean pass is grading a world the agent no longer lives in.

The rubric is not wrong. It is stale, the way a unit test goes stale when the function it covers gets refactored. Both still pass, and neither protects you. The software fix is to write the test for the current function. The agent fix is to grade the current trace, which means the same rubric definition runs in CI against a versioned dataset and against live traces as a span-attached score. The CI gate is the floor, the live score is the river, and the dataset grows weekly from the traces the rubric just flagged.

Drift 1: dataset drift

The set was written at launch, and users found intents the authors never anticipated. The eval keeps passing because the dataset never moved. The tell is offline scores flat for months while production complaints diversify and the team cannot reproduce most reported failures on the test set. The fix is to sample failing traces weekly, bucket by segment, intent, and score, and promote the hardest 5 to 10 percent into the set with version tags, so each promoted trace becomes a regression a future change cannot break.

Drift 2: tool-API drift

You mocked the tool call in CI, and the real endpoint changed its schema, error shape, or rate-limit behavior on a Tuesday. The agent retries, the retry loop times out, and it fabricates a plausible answer, while CI keeps passing because the mock still returns the old payload. The tell is tool-call latency and retries climbing while the per-response rubric still passes and cost-per-success creeps the wrong way. The fix is to score tool-call success as its own rubric on live spans, so a failing call shows up in the trace next to the failing response. The mock catches your regression, the live score catches the vendor’s.

Drift 3: prompt drift

You shipped v17 of the prompt on Friday, and the rubric was written for v3 in February. It still grades what v3 cared about, so the agent is being evaluated against the wrong contract. The tell is a senior engineer reading ten traces and disagreeing with the judge on six without being able to say why. The fix is to version the rubric in the same change as the prompt it scores, treat it like a contract test, and track judge-versus-human agreement on a small calibration set so you see when it goes overdue.

Drift 4: retrieval-corpus drift

The retriever you evaluated in March indexed 12,000 documents at one chunk size. By May the index holds 38,000, the chunker reranked on a re-embed, and the same query lands on different chunks. The generator dutifully grounds in whatever it was handed, so groundedness still reads 0.94 while the answer is grounded in the wrong material. The tell is generation rubrics holding steady while users call the bot less helpful than last quarter. The fix is to split the suite by layer: retrieval rubrics like context relevance and chunk attribution catch the index drift before the generation rubrics absorb it. A drop in context relevance with stable groundedness means the retriever moved, and the reverse means the generator did. One bisect instead of three days.

Drift 5: user-distribution drift

The set was hand-authored or sampled from launch-month traffic. Months later real users arrive with slang, code-switching, longer prompts, screenshots, and chains of follow-ups the dataset never had, and the judge calibrated against the curated set reads well below what a hand-score of live traffic would. The tell is a spot-check of production traces disagreeing with the judge by 15 points or more, after which engineers stop trusting the rubric and start reading raw traces. The fix is to calibrate the judge against production samples rather than the dataset, and to track judge-versus-human drift as its own metric.

Drift 6: agent-step compounding

Every per-step rubric reads 95 percent, and the agent makes eight tool calls a session. 0.95 to the eighth power is 0.66, so two-thirds of sessions end structurally wrong even when every individual step looks fine, because the rubric never multiplied. The tell is high per-turn metrics next to low completion, outcome, and satisfaction numbers, with tickets that say the bot kept asking the same question or said yes then said no. The fix is to score the trace as a unit, adding conversation-level metrics like completeness, role adherence, and knowledge retention, plus plan-execution scoring over the span tree. Multi-turn metrics are noisier per dollar and correlate with user experience far better.

Why more offline evals cannot fix this

The property all six share is that they happen after the eval set was frozen, and a static dataset cannot encode a hypothesis it does not yet have. A 10,000-example set from March still does not contain May’s tool-schema change, June’s prompt revision, July’s re-embed, or August’s users phrasing things a new way. Scale does not fix the snapshot, only sampling production does, which means the eval surface has to live where the agent lives. The curated dataset is the regression seed, the live traces are the working set, failures cluster and get scored as they happen, the named clusters become the next dataset entries, and the loop closes.

Score the trace on four axes, not one

Per-turn faithfulness on the final response is not enough resolution to diagnose a drifting agent. The trace score I trust now is four axes, each scored one to five.

• Factual grounding. Did the agent stay anchored in the retrieved context or confabulate. This catches retrieval and dataset drift at the response level.
• Privacy and safety. Did it leak PII, cross a tenant boundary, or comply with a jailbreak it should have refused. This catches permission and refusal regressions.
• Instruction adherence. Did it follow the current system prompt. This is the axis that drops when the prompt moved to v17 and the behavior did not.
• Plan execution. Did it pick the right tool in the right order without redundant calls or dead branches. This catches step-compounding and tool drift on the call graph.

Four axes, four kinds of regression, one composite. When the composite drops on a trace, the axes tell you which drift mode just bit you, and the same axes run in CI and on live spans so the diagnostic vocabulary is identical in both places.

The promote-back loop

Closing the gap is a workflow, not a feature, and it is five steps. Cluster failing traces into named issues, because no one triages a flat list of 800 failures. Score each cluster with the four-axis trace score and a written note on what to fix today. Promote three to ten representative traces from the cluster into the offline set with route tags and rubric labels. Re-gate, so the next CI run grades the new entries with the same rubric the production scorer used and the next change on that path cannot regress them. Then optimize the prompt against the expanded set, with the fix having to clear the rubric in CI before it ships.

Run it weekly on an active product, faster on a volatile launch. Static sets older than a quarter rarely match production, and on a fast-moving agent the drift is visible within two to three weeks. Every incident handled this way becomes a regression test nobody has to write again.

Three tradeoffs I made on purpose

• Closing the loop costs operational surface. Span-attached scores plus clustering plus a promote step is more moving parts than a folder of pytest evals. The payoff is a regression suite that ratchets stronger over time, and a new deployment can ship with tracing and offline evals alone and turn the loop on later.
• A self-calibrating judge needs its own guardrail. A rubric that tunes against live traffic can drift in a direction you did not intend, so pin a small human-labeled hold-out and alarm when the judge disagrees with it by more than the inter-rater baseline.
• Trace-attached eval is noisier per dollar than offline scoring. A four-axis rubric on a 30-second trace costs more than per-turn scoring on a short case, and the variance is higher, so sample by failure signal rather than uniformly and put a cheap classifier in front of the expensive judge.

What this comes down to

The eval set ages the day it ships, and no amount of scale fixes a snapshot. Grade the current trace, not the frozen test case: run the same rubric in CI and on live spans, score the trajectory on grounding, safety, instruction adherence, and plan execution, cluster the failures, and promote them back into the offline set every week. Offline pass is necessary and trace-attached pass is what actually protects the user. The curated set is the seed. Production is the dataset you are always catching up to.

If you have shipped an agent that passed CI and broke in production, I would like to know which drift mode caught you. For me it was tool-API drift, a vendor changing an error shape on a Tuesday while every offline rubric kept passing.