The Prototype Took an Afternoon. The Production Version Took a Quarter.

Shipping a working demo on a frontier model is easy. Shipping the same workflow as a reliable, observable, audited system is a different job, and in 2026 that job is mostly running a tight eval and observability loop.

Every LLM project I have shipped has the same shape: a working prototype in an afternoon, and a production system a quarter later. The demo is the model doing something impressive once. Production is the model doing it reliably ten thousand times a day, observably, with a rollback when it does not.

What I keep relearning is that the lever in 2026 is not parameter count, it is the loop: a prompt change goes through an eval gate, ships behind a gateway, gets scored on live traffic, and the lowest-scoring traces feed the next prompt change. The faster that loop runs, the better the product gets. Everything below is the production playbook I run now, in seven steps, learned mostly by getting each one wrong first.

What “production” actually means

A prototype becomes a production system when it has five properties, and I have shipped things missing one or two and paid for it every time.

• A reproducible eval gate. Every prompt change and model swap is tested against a frozen golden set before it ships.
• Tracing on every request. Each call emits a structured span with the prompt, model, tokens, latency, tool calls, and quality scores.
• Continuous quality scoring. Live traffic is scored on groundedness, context adherence, and task-specific judges, not just unit-tested in CI.
• A gateway in front of providers. Direct provider SDK calls buried in business logic are the thing I most regret shipping, because routing, retries, fallback, and key management end up reimplemented in every feature.
• A rollback story. Prompts are versioned, models are pinned, and any release reverts in minutes.

Anything missing one of these is a prototype that happens to be running in production.

Why the bottleneck moved from training to evaluation

Two years ago the conversation was model selection and fine-tuning. It has flipped, for three reasons I watched play out. Frontier APIs got cheap enough that using a frontier-quality model for routine traffic is economical, and self-hosting a small open model is cheaper but only narrowly. Open models closed the quality gap on narrow tasks when fine-tuned. And the hard problems moved up the stack: hallucinations, prompt injection, tool failures, drift on new user populations, and audit trails are not solved by scaling parameters, they are observability and evaluation problems.

So building for production in 2026 is mostly the discipline of running a tight eval and observability loop while picking sensible defaults for everything else.

Step 1: define the eval gate, not the model

Before I open a provider dashboard, I write two things down. The user-visible promise, meaning what the feature does and what the worst-case failure looks like for a user. And the acceptance criteria, meaning what scores on what dataset would let me ship.

Concretely that is a golden set of 100 to 500 examples covering the head and the long tail of expected inputs, plus a target on a defined metric. For a support assistant, that might be groundedness above 0.85, context adherence above 0.9, and a custom policy-compliance judge above 0.95. That is the eval gate, and every later step is a decision about whether I cleared it. Writing the gate before the model is the single habit that has saved me the most rework.

Step 2: get the data right before you pick a model

Even with frontier models, data is where most production projects fail, and three traps catch me most.

• Coverage gaps. The golden set over-represents easy cases, so I sample new examples from the production trace log weekly.
• Ground-truth ambiguity. Two annotators disagree on the correct answer, and until that is resolved the eval gate is measuring noise, not quality.
• Leakage. Golden-set examples sneak into the few-shot prompt or the fine-tuning set, so hash and dedupe.

For retrieval apps the data step also includes a chunking and indexing strategy and a separate retrieval-quality eval, precision and recall at k plus a generative check like context adherence on the final answer.

Step 3: pick a base model, and a backup

There is no single right answer, and the defaults that work for most teams look like this.

Workload Primary Backup Complex reasoning, code, tool use A frontier reasoning model The other frontier model Long-context retrieval, multimodal A long-context frontier model A second frontier model High-volume classification, routing A small frontier model A self-hosted small open model Style or schema enforcement A fine-tuned small open model A frontier model with strict JSON schema Regulated, on-prem only A mid-size open model Another open model

The move is almost never to lock in one provider. Route through a gateway, evaluate both candidates on your own task, and keep the cheaper one as primary with the pricier one as fallback for low-confidence outputs.

Step 4: train, fine-tune, or skip

For most knowledge-grounded tasks, retrieval plus a frontier model beats a fine-tuned smaller model on cost, freshness, and latency. I fine-tune only when one of these is true: I need strict JSON-only or domain-vocabulary output, I need sub-200ms latency and can host a small model, I operate in a regulated domain where the weights must stay on my hardware, or I have a style that prompting cannot enforce cheaply. Whatever I choose, I evaluate the tuned model against the same golden set and the same evaluators I would use on a frontier baseline, because a fine-tune that scores worse than the baseline plus a better prompt is a sunk cost.

Step 5: deploy behind a gateway

Direct completion calls inside application code are the pattern I have torn out of the most codebases. In 2026 the route is: app to gateway to provider to response to eval to trace and back. The gateway owns key management across providers, routing rules like a cheap model for short prompts and a frontier model for long ones, retries and fallback and circuit breakers when a provider degrades, budget caps per route and team and environment, and a single point to inject guardrails and PII scrubbing.

The point is not which gateway. The point is that something owns the routing concern instead of every feature reimplementing it. Behind the gateway, every response flows through an OpenTelemetry tracing layer so any request can be replayed, with the user ID, the prompt version, and the response all on the span.

Step 6: evaluate production traffic continuously

The eval gate stops bad changes from shipping. Continuous evaluation catches what the gate could not predict: distribution shift, new prompt-injection patterns, tool failures, and silent quality drops. In practice I run evaluators on a sample of live traces, 1 to 5 percent for high-volume products and everything for low-volume high-stakes ones. Then I alert on rolling p95 score drops, not averages, because a 10 percent drop in the worst decile of responses matters more than a 2 percent move in the mean, and the mean is exactly where a real regression hides.

Step 7: monitor, scale, and roll back

By this point the work is closer to SRE than to ML research, and five things stay on the dashboard. Latency at p50, p95, and p99 per route and provider, because frontier APIs have multi-second tail latencies that surprise teams used to web SLOs. Token consumption and cost broken down by feature, since a surprise bill almost always traces to one feature looping. Quality scores over time by user segment. Tool-call success rate for agentic flows, which often causes more user-visible damage than a model-quality dip. And prompt-version pinning, so every request traces back to a known prompt, model pin, and gateway config. When something breaks, the rollback is a one-line gateway config change, not a code redeploy.

What actually separates the teams that ship

The teams that ship fastest are not the ones with the largest model. They are the ones that have a golden set and a defined eval gate, a gateway in front of every provider call, tracing on every request with continuous evaluators on sampled or high-stakes traffic, a one-line rollback for prompts and pins, and a weekly cadence of reading the lowest-scoring traces and feeding them back into the dataset. Everything else, including model selection and fine-tuning, is downstream of getting that loop right.

If you have taken an LLM feature from demo to production, I would like to know which of the seven bit you hardest. For me it is the eval gate, because every quarter I lose to a regression is a quarter I skipped writing the gate before the model.