How I Rebuilt My LLM Eval Stack From the Floor Up, Part 2: Taking It Distributed

Part 1 built the deterministic floor. The deterministic floor works until your dataset outgrows one machine. Here are the four runners I chose between, and how to pick by bottleneck instead of fashion.

In Part 1 I rebuilt the eval stack from the floor up: deterministic checks at the base, a classifier layer above them, a judge on a residual sample. That stack works beautifully on one machine. Then the dataset grows, and it stops fitting on one machine.

Here is the number that forced the change. A 10,000-case suite at 500 milliseconds per evaluation runs in about 1.4 hours on a single worker, and that is before the judge round-trip. Fifty workers bring the same suite under two minutes. Add a four-language coverage matrix and the sequential version turns into a working day while the parallel version stays in minutes. That gap is the whole difference between “we gate every PR” and “we gate every Tuesday,” and sequential eval is the quiet reason most teams ship LLM features with no regression coverage at all.

So Part 2 is about fanning out. The lesson I landed on: the four common runners each optimize for a different thing, so you pick by your bottleneck, not by what is fashionable. Celery for throughput, Ray for compute-heavy bursts, Temporal for durability, Kubernetes for elasticity. Most teams already run one of these, and using the one you already run is usually the right call, because introducing a new orchestrator only for eval is the cleanest way to double your operational tax forever.

Why one runner does not fit

Eval volume in 2026 is not what it was two years ago. A serious regression suite for a single agent route now sits at 500 to 5,000 cases, and once you multiply by routes, language coverage, rubric count, and the per-tenant variants enterprise customers ask for, the matrix you actually want to gate CI on is well into five figures. Three forces drive it.

• Dataset growth from production. If you refresh the golden set weekly, the suite grows on a schedule, and new failures land in it every Monday.
• Many rubrics per case. A single case now scores against groundedness, faithfulness, toxicity, prompt injection, completeness, and a domain rubric, so six judge calls where 2024 had one.
• Multi-modal coverage. Image, audio, and video inputs land in the same suite as text, and a single image-grounded eval can be ten times the compute of a text-only judge.

The combined effect is that a suite that ran in five minutes last quarter runs in 45 this quarter, and overnight by the next planning cycle, and overnight means it stops running per PR. The fix is to fan out, not to trim the suite.

But fan-out is not one problem. A bursty overnight batch fails differently from a continuous live-scoring lane. A multi-modal eval that holds a vision model in worker memory needs something different from a sub-100-millisecond classifier sweep. A regulated workflow that has to replay deterministically needs something different from a throwaway PR check. There is no single optimal runner because there is no single workload.

Celery when the bottleneck is throughput

Celery is the right call when your bottleneck is raw cases per second and your stack already runs Celery. The broker is Redis or RabbitMQ, the cognitive overhead is the lowest of the four because most Python engineers have seen it, and throughput scales linearly with worker count until the broker itself becomes the bottleneck, which for typical eval payloads sits around a few thousand tasks per second per broker. Reach for it on suites in the hundreds to low thousands of cases, on eval that runs in the same process family as your app, and on CI gates where the runner only spins up during the pipeline.

The fan-out is the standard Celery pattern, a group over the case list with the results merged in the callback, and the trap I hit is the visibility timeout: a judge call that runs longer than the timeout gets redelivered, so the same case scores twice. The fix is to tune the visibility timeout together with late acknowledgement and to prefer one case per task over batched tasks. Celery is also the right first runner for a team that has none, since the on-ramp is an afternoon and its ceiling only shows up at multi-modal or audit-replay work, both of which want a different runner anyway.

Ray when the per-case work is heavy

Ray wins when the work inside each case is heavy: image, audio, or video inputs, multi-step judge chains, or scoring that needs vectorized math in the worker. Its resource model lets you ask for a GPU on the scoring task itself, which matters when the judge is a local guardrail model you run in-process, because an actor holding the model resident across cases beats cold-loading it per task. Reach for it on multi-modal eval, on teams already operating Ray for training or serving, and on suites that benefit from in-memory shuffles between scoring stages.

The trap is the head node. If it restarts mid-suite, in-flight state is lost, so for long suites you submit a Ray job with a persistent dashboard and checkpoint results to durable storage between batches rather than trusting a single long-lived run.

Temporal when eval has to recover, not restart

Temporal is the right call when eval matters enough that a mid-run failure has to recover instead of starting over. Per-activity retries, a replay-correct execution history, and workflow versioning make it the strongest choice for regulated work where the eval audit trail is part of the compliance story. Reach for it on suites that run for hours or days, on finance, healthcare, and government workloads where audit replay matters, and on pipelines whose stages (fetch, score, attach, report) each need their own retry policy.

The payoff is what happens when a worker dies on case 4,217: the activity retries on another worker, the deterministic history replays, and the final report is identical to the one you would have gotten if nothing had failed, which is genuinely hard to build by hand. The trap is timeouts, which have to be tuned per rubric family, because a heavy judge can take 30 seconds while a regex check returns in milliseconds, and one blanket timeout either starves the cheap checks or kills the expensive ones.

Kubernetes when the load is bursty and polyglot

Kubernetes is the right call when the workload swings hard across the day, the worker pool should scale to zero between batches, and the worker images are polyglot. Job and CronJob primitives give you native scheduling, autoscaling handles the bursts, and the container can be Python, TypeScript, or Java, which matters when the eval surface spans several languages. The worker code looks like the Celery worker without the broker: pull a shard from durable storage, score it, write the results back, and let the cluster handle the rest.

The trap is pod startup latency. For a small ad-hoc eval like a single PR check on 50 cases, per-pod cold start can dominate the whole run, so keep a small Celery or local pool for the small jobs and reserve Kubernetes for the large batch and overnight work.

The runner comparison at a glance

Optimization and on-ramp:

Recovery and failure modes:

Each runner is the right answer for its own column and the wrong answer for the others. Forcing one across every workload is how you end up with an idle Ray cluster during business hours or a Celery broker gasping under multi-modal payloads. The healthier shape, once you have more than one workload class, is one runner per class: a small Celery pool for PR checks, Kubernetes jobs for overnight batches, Temporal for the regulated tenant. The key is to keep the eval definitions identical across all of them so the operational complexity stays in deployment and never leaks into the rubric layer.

Keep the cost accounting honest

A distributed runner with no per-eval cost telemetry is a budget grenade. When judge calls fan out across hundreds of workers, you have to capture the spend per call, so route every judge call through a gateway that returns the exact cost, the latency including retries, the model actually used, and whether a fallback or a guardrail fired, then tag each result row with those values. Aggregate by suite, rubric, or runner and you can see where a cheap sub-100-millisecond classifier earns its place versus where a stronger judge is worth the spend, and you can A/B the same suite across two backends and pick the better dollar-per-rubric. Without that telemetry the runner choice is uninformed and the bill is a surprise.

The anti-patterns I retired

• Sequential eval that should be parallel. This is where most teams start and most stay, usually behind “we will add eval to CI when it is fast enough.” It is fast enough, you just need a runner, and even Celery on one broker fixes most of the pain in an afternoon.
• One runner for every workload. Ray for a 50-case PR check is over-engineering with an idle cluster bill, and Celery for frame-by-frame video grounding is under-powered with the broker as the bottleneck. Small pool for small checks, heavy pool for nightly batches.
• No per-runner cost telemetry. If you cannot tell whether a suite cost 4 dollars or 400, you cannot optimize the runner choice or justify a migration. Capture the per-call cost from day one.
• A custom orchestrator instead of one of the four. You will reinvent half of Temporal and then learn why the other half exists. Adopt, do not rebuild.

What I would do this week

If your eval is still sequential, the order that worked for me was to pick the runner that matches both your bottleneck and the infra you already run rather than adding a new one, wire a single worker and confirm one case scores end to end, then fan out the smallest suite you have, a single rubric on 100 cases, and measure wall time before and after. Route the judge calls through a gateway so every result row carries its cost and latency, and pick one anti-pattern to retire by the end of the quarter. The goal is not a perfect distributed system. It is an eval suite that runs in minutes instead of hours, costs what you expect, and tells you what failed when something fails.

That is the floor built and the floor scaled: deterministic checks that catch most of production for nothing, fanned out across a runner that matches your bottleneck so the suite runs in minutes instead of hours, at a cost you can see per call.

If you have not read it, Part 1 is where the floor gets built, the deterministic checks that run before any judge and would have caught the bug that started this whole rebuild: How I Rebuilt My LLM Eval Stack From the Floor Up, Part 1: The Deterministic Checks.