When Your SWE-bench Run Dies, It Is Almost Never Your Patches
· 4 min read · Originally on Medium ↗
The SWE-bench harness builds a tall stack of Docker images and runs each task in its own container. That is what makes the scores reproducible, and it is where every first-run failure actually lives. Five of them, and the fix for each.
The first time I ran the SWE-bench harness, it died partway through with a Docker error and the model never got evaluated. That turned out to be the normal experience. Almost none of the first-run failures are about your patches or your predictions file. They are about the environment the harness builds around your model, and the same five problems catch nearly everyone.
The harness builds images in three layers, a base image, then roughly sixty environment images, then one instance image per task, and runs each task in a fresh container. That design is why the numbers reproduce, and it is also why disk is the first thing to break. Here are the five failures I hit, straight from the harness setup, with the fix for each.
The 120 GB disk trap
The README asks for at least 120 GB of free disk, 16 GB of RAM, and 8 cores before you start, and the disk figure is the one that catches people because Docker Desktop ships with a virtual disk far smaller than that. The harness quietly fills it while building images and dies with “no space left on device.”
The fix is to raise Docker Desktop’s disk image size to at least 120 GB under Settings, Resources, Advanced, and on Linux to give the partition holding the Docker data directory the headroom. Then clear old layers so a half-finished previous run is not still holding space. Watching the reclaimable space before and after a build is worth the habit, since a failed build leaves dangling layers that keep eating disk until you prune them. Treat 120 GB as a floor, because the full prebuilt image set runs into the hundreds of gigabytes uncached.
cache_level, the disk-versus-speed dial
The --cache_level flag decides which image layers survive between runs, and its default keeps the base and environment images while dropping the per-task instance images. Keeping every instance image at the instance level makes reruns fast and pushes storage toward the hundreds of gigabytes. Dropping to base clears almost everything above the base image and rebuilds it next time. The rule I use: base when a run is one-off and disk is the scarce resource, the default env for balance, and instance only when I rerun the same tasks repeatedly and have the room.
There is a cheaper move for debugging a single task, which is to pass a specific instance id so the harness builds only that one instance image on top of the shared layers instead of the whole set.
ARM64 on Apple Silicon
By default the harness pulls prebuilt x86 images, and on Apple Silicon those do not match the chip, so you get platform-mismatch and manifest failures. The fix is to pass an empty --namespace so the harness builds each image locally against your architecture. The tradeoff is time, since local builds are much slower than pulls across dozens of environment images, and one catch is that the force-rebuild flag and the namespace flag cannot be combined. Native ARM64 support was still open and unmerged when I last checked, so the local-build path stays the reliable one, though Epoch AI publishes best-effort ARM64 builds for a large share of tasks that are worth trying first.
Mid-run hangs and manifest errors
A run that freezes is usually too many parallel workers fighting for the machine, sometimes a genuine deadlock that will not even respond to Ctrl-C. Cap it with --max_workers set to around the smaller of 0.75 times your core count or 24. Before assuming a deadlock, rule out a slow build, since each task has a time budget and an image that takes minutes to compile is expected. A true hang stops printing new logs.
A “manifest not found” error means the image the harness expected is missing or mis-tagged, so rebuild it, or skip local building entirely by pulling Epoch AI’s prebuilt set, which covers almost every task in the test split and the whole Verified subset. An environment image that fails on a pip read-timeout usually clears on a retry or by pointing pip at a faster mirror.
The five at a glance
Symptom Likely cause Fix No space left on device Docker disk under about 120 GB Raise the disk image size, prune old images Run hangs mid-evaluation Too many parallel workers Cap --max_workers near 0.75 times cores Manifest not found Missing or mis-tagged image Rebuild it, or pull Epoch AI's images ARM64 build fails Pulling x86-only images Pass an empty --namespace to build locally Env image build fails pip read-timeout or dependency clash Retry, or use a package mirror
Verify the harness before you trust the score
The single check that saves the most time is to run the harness against the gold patches that ship with the dataset before scoring any real predictions. Gold should come back close to fully resolved, and if it does not, the problem is your setup, not your model. That one run tells a Docker problem apart from a real change in the score.
So the checklist for every fresh run is short: 120 GB of Docker disk free, workers capped, the right namespace for your architecture, and a gold run that passes. Clear those four and the harness stops throwing Docker errors and goes back to its actual job, measuring whether your patches resolve real GitHub issues.
If you have run SWE-bench, I would like to know which of the five cost you the most time. For me it was the disk trap, because the failure lands halfway through a long build and the message points at Docker rather than at the tiny default disk that actually caused it.