Stop Optimizing Only the Prompt: An Agent Has Five Axes

An agent has five optimizable surfaces, not one. Optimizing only the prompt captures roughly a fifth of the headroom, and the four axes most teams skip are usually the ones holding the ceiling down.

The reframe that changed how I optimize agents was watching a prompt-only loop win on my eval set and lose in production. It found a system prompt that beat the held-out slice, I shipped it, and production regressed, because the real bottleneck was a pair of near-duplicate tool descriptions the agent kept confusing, somewhere the prompt optimizer could never reach.

An LLM is a function from string to string, so its optimization surface is one axis. An agent is larger. It picks tools from a registered set, decides which arguments to pass, calls a retriever, reads the result, decides whether to call another tool, and eventually produces an output. A clean score on the final response tells you nothing about the four upstream decisions that produced it. So automated agent optimization is five axes, each with its own search space, its own optimizer family, and its own per-evaluation cost:

• The system prompt, a free-form string.
• The tool descriptions the model reads when picking a tool, one free-form string per tool.
• The retrieval config: chunk size, overlap, top-k, reranker, embedding model, filters, a bounded numeric and categorical space.
• The few-shot bundle, which examples and in what order, a bounded combinatorial choice.
• The model, a discrete choice among usually fewer than ten.

Optimizing one axis while the others move is the core mistake. Lock the other four, optimize the fifth, then move to the next.

Why optimizing the prompt misses the agent

An agent that scores 71 percent on your task can fail in five distinct places, and each place has a different fix. The system prompt may be ambiguous about output format. The tool descriptions may not distinguish two near-duplicate tools the agent confuses. The retriever may return chunks too large to leave room for useful context. The few-shot bundle may show the wrong failure modes. The model may not reason at the depth the task needs.

A prompt-only optimizer can see the first of those and none of the rest. Run it long enough and it finds a prompt that locally beats your eval set, and then production regresses because the actual bottleneck was somewhere the optimizer could not reach. That is the trap most agent-optimization tooling falls into: the library optimizes a string, and the agent has four other things to optimize.

The five axes, and what each one is

Each axis is a real surface on a production agent, with a different shape and a different optimizer.

• System prompt. The most-discussed axis and the one with the strongest research behind it. Free-form text-gradient and reflective-evolutionary methods both do well here.
• Tool descriptions. The most underrated. The model reads the description and parameter docs on every turn before choosing a tool, so two tools with overlapping descriptions create chronic selection errors no system-prompt edit fixes. The reward has to come from tool-call correctness at the span level, not final-answer quality.
• Retrieval config. Chunk size, overlap, top-k, reranker, embedding model, query rewrites, and filters, a structured space of six to ten parameters. A free-form text optimizer is the wrong tool because there is no text to rewrite. Bayesian search wins because each trial is expensive, since you re-index, re-retrieve, and re-eval, and the sampler converges in 40 to 60 trials.
• Few-shot bundle. Which demonstrations live in the prompt, how many, and in what order, a pick-k-of-N problem. Bayesian search handles the smaller version, and a mutation-based optimizer handles the larger one with stylistic variation.
• Model. Sometimes the prompt is not the bottleneck, the model is, and swapping a small fast model for a larger one beats every prompt edit combined. The optimizer here is a structured comparison: run the same eval suite against each candidate with the current prompt and tools, score on quality, latency, and cost, and pick.

How to design the reward signal

Every optimizer is a search against a metric, and the metric bounds everything. If it is wrong, the optimizer finds prompts that exploit the wrongness. Three rules hold across all five axes.

• Score the step the axis controls, not just the final answer. If you optimize tool descriptions on final-answer scores alone, the optimizer cannot tell whether a failure came from tool selection, argument formatting, or downstream reasoning. Tool descriptions get scored on span-level tool-call correctness, retrieval gets context precision and recall on the retrieved chunks, the few-shot bundle gets consistency between its pattern and the answer, and the system prompt gets instruction adherence plus whatever rubric dimensions you care about.
• Use the same evaluators in CI and in production. Register the rubrics once and reuse them in the optimization loop, the CI gate, and the live monitor. If the optimizer scores high on a metric production never measures, you optimized the wrong thing.
• Hold a slice out. Split into training, validation, and a held-out test set. The optimizer sees training, validation picks the winner, and the held-out slice is never seen until final verification. A gap of more than a couple of points between training and validation means the eval set is too narrow or the optimizer is gaming the rubric.

One more guardrail: add a length penalty to the text-axis optimizers, because most LLM judges are length-biased and free-form optimizers drift toward verbosity. A soft cap in the score aggregator keeps prompts and tool descriptions tight.

The six optimizers and which axis each one fits

There are six optimizers worth knowing, and the mapping to axes is what matters more than the internals.

• Random search. A teacher model generates several paraphrases of the seed text and the highest scorer wins. It is the baseline that pays for itself: if it moves the score three to five points, the wording has headroom and a directed optimizer will move it further, and if it does not move, the wording is not the issue and no optimizer will save you.
• Bayesian search. A sampler over bounded parameters, the right pick for retrieval config and few-shot selection, and resumable across runs, which matters when each trial is expensive.
• Meta-prompt. A teacher model reads the current prompt and the failures, forms a one-sentence hypothesis, and writes a single rewrite, where the hypothesis is the auditable artifact. Good for system prompts on shorter runs where you want one strong rewrite per round.
• Text-gradient with beam search. Sample failing examples, write distinct critiques, generate improved variants from each, and keep the top beam. Use it on system prompts and tool descriptions when failures cluster into nameable critiques like classification, structured extraction, or tool selection.
• Reflective evolutionary search. Evolutionary search with Pareto-frontier selection, for when you have competing metrics like quality plus cost plus latency and want a frontier of winners rather than a single one. It can also be far cheaper in rollouts than reinforcement-learning approaches.
• Mutate-critique-refine. A loop with thinking-style mixing, the most opinionated about how a prompt should be rewritten, for when you want stylistic diversity in the candidate pool.

The decision rule that holds up: start with random search on whichever axis you suspect is the bottleneck. If it moves the score, switch to a text-gradient or evolutionary optimizer for text axes and Bayesian search for structured axes. If it does not, the bottleneck is elsewhere, a different axis, the eval, or the model, and another optimizer pass will not help.

The production loop, one axis at a time

Same evaluators on both sides of the deployment line, one axis moving at a time.

• Lock the other four axes. Pin the model, the retrieval config, the few-shot bundle, and the tool set, so the axis you are optimizing is the only thing moving.
• Build the eval set. Hand-label or synthesize 300 to 500 examples covering the failure modes you have seen, split 70 / 20 / 10 into training, validation, and held-out.
• Pick the optimizer for the axis. Free-form text gets text-gradient or evolutionary, structured search gets Bayesian, and the model axis is a full eval pass per candidate.
• Bound the run. Set a patience for how many iterations without improvement to allow, a minimum improvement that counts, and a hard cap on evaluations wired to your dollar budget.
• Run, validate, audit. Run against training, pick the winner by validation, re-score on the held-out slice, and human-audit 30 to 50 production-shape outputs. If human ratings track the eval score, you have signal, and if they do not, the evaluator is the bottleneck.
• Ship behind a flag, A/B on real traffic, promote on signal. Version the winning artifact and tag every production span so a regression is automatically attributable.
• Mine the next round. Production traces produce a fresh failure cohort, and that cohort becomes the next round’s training slice.

Two cadences run in parallel. The offline optimization pass runs when you ship a new agent or swap models. The regression gate runs on every pull request against the same evaluators the optimizer used, because the optimizer is too slow and expensive to run per PR and the eval gate is not.

The production gotchas to keep on the dashboard

• Eval-set overfit. The optimizer finds prompts that game the rubric without generalizing, and the held-out slice catches it when the validation-to-test gap exceeds a few points. Without a held-out slice, production finds it for you.
• Judge contamination. If the same model writes the optimized prompt and judges its output, the judge is biased toward its own writing. Use a different model family for the judge, or pair it with deterministic rules like schema checks and ground-truth match.
• Calibration drift on a model upgrade. A prompt tuned to one model can regress when you swap to another, and the same goes for tool descriptions and few-shot bundles, so re-run the optimization, or at least the eval, on every model swap and tag spans with the model version.
• Cost runaway from long prompts. Free-form optimizers drift toward verbosity, so track tokens per call as a guardrail and make sure the winning prompt does not double your inference bill.
• Tool-selection regressions. Tool-call accuracy can rise on the eval set and fall in production if production has tools the eval did not cover, so give every registered tool at least 10 to 20 examples, including ambiguous cases between similar tools.

A worked example: a drive-thru voice agent from 66 to 96 percent

The goal was a fast-food voice agent that survived real-customer chaos: hesitation, mid-order changes, interruptions, rushed speech. I used a brain-first architecture, treating speech-to-text and text-to-speech as interchangeable peripherals and the LLM as the brain that handles reasoning, context switching, and tool calls, because if the agent cannot understand that “actually, make that a Sprite” means removing the previous drink, no amount of realistic voice synthesis saves it.

• Synthesize the data. A synthetic generator built a 500-row ground-truth set from a one-line prompt, pairing what the customer says with the order the agent should book, covering modifications, combo meals, and no-pickles cases.
• Baseline it. The first system prompt ran against the 500 scenarios across a few candidate models. The logic was decent at around 80 percent, but the responses ran to multi-paragraph essays, and that became the control to beat.
• Simulate the edge cases. Running the agent against personas who hesitate, stutter, change their mind, lose patience, or rush was brutal. The agent talked too much and, when a user changed their mind, added both items to the cart. Success dropped to 66 percent.
• Define the eval, then optimize. I defined ten task-specific criteria including context retention, objection handling, and language switching, and let the eval surface concrete fixes: reduce decision-tree depth for menu inquiries, and restrict the agent to the menu vector store so it stops inventing dishes. Instead of hand-editing, I ran a text-gradient optimizer for 20 rounds, jointly optimizing task completion and interruption handling.
• Result. The optimized prompt scored 96 percent task completion on the held-out slice. Before, it was polite but slow and failed on complex changes. After, it was crisp: “Burger, no pickles. Got it,” and it handled the indecisive scenario cleanly.

One honest caveat. The 96 percent came from optimizing the system prompt with the other four axes locked. I had already swept the retrieval config the night before and selected the model in a separate eval pass. The 30-point gain is real, but it is the headline of a five-axis loop, not a one-axis trick.

What this comes down to

Automated agent optimization is five axes, not one: the system prompt, the tool descriptions, the retrieval config, the few-shot bundle, and the model. Each has a different search space and a different optimizer, and the reward signal has to score the step the axis controls, run the same in CI and production, and hold a slice out. Lock four axes, optimize the fifth, bound the run against a budget, audit the winner against humans, and feed production failures into the next round. Optimizing only the prompt reaches about a fifth of the agent. The other four axes are usually where the ceiling actually sits.