Last week I changed a system prompt based on a feeling. It was the first prompt change after the evaluation harness from Part 8 went live, and I was completely sure about it.
The target was the markdown table. Part 8's first nightly run caught the agent answering price comparisons with a markdown table that renders broken in the chat frontend. The fix looked obvious: add one line to the system prompt demanding plain text. I checked six conversations by hand. All six looked better. I was ready to ship it to production.
Then I ran the comparison the way Part 8 promised: the same 40 cases, the same judge, two prompts. The old prompt won. Not by a little. It won 18 pairs, lost 10, and tied 12, and the judge's rationales made the reason visible. The plain-text line had also made the agent terse, and terse answers dropped the order summary that customers actually need. My confidence was a sample size of one. The dataset was the jury.
This part is about the pattern that settled that argument: pairwise comparison, the LLM-as-a-judge pattern for A/B testing prompts and tool descriptions before they reach production. It is the harness from Part 8, upgraded to answer "which version is better?" instead of "is this version good?"
The Problem With Ship-by-Feeling
Every prompt edit is an experiment with one sample. You notice one conversation where the agent is verbose, you add "be concise", and the change ships because that one conversation got better. The dataset from Part 8 makes the agent measurable, but a nightly score cannot tell you whether a change helped. One night is noise, three nights is a signal, and by the time you have three nights of data you have already shipped the change to every user.
The variable itself is the problem. A system prompt and a tool description are the two things in an agent you cannot unit test. Part 6 proved the code is bug-free. Part 8 proved the answers are good on a fixed dataset. Neither says anything about whether your new wording is better than the old wording, because "better" is a comparison, and a score against a rubric is not a comparison.
The comparison needs a controlled experiment: one dataset, two configurations, one judge, run on the same day. That is exactly what the pairwise pattern gives you, and it is one of the two evaluation patterns the Spring AI LLM-as-a-Judge guide documents.
Direct Assessment vs Pairwise Comparison
The guide defines the two patterns side by side:
-
Direct assessment. The judge scores one response against a rubric, usually a 1 to 4 scale. Spring AI's
SelfRefineEvaluationAdvisorworks this way: it rates a response, and if the rating is too low it retries with feedback. This is the pattern Part 8 used for the nightly metrics. - Pairwise comparison. The judge picks the better of two candidate responses. The guide's words: "Pairwise Comparison: Judge selects the better of two candidate responses (common in A/B testing)."
For prompt changes, pairwise is the better tool, for two reasons. First, comparing is easier than scoring, and the guide makes that argument itself: evaluation is fundamentally easier than generation, and relative judgment is easier than absolute. "Which of these two answers is better?" is a simpler question than "Is this answer a 3 or a 4?", and simpler questions produce steadier judges. Second, rubrics drift. A "4" from last month is a "3" today, because the judge model updated or your bar moved. A pairwise verdict against the same fixed opponent does not drift.
The catch is position bias, and it comes from the paper that founded this field. The MT-Bench and Chatbot Arena paper that introduced LLM-as-a-judge measured three systematic biases in judges: position bias (a preference for whichever response appears first), verbosity bias (a preference for longer answers), and self-enhancement bias (a preference for answers that look like the judge's own style). The paper also proposed the mitigation I use: judge every pair twice with the positions swapped, and only trust the verdicts that survive the swap.
Step 1: Freeze a Baseline Before You Change Anything
A pairwise comparison is only valid if both sides ran on the same cases, the same tools, the same memory, and the same day. The baseline is not your memory of how the agent used to behave. It is a snapshot.
I keep a baseline client built from the current production prompt, and a candidate client built from the edited one. Both share the same tool registry and the same conversation memory as the agent from Parts 1 through 8. The only difference between them is the line I am testing. If you change two things at once, you will never know which one moved the score.
The snapshot has one extra requirement: save the responses and the tool call logs, not just the pass or fail. When a candidate loses, the losing responses are evidence. When a candidate wins, the winning responses are the new baseline's regression test. I store both under a timestamped directory: experiments/2026-08-06-shipping-tool/.
Step 2: Change Exactly One Thing
The candidate is the same agent with one prompt line or one tool description edited. Everything else stays identical. This sounds obvious and it is the rule I break most often, usually by "cleaning up" a second description while I am in there. Two edits, one experiment, zero information about either.
Step 3: Run the Same Cases and Judge Every Pair
Spring AI does not ship a pairwise evaluator. The evaluation testing reference documents two built-in evaluators, RelevancyEvaluator and FactCheckingEvaluator, and both are direct assessment. Pairwise is a pattern you assemble from the pieces Spring AI gives you: a ChatClient per configuration, the same dataset as Part 8, and structured output to parse the judge's verdict.
The verdict is a record:
public record PairwiseVerdict(
String winner, // "A", "B", or "TIE"
String rationale // the judge's reason, kept for the report
) {}
The judge prompt is my own template. The guide's best practices section sets the rules I follow: forced choice with a small number of options, a rationale, and a separate judge client. Mine asks for the winner and one sentence of reasoning:
private static final PromptTemplate PAIRWISE_PROMPT = new PromptTemplate("""
You are comparing two assistant responses to a customer question
in an e-commerce chat.
Question:
{question}
Context the assistant had:
{context}
Response A:
{responseA}
Response B:
{responseB}
Pick the better response on correctness, factual accuracy against
the context, helpfulness, and format. If they are equally good,
or both unacceptable, answer TIE.
Answer as JSON with two fields: "winner" ("A", "B", or "TIE")
and "rationale" (one sentence).
""");
The judge call follows the same hygiene as Part 8: temperature 0.0, and a separate client from the agent's, because the guide's own example code comments "use separate ChatClient for evaluation to avoid narcissistic bias". Parsing uses Spring AI's structured output:
PairwiseVerdict judge(String question, List<Content> context,
String responseA, String responseB) {
Prompt prompt = PAIRWISE_PROMPT.create(Map.of(
"question", question,
"context", context.stream().map(Content::getContent).toList().toString(),
"responseA", responseA,
"responseB", responseB));
return judgeClient.prompt(prompt).call().entity(PairwiseVerdict.class);
}
The swap test wraps it. Every pair is judged twice, with the order swapped, and the second verdict is mapped back to the original labels:
private String verdictWithSwap(String question, List<Content> context,
String responseA, String responseB) {
PairwiseVerdict forward = judge(question, context, responseA, responseB);
PairwiseVerdict reversed = judge(question, context, responseB, responseA);
String reversedWinner = switch (reversed.winner()) {
case "A" -> "B";
case "B" -> "A";
default -> "TIE";
};
// A pair that flips when you swap positions is a tie.
// The judge did not have an opinion; the position did.
return forward.winner().equals(reversedWinner)
? forward.winner()
: "TIE";
}
This is the rule that makes the numbers honest: only pairs where both passes pick the same winner count as decided. A flipped pair is a tie, and in my runs about one pair in five flips. Without the swap test, that 20% silently votes for whichever answer I put first, and I always put the candidate first, because I am rooting for it.
The runner is the Part 8 loop, doubled. For each of the 40 cases, run both clients and judge the pair:
for (EvalCase evalCase : evalCases) {
String baselineResponse = baselineClient.run(evalCase.userText());
String candidateResponse = candidateClient.run(evalCase.userText());
String winner = verdictWithSwap(
evalCase.userText(),
candidateClient.lastContext(),
baselineResponse,
candidateResponse);
results.record(evalCase.id(), winner, candidateClient.lastToolCalls());
}
Step 4: The Decision Gate
The verdict counts feed one gate with three conditions, and the gate is a hard rule, not a suggestion. A candidate ships only when all three pass:
- The candidate wins at least 60% of decided pairs. A decided pair is one where both swap passes agreed. With 40 cases, expect roughly 30 decided pairs after the flips, so 60% means a clear margin, not a coin flip.
- Tool discipline did not drop. This one is outside the judge entirely. I diff the tool call logs with the same deterministic assertion as Part 8. A candidate that calls different tools is a different agent, and it needs a full Part 8 nightly review, not a pairwise vote.
- No regression on money paths. Checkout, refund, and shipping questions get read by hand whenever the vote is close. The judge is a signal, not a sign-off, and the guide says it plainly: maintain human oversight for high-stakes decisions.
Anything else goes back to the drawing board, and the losing responses become the fix list. In the first experiment, the judge's rationales told me exactly what to fix: keep the plain-text instruction, but specify that the order summary line must survive. The narrowed version shipped two days later.
Real Results From the First Week
Three experiments in the first week, and one of them saved me from shipping a regression the nightly metrics would have missed for days.
Experiment 1: the blunt plain-text line. Old prompt won 18, new won 10, 12 ties. The direct-assessment metrics explained why: format compliance went up from 92.5% to 97.5%, but answer correctness fell from 89% to 82%. The pairwise judge weighed the whole answer and called the trade a loss. Verdict: do not ship. The narrowed version won the rerun.
Experiment 2: a shipping tool description. The description said "returns the delivery estimate" and the agent was using it for the wrong kinds of questions. The reworded description named the inputs and the region logic. New won 24, old won 7, 9 ties. Verdict: ship. Tool discipline was flat, and the money-path cases read clean.
Experiment 3: the swap test earns its keep. A system prompt change for the refund flow looked like a clear win on the first pass: new 21, old 18, 1 tie. The swapped pass said the opposite: old 20, new 11, 9 ties. The stable pairs, the only ones that count, had old winning 18 to 11. Ten pairs flipped, which is a position bias problem, not a prompt improvement. Verdict: inconclusive, rerun. Without the swap test that change ships on a phantom win.
The numbers from the report looked like this:
experiment: refund-flow system prompt
pairs judged: 40 (two passes each)
stable pairs: 30
first pass: new 21, old 18, tie 1
swapped pass: old 20, new 11, tie 9
stable: old 18, new 11, tie 1
verdict: do not ship (inconclusive, position bias)
The Honest Cost Section
Pairwise testing costs more than nightly evaluation, and the extra cost is the swap test.
Generations double. 40 cases times two configurations is 80 agent runs per experiment, up from 40 for a nightly pass. The agent runs are the cheap part, since they use the same tools and memory as production traffic.
Judge calls double. 40 pairs times two passes is 80 judge calls per experiment, and the judge is a separate model, so it is real spend. The discipline that keeps this sane: one experiment per day, not one per typo. Batch your prompt ideas for a week and test them together against the same baseline snapshot. Testing three variants against one baseline costs three experiments, not one.
The dataset is small and the gate is thin. 40 cases with a 60% threshold is a margin of roughly 8 to 10 votes after the flips are counted as ties. A close result gets re-run the next day, not shipped on a tiebreak. Part 8's rule still applies: one night is noise, three nights is a signal.
Position bias never dies. The swap test does not remove it. It turns a silent bias into a measured disagreement rate, and about one pair in five disagrees in my runs. If your flipped-pair rate creeps toward 30%, the judge model has changed, and that is a judge problem, not an agent problem.
The Checklist
If you take nothing else from this part, take this list.
- Freeze a baseline before editing anything. A timestamped snapshot of responses and tool logs, not a memory.
- Change exactly one thing per experiment. Two edits, zero information.
- Judge every pair twice, swapped. Only stable pairs count. A flipped pair is a tie.
- Keep the judge separate and at temperature 0.0. Same rule as Part 8, same reason.
- Gate on 60% of decided pairs. Coin flips do not ship.
- Diff tool discipline deterministically. The judge never sees the tool logs.
- Read money paths by hand on close votes. Human oversight for high-stakes decisions.
- Batch experiments. One per day, not one per typo.
- Treat a flipped-pair rate over 30% as a judge problem. Check the model changelog before touching the agent.
What Comes Next
The agent is now measured, Part 8, and every change to it is gated, this part. The last gap is the rollout itself: how a new prompt or a new model reaches real traffic without a single big-bang switch. Part 10 is the production runbook: canary traffic splits, automatic fallback when a model degrades, and cost caps that stop a prompt regression from becoming a bill regression.
What is the worst prompt change you shipped on a feeling? And did you find out afterwards? I read every response.
I write about Java, Spring Boot, and AI agents every week. Subscribe, it's free.
Bookmark this one. You will need the swap test the week the judge loves your new prompt a little too much.
