I Cut 2,490 Agent Test Runs to 206 and Kept the Same Coverage

The full matrix was 83 agents × 30 scenarios = 2,490 runs. Each one a real LLM call, 30–80 seconds. At 10 workers that's about 2.7 hours, and in practice 4–5× that once you're debugging, so we're talking well over 10,000 calls. Serialize it and it's twelve days.

I ran 206 of those. Not because I was lazy, and not because I was cutting a corner. The other 2,284 were paying real money to re-prove behavior that deterministic tests had already proven, with a local 4B model, which is the most expensive way to run a test that doesn't need a model. Same coverage, one afternoon.

If you've ever stared at a test matrix you didn't want to pay for, keep reading. I work on agent-tooltrust, an open-source gate for AI agent tool calls, and this was the design that let us run the field test at all.

The Trap Was a Whiteboard Diagram

The engine was already proven. 2,490 deterministic assertions, zero LLM calls, every decision path exercised. It was done in the boring, correct way.

But the field test kept pulling me back to the cross-product, because the cross-product is what "thorough" looks like on a whiteboard. 83 × 30. Write it down and it feels safe. Every agent, every scenario, no gaps, nothing to defend in a review. It's also a combinatorial trap, and the bill shows up in two places you don't want: compute cost, and the days it eats while you wait.

The cross-product also wins meetings, and that's the part nobody warns you about. Anyone can propose a clever sampling design, but 83 × 30 is a number you can defend in a room without explaining covering problems. So it keeps getting picked, not because it's right, but because "I tested everything" is an easy sentence. "Every scenario once, every decision type once per framework" is the true sentence, and it took me a week of staring at the matrix before I realized I was optimizing for how the number sounded in review.

The uncomfortable question was: what is a real LLM call actually for here, if the engine is already proven? And once I wrote that down, the answer was smaller than the whiteboard.

The Field Test Had One Job

The field test's real job was adapter proof. Does each framework correctly surface allow, audit, escalate, and deny in a real agent loop? That's it. That's the one thing a deterministic test can't do, because it needs a model that sometimes misbehaves.

That's a covering problem, not a cross-product problem. Once the engine is framework-agnostic, you don't need every agent × every scenario. You need every scenario covered at least once, and every decision type proven at least once per framework. Everything else is re-proving the same cells.

The Design

Total: 206 runs instead of 2,490, a roughly 12× cut for identical coverage.

Plan A covers breadth. Every scenario is exercised by at least one real agent, across 10 frameworks and 5 agent classes. If a scenario is going to behave differently under a real model, some agent in the 83 is going to hit it. Plan B covers depth. Within each framework, every decision type is hit at least once, so we know the adapter can express all four verdicts, not just the easy ones. Together they prove what the cross-product was trying to prove, without paying for the cells that add no new information.

The full breakdown, including the scenario-to-agent mapping, is in the v0.1.1 field test report.

What Worked, and What Didn't

What worked: the split. Breadth and depth are different questions, and each one has a cheap, correct answer. The moment I stopped asking "how many cells do I need to fill?" and started asking "what is the LLM call for?", the total stopped being scary. 83 and 123 are both numbers I could defend in a review. 2,490 was a number I was too embarrassed to defend, which is how I knew it was the wrong one.

What didn't work: the seven failures in Plan B. And here's the part worth stealing. All seven were not-available. The LLM didn't call the guarded tool. Not one of them was unexpected-decision, the engine returning the wrong verdict. A mock agent always calls the tool it's told to. A real 4B, given five tools at once, sometimes just answers in prose.

That distinction is the whole post. unexpected-decision means the gate is broken. not-available means the model behaved like a model. If you don't separate those two, you'll either ship a flaky gate that fails for reasons you can't fix, or ship a gate that's silently too loose because you decided to "ignore" the model's weird behavior.

It's easier to believe when you've seen the prose. The scenario asked the agent to use a guarded tool. The model wrote, in perfect English, what it intended to do, and never called anything. A deterministic test and a mock agent will never produce that sentence, because neither of them has a language center. The gap between "the gate works" and "the gate works on a model that sometimes decides talking is easier than calling" is exactly the gap only a real agent can show you. That's what Plan B was paying for, and it's the coverage the cross-product would have bought twelve times over.

Where This Breaks

I want to be clear about the assumption, because it's the part that doesn't travel. A covering design assumes independence between the engine and the adapter. That held here because the engine is framework-agnostic. If your system has cross-cutting interactions between agents and frameworks, the cross-product may actually be cheaper than finding the gap later, and I'd run it before you cut the matrix.

And the $0 failures are only trustworthy when code review already caught the real bugs. The field test is the second line of defense, not the first. Pretending it's the first is how a green coverage number becomes a false one.

The Open Question

I still can't fully answer how to decide what only a real agent can prove, versus what deterministic tests can. I made a judgment call here, and it happened to be the cheap one. I don't know a principled way to know that in general, and I'd rather own that gap than oversell the 206.

How do you decide where your deterministic tests stop and your real-agent tests begin? I'd genuinely like to hear how other people draw that line.

Code and receipts: agent-tooltrust · field test report · field test plan · design decisions — all MIT, all public.