What should an MCP tool return? I ran 72 trials instead of arguing

There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, four connected servers eating 21,077 tokens of context before anyone asks a question. The argument is about what MCP costs. Almost nobody in it has measured what agents actually do with the data a tool returns.

I ended up measuring that, not because I planned to, but because a maintainer refused to let me guess.

The question nobody wanted to answer with opinions

I contribute to CNCF Jaeger's MCP server. Last April I proposed exposing service performance metrics (latencies, call rates, error rates) as an MCP tool, and hit an immediate design fork: what shape should the output be?

Option one, summary rows: pre-aggregated stats per service, compact, cheap. Option two, per-bucket time series: the raw points, roughly 720 of them per service at default resolution, expensive but complete.

I asked which the maintainer preferred. The answer, verbatim, from the issue thread:

This type of decision should not be based on opinion, but on benchmarks with a real agent troubleshooting some issues and using this MCP tool to access metrics, where you could do A/B testing of different output formats.

Fair. So I built the A/B.

The setup

Everything below is public in jaeger-mcp-bench, including the harness, the tasks, the scorer, and a research log of everything that went wrong.

The fixture is Jaeger v2 with the spanmetrics connector, hotrod generating traffic, and Prometheus behind it, snapshotted so every run sees identical metric state. In front of the metrics API sits a thin bench server with exactly one switch: --format=summary|series. No new semantics, just the shape of what comes back.

Six troubleshooting tasks, and this part matters: three were chosen because I predicted summary would win them (point questions: current latency, ranking, threshold checks) and three because I predicted series would win (temporal questions: spike detection, correlation, trend). Designing the tasks to favor both arms keeps the deck unstacked against your own hypothesis.

Two agents, and not stripped-down tool loops: Claude Sonnet through the Claude Code CLI and Gemini 2.5 Pro through the gemini CLI, real system prompts and all, because that's what production agents actually are. Three trials per cell, 72 trials total, cells run in randomized order (seed 42) so fixture drift can't correlate with either arm. Scoring is programmatic against ground truth, not vibes.

Note: Every trial ran headless: claude -p and the gemini CLI invoked one shot per trial, no session and no one to rephrase, so a decline ended the trial rather than starting a retry.

What 72 trials said

The result I expected was wrong answers. Bad format, wrong conclusions, embarrassing agent behavior, good story.

That is not what happens. Across 72 trials there was exactly one wrong commitment, and it traced to a bug in my own bench server, not to the format (disclosed in RESULTS.md; it biased against summary, and I'd rather report my own bug than a fake finding).

The real difference lives in the decline rate. Given summary rows, agents said some version of "I cannot determine this from the available data" seven times more often than with series, almost entirely on the temporal questions. And they were right to decline: aggregation had destroyed the time axis the question needed. You cannot localize a spike in an average.

Agents don't get it wrong when you underfeed them. They give up, politely and correctly.

The stats hold up under correction: Claude's summary-versus-series gap is significant after Bonferroni (p=0.001 against a 0.0125 alpha). Gemini's, at p=0.016, sits right on the boundary, and I'd rather say that than round it in my favor. Point queries were a wash, exactly as predicted: when the question needs one number, format doesn't matter.

So series won. But that's not the useful lesson

The tool shipped with per-bucket series as the decision, backed by data instead of my taste. Fine.

The useful lesson is about what "expensive output" actually buys. The whole MCP cost argument treats tokens as waste: big responses bad, small responses good. The benchmark says the relationship is more specific than that. A compact format is cheap right up until it deletes the axis a question depends on, and then its cost is not tokens, it's the agent declining to answer at all. A "cannot determine" costs you the entire investigation loop, plus the retry, plus whatever the human does when the agent shrugs.

Put differently: the token bill of series output is visible and easy to complain about. The failure bill of summary output is invisible until you measure decline rates, and nobody measures decline rates.

If you're designing an MCP tool right now, the transferable version is:

1. Match the format to the question class, not to the token budget. Point questions tolerate aggregation. Temporal and causal questions do not.
2. Watch declines, not just errors. Models fail politely. Your error dashboards won't see it.
3. Balance your benchmark against your own prediction, or you'll build a deck stacked toward whatever you already believed.
4. When your bench produces a result you like, hunt for the bug first. Mine had one, and it flattered the winning arm.

Honest limits

Seventy-two trials is three per cell. Two model families, one fixture, six tasks, all in the observability domain. This settles which format a metrics tool in a tracing backend should return; it does not settle MCP philosophy. If you run the harness against other domains I'd genuinely like to see the numbers, and everything needed to do that is in the repo.

The maintainer was right, is the short version. The format decision took a weekend to benchmark and would have taken zero minutes to argue about forever.

I do this work for hire, mostly making LLM agents safe and observable against production infrastructure. Scope and pricing are at roshansingh.systems/#hire, or write to inbox@roshansingh.systems and tell me what your agents touch.