I changed a tool's schema four ways and watched three agent frameworks break differently — 36 runs measured

If you run agents with tools, this day always comes:

"I need to change this tool's arguments — will the agent survive?"

Tool schema changes are a daily event in agent development. An argument gets renamed. A type changes. An argument is removed. A new required argument appears.

The ideal is to fix the prompt and all call sites at the same time. In practice, "the schema changed first, and the prompts and callers haven't caught up" is a state that always happens.

In my previous article, I ran a comparison across Strands, LangGraph, and CrewAI with the same task. An argument rename (text → content) passed 9/9 in all three frameworks.

That raised a fair question:

"Was that rename the first schema change you tried — or the one that survived after a harsher one broke everything?"

In other words: did rename only work because it happens to be the easiest change?

To answer it, I split schema changes into four harshness levels and re-ran the measurement across 36 runs.

Repo (all code, all traces, all analysis scripts):

https://github.com/sunnydachs/agent-framework-showdown

The experiment design

Same task, same model across all runs, same recording proxy: 3 frameworks × 4 levels × 3 runs = 36 runs.

[The four schema-change levels]
rename: argument renamed        text -> content
type:   type changed            str -> int (a document id)
remove: argument deleted        the text arg is gone entirely
add:    new required argument   "note", never mentioned in the prompt

At each level, the only thing that changes is the word_count tool's signature.

The task prompt stays fixed at every level: "write an 80–120 word digest and verify the word count with word_count" — not one character changes.

What I measured:

• wrong-arg calls: calls whose arguments don't match the schema
• schema errors: how often the tool layer rejected a call
• recovery: did the model self-correct within the run, give up, or never notice the error at all
• silent failure: the process exits 0 but the word count was never actually verified

The result: each level dies a different way

3 runs per cell:

The direct answer to the question: rename was not "the first change I tried" — it was the one that survived.

And it was the only level that survived. One step up the ladder, and all three frameworks die in three different ways.

Strands: degrades silently

At the type level (the argument goes from str to int), Strands behaved like this:

model:  dutifully sends a numeric document id
tool:   returns "document not found"
model:  outputs the digest anyway and finishes
result: exit 0, zero verifications, no error surfaced

No error ever surfaces. Only the token consumption swells to 2.4× (rename mean 2,505 → type mean 6,004).

At the remove level, the argument-less tool returned a quiet placeholder (8 words), and the model accepted it as "verification passed" in 3/3 runs.

The process exits 0 — and nothing was verified.

That is a silent failure.

I wrote in the previous article that agent failures happen quietly. Schema change is one of the triggers that can cause exactly that.

LangGraph: fails loudly

LangGraph is the opposite.

Because the tool is called directly inside Python code, a signature change becomes an immediate TypeError.

verify node: word_count(state["draft"]) runs
tools_harsh: TypeError (the signature changed)
graph:      the draft is finished, but the result is never emitted — dead

9/9 runs across type, remove, and add crashed inside the verify node. Each run gets as far as the draft-generation LLM call, then dies at the verification point.

Silent failure effectively does not exist in LangGraph. In exchange, no run ever produces output.

The failure is loud — and it is a full outage. (Because the error is explicit and the process halts, you notice immediately — but as a service, it is completely down.)

CrewAI: does both

CrewAI sits in between.

At the type level, the model stuffed the digest text into the int field: 9 wrong-arg calls across 3 runs. The tool layer returned a JSON parse error, and the next request carrying that error was rejected by the provider with a 400.

model:          stuffs the text into an int field
tool:           JSON parse error
next request:   provider 400 -> dead

CrewAI's failure shape is not the schema mismatch itself — it dies on the way back from it.

At the remove level, CrewAI believed the quiet placeholder just like Strands and exited 0. (The word-count check the agent was supposed to perform is effectively skipped, yet the run still reports completion.)

"add" was the only harsh change that model-driven frameworks survived

One interesting reversal.

The add level — a new required argument the prompt never mentions — looks like the most unreasonable change of the four.

But Strands and CrewAI fabricated a plausible value in 6/6 runs and resumed verification:

model's fabrication: "Draft digest of AI agents news"
-> a natural description, inferred from the schema itself
-> verification resumes, 3/3 pass

The model can read the schema and reason about it.

Changes the model can infer from the schema itself get absorbed.

Changes that contradict the prompt (type) or delete the verification path itself (remove) do not.

There is a practical lesson here: when you add a new required argument, if its name and description are designed so a model can fill them in by inference, the agent is surprisingly likely to survive.

The failure shapes, summarized

Across 36 runs, each framework's way of breaking is now quantified:

Which framework is "safe"? The data's answer: none of them, alone.

The one that fails quietly needs auditing. The one that fails loudly trades that for availability. To get a schema change through safely, you have to layer defenses.

Honest limitations

• 3 runs per cell show direction, not statistical claims
• The tool is a local function — real HTTP API schema changes also involve network, latency, and status codes
• The quiet placeholder at the remove level is a design choice of the experiment; a real deployment would normally return an error
• Single model measured; schema-inference ability varies by model

Reproduce it

All code, all traces, and all analysis scripts are public. The README has the commands for every experiment so far (36 + the earlier 108 runs).

https://github.com/sunnydachs/agent-framework-showdown

This is a personal OSS project — no warranty, use at your own risk. Issues and improvement ideas are welcome.

Previously in this series: what happens when enterprise requirements hit the same three frameworks (45 runs) and do agents survive a crash (34 runs).