I built the same agent in Strands, LangGraph, and CrewAI — and recorded every LLM call to see how they actually differ

Which framework should I actually use? Every comparison article has an
opinion. Almost none of them has data.

So I built the same agent three times — once in Strands, once in
LangGraph, once in CrewAI — ran 27 executions, and routed every
single LLM call through a local recorder proxy so the logs are directly
comparable. The frameworks write different log formats, different trace
shapes, different everything. A one-file proxy in front of all of them fixed
that.

The headline finding: LangGraph's explicit verify/revise loop cut output
variance by 77% (word-count spread 13 -> 3) — at 2.5x the tokens and 2.5x
the latency. Explicit control buys determinism, and you pay for it in tokens
and latency.

https://github.com/sunnydachs/agent-framework-showdown

The task

A tech-news digest agent, identical across all three frameworks:

1. collect 5 headlines via a fetch_headlines tool
2. write a ~100-word digest
3. verify the word count via a word_count tool, revising if out of band

The tools are deterministic and local — no network, no LLM inside them —
because the thing being measured is the framework's behavior, not the tool's.
Same model behind the proxy for all three runs.

How the three frameworks differ (in code)

The philosophy split is real, and it shows up in the implementation:

• Strands (model-driven): you hand the model two tools and a system prompt. The LLM decides which tool to call, in what order, and when it's done. My implementation is 78 lines.
• LangGraph (graph-driven): you define a typed state, every node, every edge, and the loop condition. The model only makes decisions inside nodes. 115 lines.
• CrewAI (role-based): agents are team members with role/goal/backstory; the Crew orchestrates the handoff. 110 lines.

The dependency weight differs too: Strands 262MB (81 packages), LangGraph
71MB (45), CrewAI 699MB (142). CrewAI's weight is the flip side of
"fastest to prototype."

The observability design (the part that made this honest)

One recorder proxy sits in front of every framework
(proxy/rec_proxy.py): an HTTP server that forwards to any
OpenAI-compatible endpoint and writes every request/response pair as JSONL —
the full messages the framework sent (system prompt, tool schemas,
conversation history), the raw response (SSE-streamed or JSON), token usage,
latency, and status. The API key is stripped before writing.

An X-Run-Label header splits traces per run. Strands and LangChain stream
SSE, so a small reassembler (parse_sse.py) normalizes streamed responses
into the same shape as non-streamed ones (content / reasoning / tool_calls /
usage).

Same trace shape across frameworks is what lets you diff messages, tokens,
and latency exactly. Without it, "which framework is slower" is vibes.

The numbers (27 runs, 3 frameworks x 3 scenarios x 3 runs)

Three scenarios:

• base — 80-120 word band (the original task)
• tight — 95-105 words, which forces the verify/revise loop to fire
• drift — the word_count tool's argument renamed (text -> content), testing how each framework copes with schema change

Word-count spread across 3 runs (max - min, lower = more stable):

What the traces actually show

LangGraph in base mode made a single LLM call and wrote the draft in one
shot. Word-count spread: 13 words. Switch to tight, and the explicit
verify/revise loop fires: spread drops to 3 (−77%), tokens go 2,007 -> 5,262,
latency 4.7s -> 11.8s. The graph structure guarantees the loop runs — that
is the product you're buying, and the price is 2.5x on everything.

CrewAI produced byte-identical output across all 3 base and drift runs
(96/96/96 words). temperature=0 plus the role prompt dominates. The flip
side: the call structure is always 4 calls, fixed. The framework that "just
ships it" also just repeats it.

Strands self-corrected inside its own loop. In one tight run the model
wrote a 77-word draft, called check_word_count, reasoned "77 is under 95,
I need to expand it," revised itself to 103, and verified again — five LLM
calls, variable depth per run ([5, 3, 4]). The model deciding when to stop
is part of the design, and the call count being adaptive is what you get for
trusting it.

Schema drift: models adapted perfectly (this time)

The rename was absorbed 100%: all three frameworks' models called the tool
with the new argument name, zero wrong-arg calls, no error recovery
triggered. Worth being precise about what this does and doesn't show: a
one-argument rename is the gentlest possible schema change. Type changes,
removed arguments, or changed return shapes would likely break the
model-driven side (which trusts its prompt over the wire), while LangGraph
would be immune — its tool calls live in code, not in a prompt.

Honest limitations

• 3 runs per cell is a trend check, not a statistical claim. Standard guidance treats ~30 runs as the floor for median estimation.
• One task, one tool. This is exactly where model-driven frameworks shine. LangGraph's graph pays off in complex branching / approval / parallel workflows that I did not test.
• One model. Reasoning volume, tool-call tendencies, and output variance are model-dependent and will change with a different model.
• Dependency footprints move fast; numbers are from the initial install snapshot.

Reproduce it

python3 runs/run_matrix.py       # 27 runs in ~270s
python3 runs/analyze_matrix.py   # -> artifacts/matrix_report.json

The repo README has the full setup (three venvs, the recorder proxy, the
per-framework run commands). One proxy in front of everyone is the whole
trick — when the frameworks write different log formats, a same-shape
recorder is the simplest honest answer.

https://github.com/sunnydachs/agent-framework-showdown