Per-Agent Cost Tracking for Multi-Agent AI on AWS

Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance monitoring you already run: uptime, latency, error rate) says 200 OK, latency fine, everything green.

And you were silently billed about 1.4x what you should have been.

That is the part nobody shows you. Nested traces and per-agent cost are becoming common; the primitives are easy to find now. What stays rare is a data model that lets you act on them: catch the run that looks completely successful while it burns money in the middle. The paper "Why Do Multi-Agent LLM Systems Fail?" (MAST, arXiv:2503.13657) hand-annotated 150 traces across 7 state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa=0.88, and measured failure rates from 41% to 86.7%. The uncomfortable finding: many of those failures do not crash. They complete. They look fine.

In this article I build a small read-only "AWS Account Investigator" crew, wire real cost into every trace span, and then reproduce three silent-waste patterns with real Amazon Nova Pro dollars. You can run the whole thing for $0 locally. Nothing gets created, modified, or deleted in your AWS account.

If you only have two minutes, jump straight to the unique part: catching silent waste. The build up to it matters, but that section is the payoff.

I spent about a week on this against a real AWS account: a few days probing the SDK's behavior before I trusted it, then several more building the crew, watching the trace design break twice, and reading the SDK source when the docs ran out. What follows is written from that, not from a quickstart. The scars are in here on purpose, because they are the part that saves you the week.

This is for people already building AI agents who have never put a real observability layer under them. You know agents, tools, and crews. Where the tracing vocabulary (spans, traces, OpenTelemetry) is new, I define it the first time it shows up.

Contents

• Why agent observability is a different problem
• Picking the instrumentation
• Prerequisites
• Adding Traccia to your code
• The stack: AWS native and read only
• The cost bridge and one gotcha
• Modeling a multi-agent crew in traces
• The unique part: catching silent waste
• Watching it happen: the live control panel
• Build your own, at zero cost and read only
• An honest take on Traccia
• Honest caveats
• FAQ

Why agent observability is a different problem

Traditional application monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that is enough, because the work is deterministic and the failure modes are loud. An AI agent breaks all three assumptions. It decides its own control flow at runtime, it calls tools in an order you did not hardcode, and it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it is a trend, not an event.

So agent observability has to record things classic APM never needed: how many reasoning cycles an agent took, which tools it called versus which it was allowed to call, the token count and dollar cost of each step, and which agent in a multi-agent crew did what. Those attributes are what make an invisible regression visible.

This is not a fringe opinion. AWS's own Well-Architected Agentic AI Lens frames the baseline state (its "Level 1") as exactly this problem: agent costs are visible only at the account level, Cost Explorer cannot separate agents or workflows, and "teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing." The whole point of what follows is to move off Level 1: to make spending "attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level," which is AWS's own words for the target.

A quick vocabulary anchor, since the rest of the article leans on it. A span is one timed step with attributes attached (one LLM call, one tool call, one AWS read). A trace is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes (status, latency). Agent observability is the same trace structure carrying agent-specific attributes: cycle count, tokens, cost, and which agent owned the step. That is the whole idea; everything below is just putting the right attributes on the right spans.

At one run, a 1.4x overspend is a rounding error. At enterprise scale it is a budget line and a governance problem, and it shows up in four concrete ways:

• Cost control. A 1.03x-to-1.4x silent overspend per run (the real range I measured across three waste patterns), multiplied across thousands of daily runs and dozens of agents, is real money leaking with no alarm attached. Per-agent, per-tool cost on the trace is the only way to attribute and cap it.
• Accountability. When a crew misbehaves, "which agent, owned by which team, cost what" needs to be answerable. Trace-level ownership metadata turns a vague incident into a routed ticket.
• Regression detection. Agents change when prompts, models, or tools change. A known-good baseline plus per-run deltas catches the day a prompt tweak silently doubled token usage, before finance does.
• Auditability. In regulated environments you need a record of what the agent read, what it decided, and what it cost. A trace is that record.

The theme throughout: a correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose. Here is the before and after in one line. Before, a typical demo gives you one lump token count for the whole run, and an inefficient run looks identical to an efficient one. After, every reasoning step and every AWS read is a span carrying real cost, tokens, cycle count, and the owning agent's identity, so two runs that both return the correct answer and both show 200 OK are no longer indistinguishable when one of them costs 43% more.

Picking the instrumentation

Once you know you need per-agent cost, cycle counts, and tool-call attributes on every span, the next question is what to write them with. You could do a lot of this with raw OpenTelemetry, and I nearly did. The reason I did not is that agents need a vocabulary plain OTel does not ship: token counts turned into dollars, a span-level agent identity so one process can render as a real fleet, ownership metadata, and a way to view per-agent cost grouped by session. You end up building all of that yourself, or you find an SDK that already speaks it.

There are options here: LangSmith, Langfuse, and Arize Phoenix all do LLM tracing, and each is worth a look depending on your stack. I went looking for one I would trust in a codebase, which for me means two hard requirements: I can read the source, and I am not locked in. Traccia cleared both cleanly. The SDK is open source, Apache-2.0 licensed, and built on OpenTelemetry (OTel, the vendor-neutral open standard for traces and metrics, the reason you are not locked into any one backend). The spans it produces are standard OTel, the file exporter works with no account and no network, and I could read exactly what it does to my data before committing to it (I did, and the source-grounded critique later in this article is the result). It runs at $0 locally; the hosted dashboard at app.traccia.ai is optional and only comes in when you want the visualization. An open, inspectable SDK with an optional commercial backend is a split I am comfortable adopting, because the instrumentation does not trap me.

That is the real reason it is in this build: agent-native plumbing I did not want to hand-roll, source I could audit, and a real $0 offline path. It also has sharp edges, and I hit several of them; those are documented in full near the end rather than glossed over.

Why not Amazon Bedrock AgentCore Observability or Langfuse, the two obvious AWS-native alternatives? Both are good, and for many teams either is the right call. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on the AgentCore runtime, but AWS's own Well-Architected lens is blunt about the cost gap: "cost reporting stops at the AWS account level, so teams can't separate supervisor overhead from worker execution." Per-agent dollars are something you still assemble. Langfuse is the strong open-source incumbent and I would happily use it; it just was not the tool I was asked to put through its paces here. The point of this build is not "Traccia beats them." It is that whichever tracer you pick, the per-agent cost attribute and the baseline-delta detection are things you wire on purpose, and this article shows exactly how.

Prerequisites

Nothing exotic. Three things to run this yourself:

1. Python 3.10+ and the SDKs (strands-agents, strands-agents-tools, traccia, boto3). The repo pins the exact tested versions in requirements.txt.
2. AWS credentials with read-only permissions for the services the crew reads (Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, GuardDuty), plus bedrock:InvokeModel so the agent can actually call the model. The repo ships a ready-to-use policy at iam/read-only-policy.json; AWS's managed SecurityAudit + ViewOnlyAccess cover the reads, but you still add bedrock:InvokeModel on top of them.
3. Amazon Nova Pro, which is two separate steps: (a) enable model access once in the Bedrock console (us-east-1, amazon.nova-pro-v1:0) under Model access, and (b) allow bedrock:InvokeModel in your IAM policy. The console grant is not an IAM permission, so you need both.

No Traccia account is required. With no API key it writes traces to a local file, which is the $0 path used throughout this article.

Getting it running is four commands:

git clone https://github.com/simplynadaf/ai-agent-observability-aws.git
cd ai-agent-observability-aws
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

Then create the least-privilege policy once (with your own admin credentials) and attach it to whoever runs the crew:

aws iam create-policy \
  --policy-name AgentObservabilityReadOnly \
  --policy-document file://iam/read-only-policy.json

Adding Traccia to your code

Before the crew, here is the smallest version of what "wire cost into a span" actually means, because that is the one non-obvious step. Traccia auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, so on those stacks you get most of this for free. It does not yet hook Strands or raw Bedrock, so you stamp the cost yourself. It is a short function, and one attribute name will bite you (more on that below).

def stamp_llm_cost(span, result, model_id="amazon.nova-pro-v1:0"):
    usage = result.metrics.accumulated_usage
    in_tok, out_tok = usage["inputTokens"], usage["outputTokens"]
    cost = (in_tok / 1000 * 0.0008) + (out_tok / 1000 * 0.0032)  # Nova Pro, us-east-1

    span.set_attribute("llm.model", model_id)   # REQUIRED. wrong key = silently zero
    span.set_attribute("span.type", "LLM")
    span.set_attribute("llm.usage.prompt_tokens", in_tok)
    span.set_attribute("llm.usage.completion_tokens", out_tok)
    span.set_attribute("llm.cost.usd", round(cost, 6))

That is the whole idea: read the token usage the SDK already gives you, turn it into dollars against real pricing, and attach it to the span. Everything else in this article is applying this same move across a multi-agent crew and then reading the numbers back. The full wiring (init, the per-agent identity, the tool spans) is in src/crew.py in the repo.

The stack: AWS native and read only

The crew runs on Amazon Nova Pro (amazon.nova-pro-v1:0) through AWS Strands Agents using the agents-as-tools pattern. A supervisor named investigation_run delegates to three specialist sub-agents. Each specialist is a real separation of concerns, owns several read-only tools, and every one of those tools opens its own live span, so in the dashboard you see the agent, then each AWS read nested under it with a real duration. Here is the full fleet and exactly what each agent does.

1. AWS Account Investigator (supervisor, investigation_run)

The orchestrator. It does not touch AWS directly; it reads the user's question, decides which specialists are in scope, delegates to them, and synthesizes one report. On its trace span it records agent.delegated_to (which specialists it called this run), and it stamps the shared session.id that ties the whole investigation together.

The delegation is intent-routed, not fan-out-everything. The supervisor's instructions are strict: call ONLY the specialist whose domain the user actually asked about. Ask only about cost and it delegates to the Cost Analyst alone, while Health & Ops and the Security Auditor never run. Ask only about security and only the Security Auditor fires. Only a whole-account question ("what's running, any risks, and where is my spend going?") lights up all three. This matters for the trace and the bill: agent.delegated_to shows exactly which specialists ran, and a scoped question costs a fraction of a full sweep because the agents you did not need never spent a token. You can watch this live in the control panel: a cost-only prompt lights up one agent and leaves the other two idle.

The supervisor's trace in Traccia: agent.delegated_to records which specialists ran this run, and the shared session.id links the four agents into one investigation.

2. Cost Analyst (cost_analyst)

A read-only FinOps specialist. It builds a full spend picture with three tools, and all three show up as separate tool spans in its trace.

What it reports on a real run: actual MTD spend, the account's forecasted month-end total, the top services by spend, the month-over-month direction and rough percentage, and the single most expensive day compared against the daily average (a possible spike).

The Cost Analyst trace: three Cost Explorer tool spans nested under the agent, each with its real read duration and the agent's own per-agent cost. This per-agent cost figure is exactly what turns a "successful" run into a caught overspend later.

3. Health & Ops (health_ops)

A read-only SRE specialist. It inventories the account and reads health signals with five tools, so it is usually the heaviest agent on input tokens (it chains the most reads).

What it reports: running instances with their CPU, an inventory of volumes, functions, and buckets, and any notable health finding such as an unattached EBS volume.

The Health & Ops trace: five read-only tool spans and, on this run, the highest token count of the fleet (4,772) because it chains the most reads.

4. Security Auditor (security_ops)

A read-only security specialist. It runs four independent checks, each its own tool span.

What it reports: each finding stated plainly with its risk, and it explicitly says so when a check comes back clean.

The Security Auditor trace: four independent read-only checks, each its own tool span with a real duration.

Each specialist stamps its own identity onto its trace span, so from a single crew run the dashboard shows four distinct agents with their own token and cost profiles, not one agent logged four times. Each agent runs as its own top-level trace, tied to the others by a shared session.id, and carries production ownership (type, owner, team) from a catalog file. Every call is a describe or get. There is no create, no modify, no delete. The worst thing this agent can do is read a bit too much, which, as you will see, is exactly the waste we want to catch.

Observability comes from Traccia, an OpenTelemetry-native agent-observability SDK.

pip install traccia

Runs $0 by default using a local file exporter. If you set TRACCIA_API_KEY, it pushes spans to app.traccia.ai. No key, no network, no cost. (The repo pins the exact tested version in requirements.txt; the prose stays unpinned so it does not age.)

Nova Pro pricing, pulled live from the AWS Price List API (effective 2026-08-01, us-east-1):

Every dollar figure below is computed from real token counts against these two numbers.

The cost bridge and one gotcha

Here is the part most tutorials skip. Traccia auto-instruments several stacks out of the box (LangChain including BedrockChat, CrewAI, OpenAI Agents, and raw OpenAI/Anthropic/Gemini), and it ships a cost engine with a bundled pricing snapshot that covers Nova and Claude. But there is no Strands integration yet, and it does not hook raw Bedrock converse calls. So for this specific stack, Strands agents-as-tools calling Bedrock directly, you wire the cost in yourself. That is a fair amount of the value proposition for supported frameworks arriving for free, and real manual work for an unsupported one.

Strands hands you the token usage after a run. You read it, compute the cost, and stamp it onto the span, which is exactly the stamp_llm_cost function from earlier. About 40 lines once you handle all four agents and the tool spans; the full version is in src/crew.py.

The gotcha cost me a confused afternoon, and reading the SDK source explained exactly why. Traccia's cost-annotating processor only computes cost for a span when three things are all true: span.type is LLM (or unset), an llm.model attribute is present, and both token counts are set. Miss any one and the processor simply returns, with no error and no warning. I first set llm.request.model (which felt more semantically correct) instead of llm.model, so the processor silently skipped every span, and the "LLM Calls" and "Total Tokens" tiles read zero while my spans clearly had tokens on them. Set llm.model, and the tiles light up. The forgiving fail is reasonable; the fact that it is invisible is the trap. A one-line debug log ("skipping cost: no llm.model") would have saved the afternoon.

No double-counting across agents

The other question that always comes up: if the supervisor calls two sub-agents, and I sum everyone's tokens, am I counting the sub-agent tokens twice?

I wrote probes/probe_doublecount.py to check instead of guessing. Strands runs each sub-agent in its own event loop with its own metrics object. A supervisor's accumulated_usage is exclusive of its sub-agents' tokens. So the arithmetic is clean:

crew total = supervisor + sum(sub-agents)

No subtraction, no overlap, no double-count. Verified, not assumed.

Modeling a multi-agent crew in traces

Once the bridge is in, you get per-agent cost. But here is a design decision worth being explicit about, because most demos hand-wave it: how do you model a supervisor and its specialists in a trace?

You have two reasonable options. You can nest everything under one trace (supervisor is the root, sub-agents are child spans). Or you can give each agent its own top-level trace and tie them together with a shared session.id. I went with the second, because it is what a real production fleet looks like: the Cost Analyst, Health & Ops, and Security Auditor are independently owned, independently operated services. On the Traces page they show up as their own executions, each with its own cost, tokens, and duration; "Group by session" folds them back into one investigation when you want the whole picture.

session 4f4c1ade...  (one investigation, four independent traces)

  investigation_run   AWS Account Investigator   $0.006   delegated -> 3
  cost_analyst        Cost Analyst               $0.004
  health_ops          Health & Ops               $0.005   (highest total tokens: 4,772)
  security_ops        Security Auditor           $0.005
                                          crew total   $0.021

(These are the real per-agent figures from the exported run shown in the trace screenshots above, rounded to the dashboard's own cost tiles; they shift run to run with token usage. The crew total is the investigation_workflow roll-up span, which equals the supervisor's own synthesis plus the three sub-agents, no double-count. Health & Ops carries the highest token count because it chains the most reads, while the supervisor costs about the same because it writes the long final synthesis. The CLEAN and CONTEXT BLOAT numbers later in the article come from separate, labeled runs, so do not expect them to tie back to this one.)

Each agent's own trace still nests its tools underneath it (agent -> tool:running_instances -> the real boto3 call), so you keep the drill-down without pretending four separate services are one call stack. On the Traces page, "Group by session" folds all four agents from one run back into a single investigation, so you can move between the fleet view and the per-agent view without losing either.

(This is a different run from the CLEAN baseline used later; token counts and therefore dollars shift run to run. The point is the per-agent breakdown, not the absolute number.)

Good. Useful. Per-agent cost on its own is becoming common. The reason it matters here is not the number itself but the data model underneath it: once every step carries cost, tokens, cycle count, and an owning agent, you can build the thing that is still rare, which is catching a run that overspends while looking perfectly healthy. That is what the primitives let you build next.

A few things here are easy to get subtly wrong, and I hit them in roughly this order over a couple of days before the trace design held. First, all four agents come from a single crew run in a single process. Traccia bakes the agent identity into the OpenTelemetry resource at init, which is process-level, so my first version labeled every trace with one agent name: three identical "AWS Account Investigator" rows in the dashboard. Reading the SDK's enrichment processor showed that a span-level agent.id / agent.name attribute takes precedence over the process default, so stamping each agent's span with its own identity makes it show up as its own agent. Static ownership (type, owner, team, org) comes from an agent_config.json catalog the SDK auto-discovers, so the dashboard shows a real fleet with owners and teams, not four anonymous rows. No extra processes, no fake agents.

Second, separate traces need a real correlation key or they look disconnected. Every agent stamps the run's session.id, and the supervisor additionally records agent.delegated_to (which specialists it called this run). That is the explicit link that makes four independent traces read as one orchestrated investigation.

Third, the "each agent is its own trace" bit did not happen by wishing, and this one cost me a rebuild. Traccia's span_scope(parent=None) still inherits the current span if one is active, so my agents silently collapsed back into one trace until I detached the OpenTelemetry context before starting each agent's span. One small helper, verified by counting distinct trace IDs in the exported spans.

Fourth, the first time I looked at the timeline every tool span was 0ms, because I was reconstructing tool spans after the fact from the metrics object. The fix was to wrap the real boto3 call in a live span while it runs, so the timeline shows each AWS read's true duration. A 0ms bar is the kind of thing that makes a viewer distrust the whole trace, and it is worth chasing down. Then a subtler follow-on bit me: those live tool spans inherited the process-level default identity, so every tool bucketed under the supervisor and the specialists looked trace-thin. I had to stamp each tool span with its calling agent's identity too. Nothing about that was in the docs; I found it by parsing the exported traces.jsonl and noticing the agent.id was wrong.

One more touch that reads as production, not demo: each agent records both agent.tools_available (the full toolset it was granted) and agent.tools_called (what the model used this run). On this run, Cost Analyst had two tools available (month_to_date_cost and cost_forecast) and used one; Health & Ops had five and used all five. That gap is not a bug to hide, it is real information. "Has two, used one" is exactly the kind of thing you want visible when you are deciding whether an agent is over-provisioned.

The unique part: catching silent waste

Here is my disclaimer up front. I saw versions of all three of these in real runs while building the crew, then engineered them in src/waste_demo.py to trigger reliably so you can watch them on demand instead of waiting for a bad run. That reproduction is on purpose. LLM output is non-deterministic, so in production the same patterns show up on their own, just not on a schedule you can demo. And critically: detection here is delta-vs-baseline, not magic absolute thresholds. That is how real regression detection works. You capture a known-good run, then flag runs that deviate. Every number below is real Nova Pro token usage from a representative run. Your numbers will vary; the ratios are what hold.

First, the clean baseline. This is the "known good" I compare everything against.

CLEAN baseline (one exported run): crew total ~ $0.0083
  health_ops: 2,274 input / 199 output / 3 cycles

Scenario 1: The runaway loop

The agent gets stuck re-reasoning and re-reading the same things.

RUNAWAY LOOP: ~1.2x baseline ($0.0100 vs $0.0083)
  health_ops ran 4 cycles (baseline: 3)
  re-read cpu_utilization twice (baseline: once)
  health_ops input tokens ~1.6x (3,557 vs 2,274)

Same final answer. The APM span is 200 OK. What catches it: agent.cycle_count and tool.call_count. The agent looped more than its baseline and called the same tool repeatedly. No single number is "wrong." The delta is wrong.

Scenario 2: Redundant tool calls

Milder, sneakier. The agent calls a tool it already has the answer for.

REDUNDANT TOOL CALLS: ~1.03x baseline ($0.0085 vs $0.0083)
  running_instances called 3x (baseline: 1x)

A few percent on one run is the kind of thing you never notice. Multiply it across thousands of daily runs and it is a line item. The signal: tool.call_count for running_instances jumped from 1 to 3. Only visible per-tool, per-agent, and easy to miss precisely because the dollar delta is so small on a single run.

Scenario 3: Context bloat (the expensive one)

The agent drags too much context into its prompts. Every extra token in gets paid for, and it cascades.

CONTEXT BLOAT: ~1.4x baseline ($0.0119 vs $0.0083)
  health_ops input tokens elevated, output nearly 4x (803 vs 199)
  supervisor synthesis cost rises too, bloat cascades

This is the meanest one because it compounds. The sub-agent's bloat feeds a bigger blob to the supervisor, whose own synthesis cost then rises too (on this run the supervisor jumped from $0.0032 to $0.0044). The signal: llm.usage.prompt_tokens and llm.cost.usd per agent. You watch prompt tokens creep up where the work did not.

Same answer, different bill

src/compare.py puts CLEAN next to BLOAT side by side.

Two runs. Both return the right answer. Both are green in any latency-and-errors dashboard. The only place the extra 43% shows up is in the trace, on the per-agent cost attribute you stamped yourself. In the Traccia dashboard this is the moment the tool earns its place: two runs sit side by side, both "successful," and the per-agent cost column is where the bloated one gives itself away. That is the whole argument for agent-native observability in one table.

The detection logic is not clever. It is a delta check: for each agent, compare this run against the baseline and flag three things. More cycles than baseline means a possible runaway loop. Prompt tokens more than 1.25x baseline means possible context bloat. Any tool called more times than baseline means a possible redundant call. That is the whole detector, about fifteen lines in src/compare.py.

The intelligence is in having the baseline and the per-agent attributes to compare against. The trace is what makes those attributes exist.

This detector broke once during the build, and it is a good example of how instrumentation and detection are coupled. When I switched tools to emit one live span per call (the 0ms fix above), the redundant-call check stopped working, because it had been reading a call_count attribute off a single reconstructed span that no longer existed. I had to change it to count span occurrences per tool name instead. The lesson that stuck: change how you record, and you can silently break how you detect. The baseline caught it, which is the whole point.

Watching it happen: the live control panel

This is the panel you saw in the video above. Traces are the source of truth, but a wall of span JSON is not how you show a crew to a teammate. So the repo ships a small live control panel: a single-page UI that runs the real crew and animates the investigation as it happens.

You type a prompt into a command console, hit Investigate, and the view scrolls down to a graph of the crew. The supervisor sits at the top and the three specialists fan out below it, connected by wires. As the run streams, each agent lights up like a traffic signal: idle, then running (with a live activity line, "Reading Cost Explorer", "Scanning security groups"), then done, and the report reveals at the bottom. Every agent card shows the AWS services it touches as small chips, so a viewer can see at a glance that Cost Analyst reads Cost Explorer and the forecast, Health & Ops reads EC2/EBS/Lambda/S3/CloudWatch, and Security Auditor checks security groups, IAM, S3, and GuardDuty.

The panel has two modes. Live runs the real crew: real Nova Pro calls, real read-only AWS reads, real dollars on the trace, about thirteen seconds. Replay animates a saved run from a committed trace file, deterministically and for free, so you can rehearse the visual as many times as you want without spending a token. Both drive the exact same UI from the same event stream; the only difference is whether the events come from a fresh Bedrock run or a recorded one.

The backend is a small FastAPI app that streams the crew's lifecycle as Server-Sent Events. The important part is that the UI is a thin viewer over the same telemetry the trace records; it is not a second, hand-maintained source of truth. What the graph shows is what the crew did.

Build your own, at zero cost and read only

You do not need a paid plan or a live AWS bill to try this. The whole thing runs locally with the file exporter and read-only AWS credentials.

The permission surface is deliberately small: every action is a Get, List, or Describe, across Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, and GuardDuty. There is no create, no modify, no delete anywhere in the toolset. The full policy JSON is in the repo README; if you would rather not hand-roll it, AWS's managed SecurityAudit and ViewOnlyAccess policies cover the same set. Attach it, invoke Nova Pro through Strands, and you have a crew that can look but never touch. To send traces to the hosted dashboard, set TRACCIA_API_KEY; leave it unset and everything writes to a local file. Same spans either way.

The read-only shape is the same for every tool: wrap the real boto3 describe/get/list call in a live span so its duration in the trace is the true AWS read time, return the fields you need, touch nothing. src/tools.py in the repo has all seven; they are all this shape.

An honest take on Traccia

I shipped a real crew against this SDK and read its source to understand the behavior, so here is the assessment grounded in that, not in the marketing page.

What is genuinely good:

• It is OpenTelemetry-native. Spans, processors, and resource attributes are standard OTel underneath, so the data model is not proprietary and you are not locked in.
• It runs at $0 and offline by default. With no API key it writes to a local file exporter; set TRACCIA_API_KEY and the same spans push to the hosted dashboard. Same spans either way, which made local development and CI painless.
• It ships more than a tracer. There is a real cost engine with a bundled pricing snapshot (covering Nova and Claude, among others), a staleness warning when that snapshot ages, and auto-instrumentation for LangChain, CrewAI, OpenAI Agents, and the raw OpenAI/Anthropic/Gemini clients. If you are on one of those stacks, a lot of what I wired by hand would have come for free.
• The span-level agent identity model is the best part. A span-level agent.id and agent.name override the process default, which is precisely what let a single-process crew render as a four-agent fleet with real per-agent cost. That is a thoughtful design decision, not an accident.

Where it made me work, and where it could be better:

• No Strands integration yet, and it does not hook raw Bedrock. For this stack the cost bridge was manual. That is fine and it gives you control, but a Strands integration would remove the single biggest chunk of setup for AWS-native builders.
• The cost processor fails silently (at the time of writing). It skips a span with no error if llm.model is missing or span.type is not LLM. That forgiving behavior is defensible, but the silence cost me an afternoon of a zeroed dashboard. A debug log on skip would fix it outright, and it is the kind of small papercut an early-stage tool usually closes fast.
• A couple of sharp edges are only discoverable in the source (at the time of writing). span_scope(parent=None) still inherits the current context (so separate agent traces silently merge unless you detach first), and span_scope is not a context manager (you call .end() yourself). Neither is obvious from the docs today.
• Documentation is the real gap. I learned the identity precedence, the llm.model requirement, and the context-detach behavior by reading the SDK, not the docs. For a bootstrapped, early-version product that is understandable, and the SDK itself is readable enough that this was possible. But better docs would turn a day of spelunking into an hour.

Net: for supported frameworks you get a lot for free, and even off the beaten path the OTel foundation and the cost/identity model are solid. The capability is there; the polish that is missing is mostly documentation and a few developer-experience papercuts, which is exactly what you would expect from a product at this stage.

Honest caveats

I want to be straight about the limits, because that is the whole point of this article.

• I saw versions of the three waste scenarios in real runs first; the src/waste_demo.py versions just make them fire on cue. LLM output is non-deterministic. In real life these patterns appear on their own, just not on a schedule you can demo.
• Detection is delta-vs-baseline, not fixed magic thresholds. You need a known-good run to compare against, same as any regression system.
• Every dollar is real Nova Pro token usage against verified us-east-1 pricing, but the exact numbers shift run to run. Do not treat any single figure as a constant. Treat the relationship (roughly 1.4x on the worst pattern I measured) as the lesson, not the exact decimals.
• Traccia does not auto-instrument Strands or Bedrock. The cost bridge is about 40 lines you write and own. That is a feature: you control exactly what goes on the span.
• I model each agent as its own trace, grouped by session.id. That is a deliberate choice to match how a real fleet is owned and operated. If you prefer one nested trace per run, keep the supervisor as the parent instead of detaching the context. Both are valid; pick the one that matches how your team reasons about the system.
• The agent is read-only by IAM policy, not by hope.

The takeaway is not "buy an observability tool." It is that a correct-looking answer tells you nothing about whether the run was efficient, and the only place the truth lives is in the trace, on attributes you have to put there on purpose.

FAQ

What is AI agent observability, and how is it different from LLM monitoring?
LLM monitoring usually watches one model call: latency, errors, maybe token count. Agent observability watches a whole reasoning session: how many cycles an agent took, which tools it called, the cost of each step, and, in a multi-agent crew, which agent did what. Agent failures show up across a multi-step chain, not on a single call, so you need the full trace to see them.

How do I track per-agent cost on Amazon Bedrock?
Bedrock returns token usage after each call. You multiply input and output tokens by the model's per-1K price (for Nova Pro in us-east-1, $0.0008 in and $0.0032 out) and attach that dollar figure to the trace span for the agent that made the call. That is the stamp_llm_cost function in this article. AWS's own tag-based cost allocation in Cost Explorer works at the account and tag level; per-agent, per-reasoning-cycle attribution is what the trace adds on top.

Can AWS Cost Explorer show per-agent cost by itself?
Not on its own. Per AWS's Well-Architected Agentic AI Lens, the default state is that costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. Tag-based allocation plus AgentCore Observability improves this, but per-agent and per-reasoning-phase attribution comes from instrumenting the trace, which is what this build does.

Why does my multi-agent app cost more than I expected even when it works?
Because a correct answer is not a cheap answer. Agents can loop an extra reasoning cycle, re-call a tool they already have the answer for, or drag bloated context from step to step. None of that returns an error; it just adds tokens. The overspend shows up on the bill, not in a latency-and-errors dashboard, which is the "silent waste" this article is about.

Do I need a paid tool or an AWS account to try this?
No. The whole build runs at $0 locally: the Traccia SDK writes traces to a local file with no API key, and the AWS reads use read-only credentials (or the committed replay run, which needs no AWS access at all). You only need a Bedrock model grant if you want to run the live crew against your own account.

Is Traccia open source?
The SDK (traccia-py) is open source under Apache-2.0 and built on OpenTelemetry, so the spans are standard OTel and you are not locked in. The hosted dashboard at traccia.ai is the optional commercial part; you only reach for it when you want the visualization.

Does Traccia support AWS Strands Agents out of the box?
Not at the time of writing. It auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, but not Strands or raw Bedrock converse, so on this stack you stamp cost onto the span yourself (about 40 lines). On a supported framework, most of that is automatic.

Try it

The full code (crew, tools, waste demos, the live control panel, the compare view, and the double-count probe) is on GitHub: ai-agent-observability-aws. There is also a live replay of a run you can click through in the browser: https://simplynadaf.github.io/ai-agent-observability-aws/. Clone it, run python -m src.waste_demo with local export, and watch a perfect answer cost you ~1.4x. Then go instrument your own agents before your bill does the teaching for you.

To put Traccia under your own agents, the on-ramp is deliberately short and free:

1. Install the SDK (open source, Apache-2.0): pip install traccia. With no API key it writes traces to a local file, so you can see spans at $0 before you sign up for anything. Source and docs: github.com/traccia-ai/traccia-py.
2. Stamp cost onto your spans using the ~40-line stamp_llm_cost pattern above (or get it for free if you are on LangChain, CrewAI, or the OpenAI/Anthropic/Gemini clients, which Traccia auto-instruments).
3. See it in the dashboard when you want the visual per-agent cost and the side-by-side run compare: set TRACCIA_API_KEY and the same spans push to traccia.ai. Same spans either way, so nothing about your instrumentation changes.

If you build something with it, tell me what silent waste you found. That is the interesting part.

Follow me for more on AWS architecture, DevOps, and AI Infrastructure:
Portfolio | LinkedIn | Dev.to | YouTube | Email | AWS Builder Center | X