When Logs Lie: Tracing LLM Agents with OpenTelemetry

[ EXECUTIVE TEARDOWN // TL;DR ]

• Emit one span per agent step so a request becomes a readable waterfall — total at the root, a labelled child per stage.
• Use the OpenTelemetry GenAI semantic conventions (gen_ai.*) so any backend renders traces the same way and cost-per-trace is a query.
• Online tracing answers 'fast and cheap right now?'; offline evals answer 'correct?' — they're complementary, not substitutes.

The live RAG concierge on this site ships a structured trace with every request — one span per step, tagged with tokens, cost, latency, model, and outcome — because the first time an agent felt slow I had only scattered logs and no way to prove which step did it. A multi‑step agent that only logs is a black box; wire it with OpenTelemetry and you get a waterfall you can scan and point at the span that blew the budget. In the pattern I call Trinity Architecture, tracing lives in the orchestration layer, not the UI or the model client.

Logs tell you it happened; spans tell you the shape

An agent call fans out: embed the query, retrieve, rerank, generate, verify, maybe loop. When the p95 latency creeps up or the bill doubles, scattered log lines can't tell you which step did it — they have no parent, no duration, no shared trace id. Spans do. Wrap each step and one request becomes a tree: total time at the root, a labelled child per stage, and attributes on each. On IntegrateX we caught a rerank burst exactly this way when canvas executions started queuing; the hunch became one slow span you could fix.

swipe →

A trace is the request's story end to end. Logs tell you a call happened; a span tells you how long it took, what it cost, and which step in the chain blew the latency budget.

The GenAI semantic conventions make traces comparable

OpenTelemetry ships a standard vocabulary for model calls — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. Using the conventions instead of ad-hoc field names means any backend — Grafana, Honeycomb, Langfuse — renders your traces the same way, and cost-per-trace becomes a query over output_tokens rather than a spreadsheet. I route these attributes through a thin Serialization Adapter so telemetry stays lean and comparable without leaking UI or DB shapes across layers.

otel.ts — one span per model call

// wrap the model call in a span tagged with the GenAI conventions
return tracer.startActiveSpan("model.generate", async (span) => {
  span.setAttributes({
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": model,
  });
  const res = await llm.generate(prompt);
  span.setAttributes({
    "gen_ai.usage.input_tokens": res.usage.input,
    "gen_ai.usage.output_tokens": res.usage.output,
    "gen_ai.cost.usd": cost(res.usage),  // derived, for cost-per-trace
  });
  span.end();
  return res;
});

Online tracing is not offline evals — you need both

Observability watches production as it happens: latency, cost, error rates, the actual distribution of traffic. Evals score quality against a fixed set before you ship. They answer different questions — "is it fast and cheap right now?" versus "is it correct?" — and neither covers the other. A trace can tell you a span took four seconds; only an eval tells you the answer it produced was wrong. On streamerOS we even sampled traces to avoid backpressure and keep 60fps renders while offline evals guarded quality.

You can't tune what you can't see. A trace per request turns "the agent feels slow lately" into "the rerank span doubled on Tuesday" — and that's the difference between debugging and guessing.

Tracing is what makes the latency-first target measurable and autonomous loops auditable — it is why the concierge on this page can show live execution traces instead of asking you to trust it. The judgment I bring to a team is that: an agent you cannot see is an agent you cannot operate, so the trace is part of the design, not an afterthought. Continue on the roadmap.

← Lesson 894% CompressionNext Lesson 15 →The AI-Native Portfolio

~/keep-reading

• 9 min readBudgeting the Claude Context Window Before It Truncates YouThe Claude context window is one token budget shared by prompt, tools, thinking, and reply. I count tokens and guard it in code so a reply never truncates.
• 6 min readThe AI-Native Portfolio: Landing Lead Roles by Shipping the SystemA portfolio that lists 'used ChatGPT' reads junior. One that demonstrates systems architecture — RAG, agents, guardrails, latency budgets — reads like the lead they're trying to hire. Here's how to build the second one.
• 9 min readPrompt Caching: the Ordering Discipline That Cuts the BillI treat prompt caching as FinOps: order the static prefix first, keep it byte-stable, and stop re-paying to re-read an 8K preamble on every request.

YK

Yaseen Khatib · AI Systems Architect

Ships autonomous AI products solo — five in the last twelve months. More about Yaseen →

Need an engineer who can build this?

I'm Yaseen Khatib — a Senior Full-Stack AI Engineer (MERN + TypeScript) who ships production AI systems solo. Open to senior and lead roles, remote or on-site.

Get in touch →See what I've shipped

Originally published at yaseenkhatib.streamerosai.com/blog/llm-observability-opentelemetry-tracing/.