How to Pick LLM Application Observability Metrics [2026]

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Last year I watched a team burn three days on an “LLM incident” that had nothing to do with the model. p99 doubled, costs spiked, quality dipped. The only thing they had was token counts and a big pile of prompt/response logs. So the debugging session turned into archaeology.

You’ll finish this in about 45 minutes with a working “minimum viable” LLM observability setup: a trace model (LLM + retrieval + tool calls + retries), a short metrics list, and a privacy-safe logging policy you can ship without storing raw prompts.

If you’re only collecting token counts and prompt/response logs, you don’t have LLM application observability metrics. You have an expensive transcript.

This post is about the signals that explain the stuff that actually breaks in production: tool calls that time out, retrieval that quietly returns garbage, caches that “help” by serving wrong answers faster, and safety filters that suddenly start refusing half your traffic.

What is LLM application observability?

LLM application observability is instrumenting an LLM-powered system with traces, metrics, and logs so you can explain latency, cost, failures, and answer-quality changes across the full request path. That means the model call, retrieval, tool/function calls, retries, caching, and safety.

The important part is “full request path”. The model is rarely the product.

I learned this the hard way building the Walmart conversational commerce chatbot at Firework (Zealsight). We handled millions of queries daily with sub-second responses, and the real wins came from tracing the pipeline (retrieval + streaming + orchestration), not staring at token logs. At that scale, “the LLM was slow” is basically never a root cause. It’s a symptom you slap on anything you can’t see.

The minimum signals checklist (copy this into your dashboard)

This is the smallest set of signals I’d accept before calling an LLM feature “production-ready”. Keep it tight. You can always add the fancy stuff later. What you can’t do is debug p99 with vibes.

A note on numbers: graph p50/p95/p99 for anything latency-related. Keep at least 7 days of metrics at full fidelity. Logs can be sampled way harder (often 0.1%–1% for high-volume apps) if you have traces and exemplars.

How to structure LLM telemetry as traces, metrics, and events (so your APM works)

Most teams accidentally create a new observability universe for LLMs. They ship a vendor dashboard, add a giant JSON log blob, sprinkle some notebooks around. Six months later, nobody trusts any of it, and on-call falls back to… reading prompts.

I’m blunt about this: force LLM telemetry into standard primitives.

• Traces answer: “Where did the time go?”
• Metrics answer: “Is it getting worse over time, and for whom?”
• Logs/events answer: “What exactly happened on this one weird request?”

If you already run OpenTelemetry (OTel), you’re most of the way there. I wrote a deeper implementation guide in OpenTelemetry Instrumentation for AI Agents [2026]: Ship It. I’m not going to duplicate the schema here. What you need from this post is a trace shape you can standardize on and the minimum set of attributes that won’t get you hauled into a privacy review.

The trace shape I use for LLM apps

You want one trace per user request, with spans that map to real pipeline boundaries. Not “AI stuff happened here”. Real boundaries.

1. http.request (root span)
2. llm.plan (optional, if you do multi-step / agentic planning)
3. rag.retrieve (optional, if you do RAG)
   • rag.embed_query
   • rag.vector_search (or hybrid search)
   • rag.rerank
4. tool.call.<name> (0..N)
5. llm.generate (the final response)
6. response.stream (optional; track streaming UX separately)

The rule I won’t compromise on: every external dependency gets its own span.

Payments API. Span.

Postgres lookup. Span.

Vector DB call. Span.

If you don’t do this, your p99 chart is just a mystery novel where every suspect has an alibi.

Treat the LLM like a microservice. Because that’s what it is in your architecture.

Minimum span attributes that don’t leak data

I try to avoid raw prompt logging by default. Instead, I put join keys on spans so I can correlate behavior across versions and cohorts.

• request_id (server-generated)
• user_cohort (coarse: free, pro, internal)
• model.name + model.version (or provider + deployment)
• prompt_template_id + prompt_version
• rag.corpus_id + rag.index_version
• tool.name + tool.version
• cache.tier + cache.key_version
• safety.outcome (enum)

Those version fields matter more than people think. Half of the “LLM randomness” incidents I see are actually “we changed an index / prompt / tool schema and forgot to correlate it”. Then everyone argues about temperature settings like that’s the problem.

If you want a concrete example of versioning and pipeline identity, my AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026] post shows how I tie observability back to eval gates.

How to trace and measure tool/function calls (latency, errors, retries, schema mismatch)

Tool calls are where demos go to die.

Your LLM can be perfect and your app still fails because:

• the tool timed out at 2,000 ms
• the LLM produced invalid args 3 times and you retried
• the upstream API returned 429 and you backoff-looped
• the tool “succeeded” but returned an empty payload (a silent failure)

What I measure for each tool

For each tool/function, I want five things:

• Success rate: success / attempts
• Latency: p50/p95/p99 of the tool span
• Retry count distribution (histogram, not a single average)
• Error categories (timeouts vs 4xx vs 5xx vs validation)
• Schema mismatch rate (LLM produced args that didn’t validate)

If you do only one thing: break retries out as their own spans.

Retries are the sneakiest source of “why did p99 double?” because the user only sees the final response. You quietly did 2–4 extra round trips and congratulated yourself for “recovering”.

How to attribute downstream failures to the right span

I see a lot of traces where everything is marked “OK” except the root span. That’s useless.

A simple rule:

• If the tool request failed (timeout / non-2xx), mark the tool span as error.
• If the tool returned data but you rejected it (schema/validation), mark the tool span as error.
• If the tool succeeded but the LLM refused to use it, do not mark the tool span as error. Mark a decision event on the llm.plan or llm.generate span.

This distinction is how you stop blaming infra for product behavior.

If you’re building AI agents or doing agent orchestration, tool-call observability is your real “agent reliability” layer. The model is the easy part.

Cache metrics that actually matter (and how to detect cache-caused regressions)

Caches are a first-class dependency in LLM apps. Pretending they’re just an implementation detail is how you get blindsided.

You usually have at least two of these. Most teams end up with all three:

1. Prompt cache (exact match, or templated)
2. Embedding cache (query embedding reuse)
3. Semantic cache (approximate match, “close enough”)

If you don’t instrument caches, you’ll misread both cost and quality.

A semantic cache can cut cost by 30%–60% in high-repeat workloads. It can also destroy correctness when the query is similar but not identical. Faster wrong answers are still wrong.

I go deeper on cost levers in Reduce LLM API Costs 60%: 6 Techniques [2026] and per-task accounting in Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching. Here I’m focusing on what to observe so you can catch regressions before support tickets do.

Minimum cache metrics per tier

• cache.<tier>.hit_rate
• cache.<tier>.lookup_latency_ms (p95 matters)
• cache.<tier>.stale_served_rate
• cache.<tier>.bypass_rate (how often your app skips cache)
• cache.<tier>.key_version_mismatch_rate

That last one is your regression smoke alarm.

If you change prompt templates or tool schemas and don’t bump cache key versions, your cache becomes a correctness bug factory. You’ll see “great latency, bad answers” and the Slack thread will immediately blame the model.

Detecting cache-caused quality regressions without storing prompts

You can detect most cache regressions with three join keys:

• prompt_version
• cache.key_version
• eval_outcome_bucket (or a proxy)

If you run continuous evals, correlate “bad outcome” with cache hit/miss.

If you don’t have full evals yet, use proxies:

• user rephrase rate within 30 seconds
• user click-through on citations (for RAG)
• “thumbs down” rate
• session abandonment after answer

They’re imperfect. They beat guessing.

RAG observability metrics for production (and how to debug answer quality drops)

Retrieval-Augmented Generation (RAG) is where most “LLM apps” become real products. It’s also where people ship the least observability.

RAG failures are usually boring:

• retrieval returned nothing
• retrieval returned duplicates
• retrieval returned irrelevant chunks
• reranker got slower and you silently lowered k
• index got rebuilt and semantics shifted

On the Walmart chatbot, one lesson kept repeating: retrieval quality dominated answer quality at scale. Model upgrades were incremental. Retrieval regressions were catastrophic.

Minimum RAG observability metrics

• Latency by stage: embedding, search, rerank
• Empty-result rate (k_returned = 0 or usable_chunks = 0)
• Top-k distribution: requested vs returned vs used
• Chunk overlap ratio: duplicates / near-duplicates in the final context
• Citation coverage: how often your answer cites at least 1 retrieved chunk
• Context token budget: tokens spent on retrieved context vs generation

If you want my take on why “just increase the context window” is a trap, see RAG Context Window Limits: Why Bigger Is Not Better [2026].

Debugging playbook: “answer quality dropped”

When someone tells me “quality dropped”, I don’t start by debating model upgrades. I run this sequence:

1. Check rag.empty_result_rate for a spike. If it goes from 0.5% to 5%, that’s your incident.
2. Check rag.search.latency_ms p99. If p99 jumped, some callers will time out and fall back.
3. Compare rag.index_version. If the index rebuilt, correlate the quality change with the version boundary.
4. Look at rag.chunk_overlap_ratio. If overlap jumped, your chunking pipeline probably changed.
5. Check rag.k_used. If you silently started using fewer chunks, you probably introduced a budget clamp.

This is exactly why you track index identity in spans. Without it, you can’t correlate, so you can’t fix.

If you’re doing retrieval-augmented generation seriously, you’ll eventually want offline evals and replay harnesses. My Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates post is the next step.

Refusal rate and safety outcomes (without storing sensitive prompts)

If your refusal rate is “unknown”, you don’t have a production system. You have a liability.

Track these as metrics:

• safety.refusal_rate
• safety.safe_completion_rate
• safety.policy_trigger_rate (by category)

Then correlate them with:

• model.version (providers change behavior)
• prompt_version (you change behavior)
• tool.name (tools can trip policy boundaries)
• user cohort

A refusal spike can come from:

• an upstream model update
• an overly aggressive content filter
• a new tool that surfaces sensitive data
• a prompt injection chain that tries to jailbreak the system

If you’re not threat-modeling this, start here: AI security. I also keep a checklist in AI security that matches how I think about “safe by default” telemetry.

How to classify outcomes without logging prompts

Use an enum outcome plus a coarse reason code.

Example outcomes:

• ok
• refused
• partial
• safe_completion
• tool_blocked

Reason codes can be your taxonomy. Keep it stable and versioned. If you change your categories every month, you’re destroying your own trend lines.

The point is to measure behavior without hoarding content.

Privacy-safe LLM logging: what to store, what to hash, sampling, retention

“Just log the prompts” is the GenAI equivalent of “just run it as root”. It works right up until the day it ruins your week.

I’m not saying “never store prompts”. I’m saying: make prompt storage an explicit, reviewed exception.

I wrote a dedicated privacy playbook in Data Privacy in RAG Redaction and Retention [2026 Playbook] and the security angle in Prevent Sensitive Data Leakage in RAG: The 2026 Playbook. Here’s the minimum policy I’d ship.

Default: store structure, not content

Store:

• prompt template ID + version
• tool names + versions
• retrieval corpus ID + index version
• token counts and cost estimates
• safety outcomes
• hashes of prompts/responses (salted)

Don’t store by default:

• raw user prompts
• raw model outputs
• raw retrieved chunks

If you need debugging content, use short-lived secure capture:

• gated behind a feature flag
• enabled for a single request_id
• TTL of 24 hours (or less)
• access logged and limited to on-call

Redaction and hashing that’s actually useful

• Redact obvious PII before telemetry export.
• Hash the canonicalized prompt (lowercased, whitespace-normalized) with a rotating salt.

Canonicalize because otherwise you can’t correlate duplicates.

Rotate salts because otherwise your logs become a tracking vector.

Sampling strategy for high-volume LLM apps

A sane default I’ve used:

• 100% metrics (cheap)
• 10% traces for baseline, plus 100% traces on errors
• 0.1% logs (structured events), plus “burst” logging for a single incident window

If you’re doing millions of requests/day, logging every prompt is not just risky. It’s expensive.

If you need to reason about cost tradeoffs, I keep my own datasets on pricing and performance. Based on the benchmark and cost data I maintain at kunalganglani.com/llm-prices and kunalganglani.com/llm-benchmarks, the gap between “fast cheap model” and “frontier model” can be multiple times in $/request. Observability that can attribute cost to spans is how you stop arguing in Slack and start making decisions.

Implementation notes: don’t create an observability silo

The goal isn’t “an LLM dashboard”. The goal is: when p99 spikes, your on-call uses the same workflows they use for every other service.

A few practical moves:

• Emit tool spans as normal client spans. Your APM already knows how to graph dependency latency.
• Put LLM metadata on spans as attributes, not log blobs.
• Use exemplars to attach “this trace is a p99 outlier” to your histograms.
• Keep a strict boundary between telemetry and product analytics. They can share IDs, not raw content.

If you’re running production AI systems, treat observability changes like API changes. Version them. Review them. Test them.

The uncomfortable prediction

Within 12 months, “token logs” will be viewed the way we now view console.log in a backend service. Fine for a demo. Embarrassing in production.

If you’re building LLM apps, pick 10–12 signals, wire them into traces/metrics/events, and ship a privacy policy with the same seriousness you ship auth.

Then do the hard part. When your dashboards light up, don’t blame the model. Fix your pipeline.

Originally published on kunalganglani.com