For a long time, my mental model of observability was this: you import an SDK, sprinkle some calls through your code, each call fires off data to a server somewhere, and a dashboard reads it back. A logging system with extra steps.
That model is wrong in a specific, interesting way. And I couldn't see how it was wrong until I stopped looking at the dashboards and started looking at what actually gets emitted, and how.
The seductive wrong model
The wrong model is seductive because the plumbing really does look identical. Logging: emit, store, search. Observability: emit, store, query. Same loop, right?
So my working theory became: observability is logging plus some fancy logic to analyze the logs.
Close. But no. The difference isn't in the analysis. It's in the emission — and it splits into three mechanisms that have almost nothing in common with each other.
Descent one: metrics aren't events at all
A metric is not a record you write. It's a number sitting in your app's memory.
requests_total.increment() // 1, 2, 3...
request_duration.record(0.23) // adds to a histogram
Nothing is sent when this line runs. The number just changes in RAM. Periodically — every 15 seconds, say — either a backend scrapes an endpoint your app exposes, or a collector ships the current values out.
That's why metrics are absurdly cheap: a million requests is one counter reading "1,000,000", not a million records. You could never reconstruct a clean p99 latency graph by parsing log text. The histogram was built for it at write time.
And the stateless-container objection answers itself: the in-memory counter is disposable. Each instance flushes to the backend on a schedule (on serverless, a sidecar collector even does a final flush at shutdown), and the backend sums across instances. The durable truth never lived in your app.
Descent two: logs are the familiar part
Logs work exactly the way I always assumed everything worked: an event, written out, shipped, searched.
The only upgrade observability adds is one field, injected automatically:
log.info("payment failed", { trace_id: "abc123" })
That trace_id looks trivial. It's the glue for everything else.
Descent three: the trace, or the part that felt like magic
Distributed tracing was the piece I couldn't demystify. A request hops across five services and somehow the platform shows me the whole path as one tree. Surely something sophisticated is happening.
It isn't.
When service A calls service B, the SDK stuffs the trace ID into the outgoing request. For HTTP, it's literally a header — traceparent, a W3C standard. For gRPC, the same ID rides in the metadata (which is HTTP/2 headers underneath). For a message queue, it goes in the message headers.
Service B reads the header and says: I'm part of trace abc, my parent is span 1. Each service exports its spans independently. The backend reassembles the tree later, matching parent and child IDs like a linked list.
That's the entire trick. No agent watching the network. No distributed coordination. An ID, copied into a header, over and over.
The click
Here's where it snapped into place.
Observability is not one system. It's three mechanically different emitters that happen to share a pipeline:
- a number in RAM, sampled on a timer
- an event, enriched with an ID and shipped
- an ID in a header, propagated hop by hop and reassembled later
And the reason no amount of after-the-fact log analysis can replace it: if the trace ID was never propagated at request time, no analysis can invent it. The structure has to exist when the data is born. The dashboard is downstream of a decision made in a header, milliseconds earlier, inside your request path.
Oh. It's just a counter, a log line with an ID, and a header. That's it.
The fancy part was never the analysis. It was agreeing — across languages, frameworks, and vendors — on what to emit and what to call it. That's what OpenTelemetry actually is: not a tool, a treaty.
Observability is the ability to understand a system's internal state purely from what it emits — so you can answer questions you never thought to ask in advance. It works because the emitted signals carry their relationships with them: metrics as live numbers, logs as events, traces as an ID passed hand-to-hand through every request. The linking happens at emission time; everything after is just reading.
