How I Added OpenTelemetry Tracing to 47 Services With Claude Code in 9 Days

TL;DR

I had 47 backend services that emitted logs but no traces, so every incident turned into a cross-repo grep party. I used Claude Code to instrument all of them with OpenTelemetry in 9 days — not by asking it to "add tracing," but by writing one reference implementation by hand and turning our conventions into an executable validator the agent had to pass. Here's the loop, the code, and the five things I'd do differently.

The Problem

Our backend is 47 services: mostly Node.js 22.x with Express or Fastify, plus a handful of Python 3.13 FastAPI services. Every one of them had structured JSON logs. None of them had distributed tracing.

That's a fine setup right up until a request crosses four service boundaries and gets slow somewhere. Then the debugging session looks like this:

1. Find the request ID in the edge service logs.
2. Hope the next service logged the same request ID.
3. Discover it called the header x-request-id instead of x-correlation-id.
4. Give up and add a console.log.

Our median incident spent 40+ minutes just on "which service is actually slow." I timeboxed a manual fix and estimated it honestly: 47 services × roughly half a day each for setup, span conventions, config, and a smoke test, comes out around six weeks of focused work. Nobody was going to fund six weeks of plumbing.

The constraint that made this interesting: tracing instrumentation is repetitive but not identical. Auto-instrumentation gets you 70% of the way, and then every service has its own bespoke 30% — a custom HTTP client, a queue consumer, a cron entrypoint that isn't a request at all. That mix is exactly where an AI coding agent is strong and exactly where it will quietly invent conventions if you let it.

How I Solved It

Step 1: Write the reference implementation by hand

My first attempt was a 2,000-word CONVENTIONS.md describing how spans should be named, which attributes were required, and how to wire the SDK. The agent followed maybe 70% of it and improvised the rest — different attribute names in half the services, startSpan in some places and startActiveSpan in others.

So I threw the prose away and instrumented one service by hand, carefully, and made that service the spec:

// tracing.js — the reference every other service was told to mirror.
// Imported for side effects before anything else in the process.
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { resourceFromAttributes } from '@opentelemetry/resources'
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME,
    [ATTR_SERVICE_VERSION]: process.env.GIT_SHA ?? 'dev',
    'deployment.environment.name': process.env.APP_ENV ?? 'local',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.OTEL_COLLECTOR_URL}/v1/traces`,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Noisy, zero-signal, and it doubles span volume.
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
  ],
})

sdk.start()
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)))

Then the one piece of manual instrumentation that auto-instrumentation can't do for you — wrapping a domain operation:

import { trace, SpanStatusCode } from '@opentelemetry/api'

const tracer = trace.getTracer('billing')

export async function chargeInvoice(invoiceId, amountCents) {
  // Span name = "<domain>.<operation>", low cardinality, never includes an ID.
  return tracer.startActiveSpan('billing.charge_invoice', async (span) => {
    try {
      span.setAttribute('billing.invoice_id', invoiceId)
      span.setAttribute('billing.amount_cents', amountCents)
      const result = await gateway.charge(invoiceId, amountCents)
      span.setAttribute('billing.gateway_status', result.status)
      return result
    } catch (err) {
      span.recordException(err)
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message })
      throw err
    } finally {
      span.end()
    }
  })
}

Roughly 90 lines of real, working, opinionated code. Handing the agent that file and saying "do this, for this service" worked dramatically better than any description of it. Agents pattern-match. Give them a pattern.

Step 2: Turn the conventions into a script that fails

Prose conventions are unenforceable, and "looks right to me" doesn't scale across 47 pull requests. So I spent half a day writing a validator and told the agent it was not done until the validator passed:

# check_tracing.py — run per service in CI. Exit 1 blocks the PR.
import re, sys, pathlib

FORBIDDEN_ATTR = re.compile(
    r"setAttribute\(\s*['\"](?:.*\b(?:email|user_id|token|url|path|query)\b.*)['\"]"
)
SPAN_NAME = re.compile(r"startActiveSpan\(\s*['\"]([a-z0-9_]+\.[a-z0-9_]+)['\"]")
TEMPLATED_SPAN_NAME = re.compile(r"startActiveSpan\(\s*[`'\"].*\$\{")

def check(service: pathlib.Path) -> list[str]:
    errors = []
    if not (service / "tracing.js").exists():
        errors.append("missing tracing.js bootstrap")
    for f in service.rglob("*.js"):
        src = f.read_text()
        if TEMPLATED_SPAN_NAME.search(src):
            errors.append(f"{f}: interpolated span name (cardinality bomb)")
        for m in FORBIDDEN_ATTR.finditer(src):
            errors.append(f"{f}: high-cardinality or PII attribute: {m.group(0)}")
        for m in SPAN_NAME.finditer(src):
            if m.group(1).split(".")[0] != service.name.replace("-", "_"):
                errors.append(f"{f}: span namespace != service name: {m.group(1)}")
    return errors

if __name__ == "__main__":
    problems = check(pathlib.Path(sys.argv[1]))
    print("\n".join(problems) or "tracing conventions OK")
    sys.exit(1 if problems else 0)

This single file changed the economics of the whole project. Before it, I reviewed taste. After it, I reviewed judgment calls only — the validator caught every mechanical violation before I ever opened the diff.

Step 3: Prove the spans actually arrive

A service can pass every static check and still export nothing, because the SDK started after the HTTP framework was imported, or the collector URL was wrong. So each service also got a smoke test that boots the process against a local collector and asserts on real exported spans:

// tracing.smoke.test.js
test('http request produces a parented db span', async () => {
  await request(app).get('/invoices/42').expect(200)
  await flushSpans()

  const spans = collector.finished()
  const http = spans.find((s) => s.name === 'GET /invoices/:id')
  const db = spans.find((s) => s.name.startsWith('pg.query'))

  expect(http).toBeDefined()
  expect(db?.parentSpanContext?.spanId).toBe(http.spanContext().spanId)
  // The route is templated, so no invoice IDs leak into span names.
  expect(http.name).not.toMatch(/42/)
})

That parent-child assertion caught the single most common real failure: context propagation silently broken, producing 47 disconnected single-span "traces" that look fine in a list view and are worthless in a waterfall.

Step 4: Run the loop

flowchart LR
    A[Pick next service<br/>grouped by framework] --> B[Agent instruments<br/>against reference impl]
    B --> C{check_tracing.py}
    C -- fail --> B
    C -- pass --> D{smoke test<br/>vs local collector}
    D -- fail --> B
    D -- pass --> E[Human review:<br/>judgment calls only]
    E --> F[PR merged]
    F --> A

The two validators ran inside the agent's own loop, so most iteration happened without me. My job was the last box: does this service's custom queue consumer actually deserve a span here, and is this attribute worth its storage cost?

Nine days, 47 services, about 90 minutes a day of my attention. The first six services took four of those days. The remaining 41 took five.

Lessons Learned

1. A working reference implementation beats any style guide

My 2,000-word convention doc produced ~70% compliance. A 90-line reference file produced near-perfect structural compliance immediately. If you find yourself writing a long prose description of how code should look, stop and write the code instead — then point at it.

2. Make the convention executable or it doesn't exist

The validator was the highest-leverage half-day of the project. Anything you'd write as "please always..." in a prompt should be a script that exits non-zero. Prompts are advice; exit codes are physics. It also means the convention survives you: the next person to add a service gets the same enforcement without reading a single doc.

3. Cardinality is where an agent will hurt you

Left alone, the agent wrote genuinely reasonable-looking code like startActiveSpan(`charge invoice ${invoiceId}`) and span.setAttribute('user.email', email). Both are defensible as "descriptive." Both are catastrophic — unbounded span names wreck your backend's indexing and cost, and PII in attributes is a compliance incident that lives in your telemetry store for the retention period.

The agent isn't being careless; it's optimizing for readability because nothing told it to optimize for cardinality. That tradeoff is invisible in the diff and expensive in production, which makes it exactly the kind of rule that belongs in a linter rather than a review comment.

4. Adding instrumentation is mechanical; removing the old logging isn't

My original plan included "delete the now-redundant logging." I cut that from the agent's scope on day two. Deciding whether a log line is redundant with a span requires knowing who greps for it at 3am — that's tribal knowledge the codebase doesn't contain, and an agent confidently deleting an on-call runbook's load-bearing log line is a bad trade against the time it saves.

Split the work by information availability, not by difficulty. The mechanical 90% went to the agent. The 10% that needs context living in someone's head stayed with people.

5. Batch by framework, not alphabetically

I started in repo order and burned tokens re-establishing context every single service. Grouping by stack — all Fastify, then all Express, then all FastAPI — meant each batch reused the same mental model, the same gotchas, and the same recently-solved problems. Same work, noticeably fewer tokens and far fewer wrong turns.

Adjacent tasks are cheaper than shuffled tasks. Order your backlog by similarity, not by convenience.

What's Next

Three things I'm working on now:

• Sampling policy. We're at head-based 100% sampling in staging and it's already loud. Tail-based sampling that keeps every error trace and 5% of the boring ones is the obvious next move.
• Trace-driven perf work. Now that waterfalls exist, I want to feed real slow traces back to the agent as the starting point for optimization instead of vague "this endpoint feels slow" prompts.
• Generated service dependency graphs. The traces already encode the real call graph, including the three calls nobody remembered existed. Rendering that automatically beats maintaining an architecture diagram by hand.

The pattern generalizes well beyond tracing. Any sweeping, repetitive, convention-heavy migration — feature flag SDKs, error reporting, auth middleware — fits the same shape: one hand-written reference, one executable validator, one smoke test that proves it's live, then let the agent grind.

Wrap-Up / CTA

If you're sitting on a "someday" observability migration because it's six weeks of boring work, it probably isn't six weeks anymore. But the leverage isn't in the prompt — it's in the reference implementation and the validator you write before you start.

If you try this, I'd genuinely like to know what your validator ends up catching. Mine caught 31 cardinality violations I would have merged without noticing.

• 💬 Comment below with the worst high-cardinality span name you've shipped. I'll go first: GET /users/8f21c...
• 🚀 Follow me here on Dev.to — I write up these AI-agent engineering experiments as I run them.
• 💡 Try it yourself with Claude Code on one service this week. One service is enough to find out whether your conventions are real or imaginary.