From Flat Logs to Execution Trees: Debugging Modern AI Agents

An agent trace is usually written as a sequence of events because append-only data is simple to produce:

span_started
span_started
span_ended
span_started
span_ended
span_ended

Developers do not want to debug that sequence directly. They want to see the causal structure:

research_agent
├─ search_web
├─ query_database
├─ call_finance_api
│  ├─ attempt_1          timeout
│  └─ attempt_2          ok
└─ summarize_results

The event stream is optimized for writing. The execution tree is optimized for understanding. Building a reliable tree requires more than sorting by timestamp: events may arrive out of order, siblings may run concurrently, spans may be incomplete, and retries may fail while the parent operation still succeeds.

The Minimum Event Contract

Every event needs stable trace and span identity. Start events establish parentage; end events establish outcome and duration.

type SpanKind = 'run' | 'model' | 'tool' | 'retrieval' | 'decision' | 'fallback';

type TraceEvent =
  | {
      event: 'span_started';
      traceId: string;
      spanId: string;
      parentSpanId: string | null;
      name: string;
      kind: SpanKind;
      timestampMs: number;
    }
  | {
      event: 'span_ended';
      traceId: string;
      spanId: string;
      timestampMs: number;
      status: 'ok' | 'error' | 'cancelled';
      errorCategory?: string;
      metadata?: Record<string, string | number | boolean | null>;
    };

Timestamps alone cannot establish parentage. Two events that occur next to each other may be siblings, unrelated concurrent work, or operations from different traces.

Assemble Events Into Spans

Start and end events may not arrive together. A buffered exporter can deliver the end first, and a crashed process may never deliver an end at all.

Represent assembly state explicitly:

type AssembledSpan = {
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  name: string;
  kind: SpanKind;
  startedAtMs: number;
  endedAtMs?: number;
  status: 'open' | 'ok' | 'error' | 'cancelled';
  errorCategory?: string;
  metadata: Record<string, string | number | boolean | null>;
};

type AssemblyDiagnostic = {
  code:
    | 'duplicate_start'
    | 'duplicate_end'
    | 'end_without_start'
    | 'span_left_open';
  spanId: string;
};

class SpanAssembler {
  private readonly spans = new Map<string, AssembledSpan>();
  private readonly pendingEnds = new Map<
    string,
    Extract<TraceEvent, { event: 'span_ended' }>
  >();
  readonly diagnostics: AssemblyDiagnostic[] = [];

  accept(event: TraceEvent): void {
    if (event.event === 'span_started') {
      if (this.spans.has(event.spanId)) {
        this.diagnostics.push({ code: 'duplicate_start', spanId: event.spanId });
        return;
      }

      const span: AssembledSpan = {
        traceId: event.traceId,
        spanId: event.spanId,
        parentSpanId: event.parentSpanId,
        name: event.name,
        kind: event.kind,
        startedAtMs: event.timestampMs,
        status: 'open',
        metadata: {},
      };

      this.spans.set(event.spanId, span);

      const pending = this.pendingEnds.get(event.spanId);
      if (pending) {
        this.pendingEnds.delete(event.spanId);
        this.applyEnd(span, pending);
      }

      return;
    }

    const span = this.spans.get(event.spanId);
    if (!span) {
      if (this.pendingEnds.has(event.spanId)) {
        this.diagnostics.push({ code: 'duplicate_end', spanId: event.spanId });
        return;
      }

      this.pendingEnds.set(event.spanId, event);
      return;
    }

    if (span.status !== 'open') {
      this.diagnostics.push({ code: 'duplicate_end', spanId: event.spanId });
      return;
    }

    this.applyEnd(span, event);
  }

  private applyEnd(
    span: AssembledSpan,
    event: Extract<TraceEvent, { event: 'span_ended' }>,
  ): void {
    span.endedAtMs = Math.max(span.startedAtMs, event.timestampMs);
    span.status = event.status;
    span.errorCategory = event.errorCategory;
    span.metadata = event.metadata ?? {};
  }

  finish(): { spans: AssembledSpan[]; diagnostics: AssemblyDiagnostic[] } {
    for (const spanId of this.pendingEnds.keys()) {
      this.diagnostics.push({ code: 'end_without_start', spanId });
    }

    for (const span of this.spans.values()) {
      if (span.status === 'open') {
        this.diagnostics.push({ code: 'span_left_open', spanId: span.spanId });
      }
    }

    return {
      spans: [...this.spans.values()],
      diagnostics: [...this.diagnostics],
    };
  }
}

This assembler buffers an end event until its start arrives. At finalization, unmatched ends and open spans remain visible as diagnostics. It does not invent timestamps or mark incomplete work successful.

For an unbounded live stream, pending events need a size limit and expiration policy. Otherwise malformed or hostile input can create an unbounded map.

Build a Forest, Not Just One Tree

A valid trace normally has one root, but a renderer should handle multiple roots and orphans without crashing.

type SpanNode = AssembledSpan & { children: SpanNode[] };

type TraceForest = {
  roots: SpanNode[];
  orphans: SpanNode[];
  duplicateIds: string[];
};

function buildForest(spans: AssembledSpan[]): TraceForest {
  const nodes = new Map<string, SpanNode>();
  const duplicateIds: string[] = [];

  for (const span of spans) {
    if (nodes.has(span.spanId)) {
      duplicateIds.push(span.spanId);
      continue;
    }

    nodes.set(span.spanId, { ...span, children: [] });
  }

  const roots: SpanNode[] = [];
  const orphans: SpanNode[] = [];

  for (const node of nodes.values()) {
    if (node.parentSpanId === null) {
      roots.push(node);
      continue;
    }

    const parent = nodes.get(node.parentSpanId);
    if (!parent) {
      orphans.push(node);
      continue;
    }

    parent.children.push(node);
  }

  const sortChildren = (node: SpanNode): void => {
    node.children.sort((a, b) => {
      return a.startedAtMs - b.startedAtMs || a.spanId.localeCompare(b.spanId);
    });
    node.children.forEach(sortChildren);
  };

  roots.sort((a, b) => a.startedAtMs - b.startedAtMs);
  roots.forEach(sortChildren);

  return { roots, orphans, duplicateIds };
}

Validate parent links for cycles before recursively sorting or rendering. A malformed trace where A is the parent of B and B is the parent of A can otherwise cause infinite recursion. Cycle detection can use a depth-first search with visiting and visited sets.

Orphans should appear in a separate “unattached spans” section with diagnostics. Hiding them makes instrumentation gaps look like missing work.

Tree Order Is Not Completion Order

Sort siblings by start time for a stable display, but do not imply that one caused the next. Parallel children can overlap completely.

A useful UI combines two views:

• Tree: Parentage, retries, fallbacks, and handoffs.
• Timeline: Overlap, waiting, time to first output, and critical latency.

tree                              timeline
research_agent                    |----------------------|
├─ search_web                     |-----|
├─ query_database                 |-------------|
├─ finance_api                    |--------|
│  └─ retry                           |----|
└─ summarize_results                            |------|

The tree explains why work happened. The timeline explains when it happened.

Do Not Sum Span Durations

Adding every span duration usually overstates trace time for two reasons:

1. Parent spans include the time of their children.
2. Parallel child spans overlap.

If a root lasts 2 seconds and contains two parallel 1-second tools, summing all three spans reports 4 seconds. The wall-clock run still lasted 2 seconds.

Use separate metrics:

• Trace wall time: Root end minus root start.
• Span duration: End minus start for one operation.
• Self time: Span duration minus the union of child intervals, when that analysis is needed.
• Critical path: The dependency path that determines the run’s completion time.

Computing a true critical path requires dependency semantics, not just parentage. Sibling tools may all be required, any one may be sufficient, or one may be cancelled after another succeeds. The trace schema needs to represent those decision rules before a UI can label a critical path confidently.

Model Retries and Fallbacks as Children

Retries are separate attempts with separate outcomes:

load_pricing                  ok
├─ attempt_1                  error: timeout
├─ attempt_2                  error: invalid_response
└─ fallback_to_cache          ok: age_minutes=18

The parent can legitimately succeed even when children fail. Do not automatically propagate the worst child status to the parent. The parent’s status should describe whether the operation fulfilled its contract; child statuses explain how.

Quality gates can still warn when a successful parent depended on stale fallback data or exceeded an attempt budget.

Preserve Partial Traces

Processes crash, clients disconnect, serverless invocations end, and exporters drop events. An open span is evidence, not clutter.

Render incomplete spans distinctly and include the reason when known:

generate_answer     open: completion event missing
stream_response     cancelled: client disconnected
tool_call           unknown: adapter ended before callback

Do not silently close every open span at the trace’s last timestamp. That invents duration and status. A UI may estimate a visible range, but it should label the estimate.

Compare Trees Through Invariants

Execution trees are useful regression artifacts, but exact snapshots are brittle. Compare durable properties:

• Required and forbidden span kinds
• Parent-child relationships
• Attempt and fallback counts
• Terminal status
• Model and token budgets
• Presence of open spans or orphans
• Capability and adapter diagnostics

Ignore random IDs and normalize timestamps. For concurrent siblings, compare sets or parentage rather than one exact ordering.

Keep the Event Store Append-Friendly

Newline-delimited JSON works well for local traces because each event is independently appendable and replayable. A reader can stream events through the assembler rather than loading every run into memory.

Index or partition by trace ID when files become large. Bound retention and avoid storing raw prompts, tool payloads, retrieved documents, credentials, or user data by default. Tree reconstruction needs identity and lifecycle, not complete application content.

Validation Checklist

Before trusting an execution tree, verify:

• All events belong to the expected trace.
• Span IDs are unique.
• Parent IDs resolve or appear as visible orphans.
• Parent links contain no cycles.
• Start and end events are not duplicated.
• End times do not precede start times.
• Open spans remain visible.
• Numeric metadata is finite and bounded.
• Missing adapter capabilities are reported.
• Payload and privacy policy passed before storage.

Validation errors should be separate from agent errors. A malformed trace may describe a successful agent run while still being unusable for debugging.

Final Thought

Flat events are not the enemy; they are the practical storage format. The mistake is treating their arrival order as the execution model.

Assemble lifecycle events into spans, validate identity and parentage, preserve incomplete data honestly, and render both tree and timeline views. Then retries, fallbacks, parallel work, and silent failures become properties of a system you can inspect rather than clues scattered through a terminal transcript.

That is the real shift from flat logs to execution trees: not more telemetry, but trustworthy structure.