Four different things are called "replay" in our agent runtime. I read the ledger.

The short version

replay means four different things in our codebase. Here they are up front:

All four rest on one thing: the ledger that hits the database first is the authority.

Not the logs. Not the event stream. Not the SSE feed scrolling in your console.
The rows in the tables. That sounds unremarkable until you notice it is what lets three
of those four survive a process restart.

1. Why the word needs splitting

In a demo, these three sentences look like one feature:

• "Here's every step of that run."
• "Lost your connection? Refresh — the missing steps come back."
• "Bad answer? Hit replay."

Engineering-wise they are nothing alike. The first is a read. The second is a
read plus a subscription. The third is a write — it calls the model again,
sends the HTTP request again, spends the money again.

The cost of collapsing them into one word is a user who assumes "replay" is safe
and sends two emails.

So the order below goes from cheapest to most expensive.

2. The tables, and one number that shows up three times

Five tables, all in one file (server/app/kernel/runtime/db/models/runs.py, 396 lines):

runs                    one execution                        Run           :25
run_steps               one step inside it                   RunStep       :120
run_step_tool_calls     execution control for one tool call                :180
run_artifacts           files this execution produced        RunArtifact   :242
run_cost_entries        usage and cost for one metered call                :284

Three more cover long-running work (models/tasks.py, 98 lines): tasks,
task_checkpoints, task_events. Section 7 uses them.

Now the detail worth stopping on: 8192 appears three times in this ledger,
and it means something different each time.

Twice on the run and the step, where summaries are truncated:

input_summary=input_summary[:8192] if input_summary else None,

Once on a tool call result, where anything larger is offloaded to object storage
and the ledger keeps a pointer, a byte count and a sha256:

if len(encoded_result) > 8192:
    ...
    artifact = self.trace_writer.create_artifact(
        run_id=record.run_id,
        step_id=record.run_step_id,
        artifact_type="json",
        storage_key=storage_key,
        mime="application/json",
        size_bytes=len(encoded_result),
        sha256=hashlib.sha256(encoded_result).hexdigest(),
        meta={"kind": "tool_result", "tool_call_id": record.tool_call_id},
    )

The asymmetry is deliberate. Summaries are for humans; losing the tail is fine.
Tool results get reconciled and replayed; losing a byte is not fine.

Section 12 covers a consequence of that asymmetry we have not handled well yet.

3. Replay #1: reassembling the evidence

The cheap one. A GET:

GET /api/v1/observe/runs/{run_id}/replay

One sentence of behaviour: query the five record types by run id, add approvals and
feedback, return the bundle. The implementation
(server/app/modules/observe/application/service.py:212) returns seven keys:

return {
    "run": run,
    "steps": steps,
    "artifacts": artifacts,
    "costs": costs,
    "approvals": approvals,
    "feedback": feedback,
    "trace_spec": to_runtrace_spec(run, steps, artifacts, costs),
}

Six raw record sets, plus trace_spec — the same data flattened into something you can
hand to a tracing backend (kernel/runtime/runs/exporter.py:88). That spec carries two
rollups alongside the timeline: usage_summary (prompt tokens, completion tokens,
embeddings, reranks, milliseconds, storage bytes, requests, vectors) and charge_summary
(amounts grouped by currency).

Nothing here executes. No model call, no tool call, no cost. It is a database read,
so you can call it at any point after the run ended, and the ten-thousandth call costs
what the first one did.

Every query carries tenant_id and workspace_id in its where clause — reading another
workspace's ledger is closed off at the SQL level, not at a middleware you can misconfigure.

4. Replay #2: catching up after the connection drops

The second-cheapest, for the "tab is open, wifi died" case:

GET /api/v1/runs/{run_id}/stream?last_event_id=st_xxxx

Handled at server/app/api/v1/workflow/streaming.py:401. The part that matters:

if last_event_id:
    step_query = select(RunStep).where(
        and_(
            RunStep.id == last_event_id,
            RunStep.run_id == run_id,
            ...
        )
    )
    last_step = db.exec(step_query).first()
    if last_step:
        last_step_time = last_step.created_at
        known_step_ids.add(last_step.id)

steps_query = select(RunStep).where(
    and_(
        RunStep.run_id == run_id,
        ...
        RunStep.created_at > last_step_time if last_step_time else True,
    )
).order_by(RunStep.created_at)

Look at where it reads from: select(RunStep). The database. Not an in-memory ring
buffer, not a broker offset.

That choice buys a specific property: you can reconnect an hour after the run finished,
hand over your last_event_id, and still get the steps you missed. An in-memory buffer
cannot do that — a restart empties it. A broker can, but then you need a broker.

The SSE id: field is the step's primary key (streaming.py:432), so the Last-Event-ID
that browsers resend automatically is already a row id in the ledger. No second cursor
scheme to keep in sync.

One more detail worth borrowing: that query sets populate_existing=True, with a comment
explaining why — the execution side writes from its own session, so this tailer has to
bypass anything its own session cached earlier. That is the kind of line nobody can
reconstruct three months later without the comment.

5. Replay #3: the same idempotency key, twice

This one happens below the surface, inside the tool gateway.

Every tool call gets a run_step_tool_calls row. The table carries three unique
constraints (models/runs.py:182):

UniqueConstraint("tenant_id", "workspace_id", "run_step_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "run_id", "tool_call_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "idempotency_key", ...)

The third is the interesting one. When the same key arrives again and the row is already
terminal:

if existing.status in {"succeeded", "failed"}:
    payload = existing.result_json or {}
    ...
    return ToolExecutionClaim(
        record=existing,
        run_step=step,
        replayed=True,
        cached_response=ToolResponse(
            result=payload.get("result"),
            success=existing.status == "succeeded",
            error=existing.error_message,
            metadata={..., "idempotent_replay": True},
        ),
    )

Last time's result comes back; nothing leaves the process. The metadata carries
idempotent_replay: True so callers can tell this apart from a fresh execution.

If the earlier result was large enough to live in object storage,
load_cached_response (tool_calls.py:636) fetches the artifact — after checking tenant,
workspace, run and step all match, and raising Tool result artifact scope mismatch
if any of them does not.

The point of this layer: replay #4 is only safe to offer because this one exists.
When you re-run, the tool calls whose idempotency keys did not change are not actually
executed a second time.

6. A status that admits we don't know

This is the design I would point at first if someone asked what is unusual about this
ledger.

Claiming a tool call takes a lease (60 seconds by default, widened by the gateway to the
tool's timeout). An expired lease means the executor may be dead. Retry or not?

The code answers by asking whether the request actually left (tool_calls.py:309):

lease_expired = (
    existing.lease_expires_at is not None
    and _aware_utc(existing.lease_expires_at) <= now
)
if lease_expired and existing.outbound_started_at is not None:
    existing.status = "in_doubt"
    ...
    raise ConflictError("Tool call outcome is in doubt")
if lease_expired and existing.outbound_started_at is None:
    existing.status = "claimed"
    existing.attempt_count += 1
    ...

Two branches, split on one field, outbound_started_at:

• Died before going out — safe. Re-claim, bump the attempt count.
• Died after going out — mark it in_doubt, do not retry, park the step at paused, raise a conflict.

The second branch is the honest one. On the other end is a real system: an order endpoint,
an email, a transfer. The request left and no response came back.
We don't know whether it happened, so we don't guess. The ledger records "in doubt"
and a human decides.

Auto-retrying here is wrong in the specific way that only surfaces when someone gets two
copies of the same email.

7. Replay #4: actually running it again

The expensive one. Three separate paths, three different mechanisms.

(a) Workflows: replay and retry

POST /api/v1/workflows/{workflow_id}/runs/{run_id}/retry
POST /api/v1/workflows/{workflow_id}/runs/{run_id}/replay

The two implementations differ by one check
(modules/workflow/application/service.py:804 and :821): retry requires the source run
to be failed or canceled; replay does not. Both load the original inputs, execute
again, and put source_run_id and control_action in the response.

(b) Agent tasks: replaying a persisted snapshot

More interesting (server/app/wiring/task_drivers.py:82). Rather than "take the inputs and
run", it loads the previous ResponseInteraction snapshot and deliberately strips the
identifiers that belonged to the failed attempt before queueing a new one:

execution_json["assistant_message_id"] = generate_thread_message_id()
payload = dict(execution_json.get("payload") or {})
if payload:
    # Drop identifiers that belong to the attempt being replaced so the
    # replay creates its own response, run and task.
    payload.pop("task_id", None)
    payload.pop("run_id", None)

The old task is then moved to CANCELED with a forward pointer,
retried_as_interaction_id, in its progress payload. The comment is blunt about why:
leaving it queued would report work this task will never perform.

If there is no snapshot, it does not improvise — it fails explicitly with a dedicated
error code, SNAPSHOT_MISSING_ERROR_CODE. No evidence, no replay. I like that one.

(c) Knowledge ingestion: lineage that actually lands in the ledger

The only one of the three that writes the lineage into runs
(modules/knowledge/application/runtime_service.py:848):

run = self.trace_writer.create_run(
    ...
    source_run_id=previous_run.id,
    attempt_no=max(previous_run.attempt_no + 1, task.retry_count + 1),
    request_id=f"knowledge-ingest:{task.id}:{task.retry_count + 1}",
)

runs has both source_run_id and attempt_no, plus a dedicated index,
ix_runs_scope_source_created. This path uses them.

The other two do not. That is item ① in section 12.

8. Why the ledger is trustworthy

Three reasons, all in the code.

Status changes are conditional UPDATEs, not read-modify-write.
The where clause at writer.py:375 carries the old value:

result = self.db.execute(
    update(Run)
    .where(
        Run.id == run_id,
        Run.tenant_id == self.ctx.tenant_id,
        Run.workspace_id == self.ctx.workspace_id,
        Run.status == old_status,
    )
    .values(**values)
    ...
)
if result.rowcount != 1:
    ...
    raise RuntimeTransitionError(f"Concurrent run transition rejected: {old_status} -> {target_status}")

Two executors racing to change the same run: one wins, the other sees rowcount != 1
and is rejected. Not last-write-wins — someone jumped the queue, so error out.

Success is an irreversible terminal state.
From the transition table in kernel/runtime/status.py:

ExecutionStatus.SUCCEEDED: frozenset(),
ExecutionStatus.FAILED: frozenset({ExecutionStatus.RETRYING}),
ExecutionStatus.CANCELED: frozenset({ExecutionStatus.RETRYING}),
ExecutionStatus.EXPIRED: frozenset({ExecutionStatus.RETRYING}),

SUCCEEDED reaches nothing. A success written into the ledger cannot be walked back,
not even to failed. Failures can move to retrying; successes go nowhere.

Outbound notification goes through a transactional outbox, not a live broadcast.
Creating a run and every status change write an outbox row (writer.py:282 and four other
sites) inside the same database transaction as the business data.

The live event bus, by contrast, is best-effort — the last line of _emit_event is:

except Exception:
    return

Swallowed. That is the right call: a failed notification must never block the ledger
write. It also means one thing for anyone verifying behaviour —
reconcile against the ledger, not against what you saw on the event stream.

9. The ledger belongs to the ports, not to the loop

The previous piece argued that governance is a property
of the port rather than of the agent loop. This one adds a parallel claim.

Count who writes to TraceWriter:

Five kernel ports, five policy gateways, one ledger.

Which means: you do not instrument the agent loop, and you do not instrument the
workflow engine. If an operation left through a port, it left a row. The agent loop
calling tool_port.invoke leaves one; a DAG workflow's tool node calling the same
tool_port.invoke leaves one — in the same table, with the same schema.

The converse holds too, and it is the real boundary of this design:
a call that bypasses the ports leaves nothing in the ledger. That is not a bug, it is
what layering means. The ledger records governed operations, not everything the process
happened to do.

10. The boolean the platform computes for you

Mechanism aside, the question a user actually has is simpler: is there enough evidence
for this run?

GET /api/v1/runs/{run_id} returns thirteen governance evidence items
(kernel/runtime/runs/service.py:530 onward):

actor_scope        subject_version     capability_binding   permission_scope
secret_boundary    egress_policy       audit_record         cost_attribution
trace_timeline     tool_call           knowledge_citation   child_workflow
replay_ready

The last one is the boolean. Its criteria are at service.py:516:

replay_missing: list[str] = []
if not steps:
    replay_missing.append("steps")
if response_timeline_applicable and not response_events:
    replay_missing.append("response_events")
if cost_attribution_applicable and not cost_entries:
    replay_missing.append("costs")
if knowledge_citation_applicable and not citations:
    replay_missing.append("citations")
if tool_governance_applicable and not tool_calls:
    replay_missing.append("tool_calls")
if tool_governance_applicable and not audits:
    replay_missing.append("audits")

Note the _applicable guards: it judges against what this run actually did. A pure
chat run has no tool calls and is not marked deficient for lacking them. A run that did
call a tool, but has no matching audit rows, comes back fail — and names the missing
category in missing.

I find that more useful than a docs page promising "full replay". It is a field you can
query, not an adjective. And it can fail — a check that always returns pass is not a
check.

11. Fingerprints in the ledger, not payloads

A ledger you keep for a long time is at risk of becoming a disclosure surface.

The handling starts at tool_calls.py:59. Arguments are redacted before they are
persisted; a key matching one of sixteen sensitive names becomes [REDACTED]:

api_key       apikey          access_token   authorization
client_secret cookie          credential     password
private_key   refresh_token   secret         secret_access_key
session_token token           x_api_key

Keys are normalized before comparison — camel case split, non-alphanumerics folded to
underscores — so apiKey, API-KEY and api_key are treated alike.

One exception is worth knowing: if the value is a dict carrying a secret_id, it is
not redacted. It is already a reference rather than a plaintext, and blanking it would
destroy the one thing you want later: which secret this call used.

Then size. Arguments over 8192 bytes are not stored; three things are kept instead:

return {
    "truncated": True,
    "size_bytes": summary["size_bytes"],
    "request_hash": summary["payload_hash"],
    "argument_names": sorted(str(key) for key in value),
}

request_hash is computed over the pre-redaction original — canonical_request_hash
does a sorted, compact JSON dump and sha256s it.

The effect: the ledger holds no payload, but idempotency still works. The same key
arriving twice is compared on request_hash; a mismatch raises
Tool call identity was reused with different input.

Reconcilable, but not leaky. Second-nicest thing in this ledger, after in_doubt.

12. Seven things that don't line up yet

This section is entirely about our own problems, ordered by impact.

① Workflow replay/retry never writes lineage into the ledger.

runs has source_run_id and attempt_no, plus an index built for them. Across the
whole repo there are 19 create_run( call sites and exactly one passes
source_run_id — the knowledge ingestion path from section 7.

Workflow replay goes through execute_workflow into engine.execute, and the engine
creates the run like this (modules/workflow/runtime/engine.py:145):

run = self.trace_writer.create_run(
    mode=plan.mode,
    subject_kind=plan.subject_kind,
    subject_id=plan.subject_id,
    subject_version_id=plan.subject_version_id,
    input_summary=input_summary,
    run_id=plan.run_id,
)

No source_run_id. No attempt_no.

Impact: source_run_id exists only in the HTTP response body. If the caller does
not store it, the "B is a replay of A" relationship is gone — unqueryable in the ledger,
and ix_runs_scope_source_created indexes nothing useful.

Workaround today: have the caller keep the source_run_id it got back.
Intended fix: thread source_run_id and attempt_no through those two paths into
create_run. I intend to open an issue for this; I had not filed it when this was
written, so there is no link here.

② Re-execution reads a truncated copy of the inputs.

Section 2 noted that input_summary is cut at 8192 bytes. Workflow replay loads inputs
like this (modules/workflow/application/service.py:151):

def _load_run_inputs(self, run: Run) -> dict[str, Any] | None:
    if not run.input_summary:
        return None
    import json
    try:
        parsed = json.loads(run.input_summary)
        ...

It json.loads the summary.

So a run whose inputs exceeded 8KB will fail to parse on replay (truncated JSON generally
is not valid) and return Replay requires inputs or a parseable run input_summary.

Workaround today: pass inputs explicitly instead of letting it read from the ledger —
both endpoints accept an override.
Intended fix: send large inputs to an artifact and keep a pointer, exactly like tool
results in section 2. Same status: issue intended, not yet filed.

③ generate_ulid() does not generate a ULID.

The source says so itself (kernel/commons/ids.py:9):

def generate_ulid() -> str:
    """Generate a ULID-like sortable ID.

    For now, we use UUID4 with prefix. In production, consider using
    python-ulid or similar library for true ULID generation.
    """
    return f"id_{uuid.uuid4().hex}"

UUID4 is random. Not sortable at all — neither the name nor the "sortable" in the
docstring holds.

Bounded but real impact: everything that needs chronological order has to use
created_at rather than the id. The catch-up in section 4 does exactly that. Arguably
forced into the correct implementation.

④ Catch-up uses a strict greater-than.

Following from ③: created_at comes from Python's datetime.now(UTC), and the filter is
RunStep.created_at > last_step_time.

Theoretical consequence: if two steps land on an identical timestamp and the client's
last received event was one of them, the other is skipped by the strict comparison.

I did not reproduce this. datetime.now() resolves to microseconds on modern Linux,
and two steps in one run colliding on the same microsecond takes unusual conditions.
It is listed as a design fragility, not an observed bug — please don't repeat it as
one. The fix is easy once ③ is done: order by (created_at, id).

⑤ Ids in the ledger come in three shapes.

Because generate_ulid() already returns an id_-prefixed string, anything that adds its
own prefix ends up double-prefixed:

All three work. The third just gives no hint which table the row belongs to.
Cosmetic, no functional impact — but you notice it the moment you start reading rows.

⑥ The Run.status docstring lists 6 statuses; there are 11.

On the model (models/runs.py:79):

"""Status: queued, running, paused, succeeded, failed, canceled."""

ExecutionStatus has eleven: those six plus preparing, waiting_input,
waiting_approval, retrying, expired. Steps add skipped on top.

Impact: anyone writing a client from that comment misses five states. Documentation
drift; a one-line fix.

⑦ The soit runs replay line in the console is display copy — that CLI does not exist.

From the run detail adapter (web/app/console/adapters/run-detail.ts:227):

ledger_code: {
  command: `soit runs replay ${run.id} --dry-run`,
  output: `replaying ${detail.steps.length} steps · verdict on record: ${run.status}`,
},

It renders as a code sample explaining what that panel shows.
But there is no soit CLI in the open-source repo — server/pyproject.toml has no
[project.scripts], and server/scripts/ has no matching entry point.

The thing that does work is the HTTP endpoint from section 3. There is a replay script in
the repo, but it is for the outbox (server/scripts/replay_outbox_event.py, 41 lines —
it returns one terminally failed domain event to the pending queue), which is a different
thing entirely.

I went back and forth on including this. Including it says we haven't kept our own
console copy honest. Leaving it out means a reader types the command from a screenshot and
gets nothing. Included, in the end — the gap between demo copy and real capability is
exactly the kind of thing a reader is entitled to know.

Coming clean

• No fresh live run behind this piece. Every conclusion comes from reading soit/ at commit fb46f20, plus the tests already in the repo. I did not stand up an environment, execute a run and then call the replay endpoint. Every claim carries a file and a line number; go check them.
• Item ④ in section 12 is an inference, not an observation. I did not construct the colliding-timestamp case. It is listed because a design should not depend on timestamp uniqueness, not because we have seen it break.
• Replay does not promise identical output. Replay #4 genuinely runs again — models have temperature, tools talk to real systems, external data moves. What is promised is the same inputs, the same governance policy and complete evidence. There is no record-and-stub harness for tools in the repo.
• This is all the community edition. Every path above is in github.com/soit-ai/soit and readable right now.
• Disclosure: I maintain SOIT.

One-line version

"Replayable" here is not an adjective. It is a field the platform computes, that you can
query, and that can come back fail — backed by five database tables, four replay paths
with very different costs, and one status willing to admit we don't know whether the other
side did the thing.

Try it, and come argue

The repo is github.com/soit-ai/soit. To check the claims above:

1. Start it, send any message, take the run_id.
2. GET /api/v1/runs/{run_id} and look at replay_ready among the thirteen evidence items — if it is fail, missing names what is absent.
3. GET /api/v1/observe/runs/{run_id}/replay and see what the seven keys hold.

If any of the seven items in section 12 is wrong, open an issue and say so. I would rather
learn where this doesn't line up than be told the design is nice.