AI agents become fragile when they are connected in long synchronous chains. An event bus lets them work independently, wait for people and tools, recover after restarts, and place policy between a model's recommendation and a real action.
Most AI agent demos fit inside a single request:
User -> Agent -> Tool -> Agent -> Response
The agent makes a plan, calls a tool, gets an answer, and returns a response. It is easy to understand and easy to demo.
Then the same agent meets a real workflow.
It needs data from several systems. One tool takes five minutes. Another agent has to review the result. A production change needs human approval. While the approval is pending, one of the services restarts.
The model is no longer the hardest part. Coordination is.
At that point, adding a better prompt will not fix the system. The agents need a way to work independently, survive failures, and resume after the original request has ended. That is an event-driven architecture problem.
Synchronous chains work until somebody has to wait
Imagine an operations agent investigating a slow checkout service. It needs to collect metrics, inspect dependencies, run a diagnostic job, propose a recovery action, wait for approval, execute the change, and verify recovery.
With direct calls, each component knows what comes next. The first agent waits for the second. The second waits for a tool. The request stays open while a person decides whether to approve the change.
This works when every step is fast and available. In production, that assumption does not last long.
A timeout may leave a tool running after its caller has given up. A retry may execute the same action twice. A restart can erase the current plan. Adding a security review means changing an integration that was already working.
Human approval exposes the problem most clearly. A decision may take minutes or hours. Holding an HTTP request open across that wait is a poor way to preserve a workflow.
Replace direct calls with facts
In an event-driven design, an agent publishes what happened. Other components decide whether that fact matters to them.
ServiceLatencyIncreased
|
v
DiagnosisRequested
|
v
DependencyFailureSuspected
|
v
RecoveryProposed
|
v
HumanApprovalRequired
|
v
RecoveryApproved
|
v
RecoveryCompleted
The service health agent can publish DiagnosisRequested without knowing which diagnostic agent will handle it. The diagnostic agent can publish DependencyFailureSuspected without calling the remediation agent directly.
An audit service, an observability pipeline, and a security agent can all react to the same event. If one consumer is temporarily unavailable, it can process the event after it recovers, subject to the broker's retention settings.
This is the useful part of an event bus. Components participate in the workflow without being wired directly to one another. It is the same producer and consumer separation described in AWS Prescriptive Guidance for event-driven AI, but the pattern is not tied to any cloud or broker.
An event is not a command
Agent systems get dangerous when every message is treated as interchangeable.
An event records a fact:
{
"type": "ServiceLatencyIncreased",
"eventId": "evt-204-01",
"incidentId": "incident-204",
"service": "checkout-api",
"p95LatencyMs": 1840
}
The exact schema is up to the system. When events cross team or platform boundaries, the vendor-neutral CloudEvents specification provides a common envelope for event metadata.
A model proposal describes the action it wants to take. It does not choose the command ID or the idempotency boundary:
{
"action": "ScaleService",
"incidentId": "incident-204",
"resource": "checkout-api",
"targetState": {
"replicas": 12
}
}
An agent decision is a recommendation. It can include evidence and confidence, but it is still a proposal.
These distinctions matter. An LLM saying "scale the service" does not mean the service was scaled. It also does not mean the action was authorized.
A safer path is:
Event -> Agent proposal -> Policy check or human approval
-> AuthorizedCommand -> Deterministic idempotency key
-> Execution -> Outcome event
The agent interprets the situation and proposes an action. A policy layer checks permissions, limits, and approval requirements. When the action is allowed, the policy layer returns a canonical AuthorizedCommand.
That separation remains useful even when the model changes. The model can become more capable without quietly gaining permission to restart production or issue a refund.
Agent memory is not workflow state
Agent platforms talk a lot about memory. Conversational memory and execution state solve different problems.
Memory may contain a conversation summary, user preferences, or retrieved knowledge. Workflow state answers operational questions:
- Which steps finished?
- Which command is waiting for approval?
- Was this event already processed?
- What should resume after a restart?
A conversation transcript is a fragile place to store those answers. It may be summarized, truncated, or interpreted differently after a model update. If the only record that a refund was issued is a sentence in a context window, the system may eventually issue it again.
Store workflow state explicitly. Keep event IDs, command status, approvals, retry counts, and execution results in durable storage. Give the model only the context it needs for its current decision.
The broker will not make the system reliable for you
Moving work onto an event bus changes the failure modes. It does not remove them.
Duplicate events
Design consumers so they can see the same message more than once. Track stable event IDs, and require idempotency keys for tools that change state. Restarting a service twice or issuing the same refund twice is not a retry strategy.
Events arriving out of order
An approval may arrive after a proposal has expired. A recovery result may appear after a newer action has started. Put versions and timestamps on messages, and reject transitions that no longer match the current workflow state.
Agent feedback loops
Agent A publishes an event that wakes Agent B. Agent B responds with an event that wakes Agent A. Both agents can keep generating messages and spending tokens without completing useful work.
Track causation depth. Set limits for time, tokens, and workflow steps. Route repeated failures to human review.
Conflicting decisions
Two agents may propose opposite actions for the same resource. Do not allow both commands to run. Serialize changes for that resource or use a version check before execution.
The consumer itself should be boring. That is a compliment:
import hashlib
import json
def derive_idempotency_key(command):
material = {
"workflowId": command.workflow_id,
"workflowVersion": command.workflow_version,
"action": command.action,
"resource": command.resource,
"targetState": command.target_state,
}
canonical = json.dumps(material, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(canonical.encode()).hexdigest()
def handle(event, store, agent, policy, executor):
if store.already_processed(event.id):
return
workflow = store.load_workflow(event.incident_id)
if not workflow.accepts(event.type, event.version):
store.mark_processed(event.id, outcome="stale")
return
proposal = agent.propose(workflow.context_for(event))
verdict = policy.evaluate(proposal, workflow)
if verdict.requires_human:
store.request_approval(workflow, proposal)
return
if not verdict.allowed:
store.record_denied(workflow, proposal, verdict.reason)
return
authorized = verdict.authorized_command
idempotency_key = derive_idempotency_key(authorized)
result = executor.run(
authorized,
idempotency_key=idempotency_key,
)
store.append_event_once(
workflow,
result.to_event(),
dedupe_key=idempotency_key,
)
store.mark_processed(event.id, outcome=result.status)
The executor claims the idempotency key before it changes the external system. If the key already exists, it returns the stored result with a replayed status instead of running the action again. Otherwise, it applies the action and returns applied.
This does not give an arbitrary external API exactly once behavior. The tool must support idempotency, or the executor needs a way to reconcile an outcome when it cannot tell whether the action completed. The model can handle the ambiguous reasoning, while policy and execution remain deterministic and testable.
Not every agent needs an event bus
A document summarizer does not need a distributed control plane. A synchronous request is usually enough when one caller expects an immediate answer, the task finishes quickly, and a retry cannot cause harm.
Events begin to earn their cost when work continues past a request timeout, several agents act independently, a human interrupts the workflow, or an action has operational or financial consequences.
There is a price. Teams have to manage message schemas, retention, tracing, failed-message handling, and asynchronous debugging. Use an event-driven design when those costs buy reliability, not because the architecture sounds more sophisticated.
Start with the step that blocks
You do not need to rebuild a synchronous agent system all at once.
Find the step that blocks the longest. It is often a human approval or a slow external tool. Give that step a durable state record and an event that can resume the workflow. Then add an idempotency key to every command that changes something outside the agent system.
Take one agent workflow you already run and draw the waits, retries, approvals, and side effects. Where those boundaries are hidden inside a single synchronous chain, that is where the first event belongs.
Better models will improve agent decisions. They will not recover a lost approval, prevent a duplicate command, or resume a half-finished workflow. Once agents work across services and over time, they inherit distributed-system failures.
Failures still happen in an event-driven design. Durable workflow boundaries make it possible to stop safely and resume later.
