AWS Lambda Durable Functions and the Replay Bug in Human-in-the-Loop Agents

Ciao 👋

Here is a sentence that should not be possible: my workflow executed an action a human explicitly did not approve. Not a different user's action, not a race condition, not a bug in my business logic. The human looked at the screen, saw "refund," clicked approve, and the system went and did "escalate" instead.

Nobody tampered with anything. The code looked correct. The culprit was a feature I had just started using and did not fully understand: AWS Lambda durable functions, and the one thing about them that will quietly wreck you if you miss it.

This is the story of building a human-in-the-loop agent workflow, watching it betray the human, and the one-line structural fix that stops it. Everything here runs locally with no AWS account, and I deployed it to real Lambda too. Let's get into it.

What AWS Lambda durable functions actually are

Quick grounding, because this is new. AWS Lambda durable functions, released in late 2025 and now available across several regions, let you write a long-running, multi-step workflow as a single ordinary Lambda handler. The function can pause for up to a year, waiting on a human or an external event, without paying for idle compute, then resume exactly where it left off.

Historically you reached for AWS Step Functions to do this: a separate state machine to orchestrate, task tokens to thread, JSONPath to move data between states. Durable functions collapse all of that into normal top-to-bottom code. You add two primitives to your handler:

• context.step(...) wraps a unit of work. Its result is checkpointed.
• context.wait_for_callback(...) suspends the whole execution until something outside calls back with an answer.

That is genuinely lovely. One file, one handler, code you read top to bottom. But the mechanism that makes it work is also the trap, and the trap has a name: replay.

The workflow: an AI agent that asks permission

The thing I wanted to build is very 2026: an AI agent that proposes an action, a human approves it, then the system executes. Refunds, moderation calls, infrastructure changes, this shape is everywhere right now. Agents are powerful and occasionally unhinged, so a human approval gate in the middle is just good sense.

In durable-function form it is beautifully short:

@durable_execution
def handler(event, context: DurableContext) -> dict:
    data = json.loads(event) if isinstance(event, str) else event
    proposal = context.step(agent_proposes(data["amount"]), name="agent_proposes")

    def notify_human(callback_id, ctx):
        pass  # send callback_id to your approver (email, Slack, a ticket)

    raw = context.wait_for_callback(notify_human, name="approval")
    decision = json.loads(raw) if isinstance(raw, (str, bytes)) else raw

    if decision == "approved":
        result = context.step(execute_action(proposal), name="execute_action")
        return {"status": "executed", "approved_proposal": proposal, "result": result}
    return {"status": "rejected", "approved_proposal": proposal}

The agent proposes. We show the proposal to a human and suspend. Days later the human approves or rejects, we resume and act. Clean. Ships itself.

Now let me show you the version of this that is subtly, dangerously broken, because it is the version most people write first.

Replay: the thing nobody tells you on day one

Here is the mechanism that makes durable functions durable. When your function suspends at a wait and later resumes, Lambda does not magically continue from the middle of your Python. It re-runs your handler from the top, and for every operation you wrapped in a context.step(...), instead of executing it again, it injects the checkpointed result from the first run. That skip-what-is-done replay is how it rebuilds your function's state on a fresh container after a wait that might have lasted a week.

Read that twice, because the consequence is the whole article. Anything inside a step runs once and is remembered. Anything outside a step runs again on every replay.

I did not believe how literal this was, so I instrumented it. I put a counter on a line outside any step, and another on a line inside a step, then ran one suspend-and-resume:

outside-step executions: 2
inside-step executions : 1

The non-step line ran twice. Once on the first pass, once on replay after the human responded. The step-wrapped line ran once. That is not a quirk, it is the entire contract, and it is stated plainly the moment you go looking. The problem is you usually go looking after it has already bitten you.

The betrayal, live

So what happens if the agent's proposal is generated outside a step? A real agent is non-deterministic. Ask it twice, you can get two different answers. That is the entire point of an agent.

Watch what that does to the approval gate. The agent proposes outside a step. The human sees proposal number one and approves it. The function suspends. The human's approval arrives, the function replays from the top, and the agent call, being outside a step, runs again and produces a different proposal. That second proposal is the one that flows into the execute step. The human approved one thing. The system did another.

I built exactly this, unsafe version and safe version side by side, with a mock agent that returns a different action each time it actually runs. Here is the real output:

UNSAFE  (agent call outside a step)
  human approved   : refund
  actually executed: escalate
  >>> EXECUTED WHAT WAS NEVER APPROVED

SAFE    (agent call inside a step)
  human approved   : refund
  actually executed: refund
  MATCH

There it is. In the unsafe workflow the human approved refund and the system executed escalate, because the agent re-ran on replay and changed its mind after the approval. In the safe workflow the proposal was checkpointed in a step, so the value the human saw is the exact value that executed. Same logic, same everything, one structural difference.

This is not a hypothetical warning. It is a passing test in the repo that asserts the divergence. I made the failure reproducible on purpose, because a gotcha you cannot reproduce is just a ghost story.

The fix is one line of structure

The entire fix is: put the non-deterministic thing inside a step.

# unsafe: re-runs on replay, can change after approval
proposal = agent_proposes(amount)

# safe: checkpointed once, identical at approval time and execution time
proposal = context.step(agent_proposes(amount), name="agent")

That is it. Once the proposal is checkpointed, replay injects the stored value instead of calling the agent again. The human and the executor are guaranteed to see the same thing. No re-run, no drift, no betrayal.

The rule that falls out of this is simple and worth tattooing somewhere: anything non-deterministic or with a side effect goes inside a step. Agent calls. API requests. Timestamps. Random values. Anything that could return a different answer or fire twice if run again. Outside a step is only safe for pure, deterministic glue.

The other gotchas that cost me time

While building this I hit a cluster of smaller gotchas that are in no headline, and every one of them cost me real time. Saving you that time is half the reason this article exists.

Do not assume your input is a raw JSON string. The SDK deserializes JSON input before your handler sees it, so an event sent as {"amount": 500} arrives as a dict, and it stays that type across replays. Call json.loads(event) on it out of habit and you get TypeError: the JSON object must be str, bytes or bytearray, not dict. Guard the parse so your handler works whether the caller hands you a string or an already-decoded object: data = json.loads(event) if isinstance(event, str) else event.

Callback results come back as raw bytes. When the human approves, the answer arrives as raw bytes, and even a simple string comes wrapped. Sending b'"approved"' shows up in your handler as the string "approved" with the quotes still attached. You have to json.loads it yourself. Compare it directly to approved and every approval silently reads as a rejection.

You cannot make an existing function durable. Durable execution is a create-time setting only. You cannot flip it on for a function you already have. CloudFormation replaces the resource; the CLI needs --durable-config at creation. Plan for it up front.

Never invoke $LATEST. Because executions can live for a year, an in-flight execution can be broken by a code change that moves under it. Publish a numbered version or an alias and invoke that.

Deploying it to real Lambda

Local proof is great, but I wanted to see it run on actual AWS Lambda, so I deployed it in eu-west-3 through the console. Create function, author from scratch, Python 3.14, and the Durable execution section is right there at creation time. The console even creates the execution role for you, which sidesteps a pile of IAM setup.

Two things tripped me on the way, both worth repeating. First, the deployed handler name must match Lambda's configured handler; my function was called handler while Lambda expected lambda_function.lambda_handler, and the mismatch throws Runtime.HandlerNotFound after quietly retrying a few times. Rename one to match the other. Second, when I first tested it the execution failed, and I could not see why until I stopped guessing and read the actual event history, which showed the handler crashing before any durable operation ran. Read the execution history; it is the flight recorder of durable functions.

Once the handler name lined up and I tested with {"amount": 500}, the execution reached the approval gate and did the most satisfying thing in this whole build: it suspended, and just sat there.

It sat in the Running state, suspended, waiting for a human callback that would never come, burning zero compute the entire time. That single row is the whole pitch for durable functions: a workflow paused mid-flight for a human decision, patient and free, ready to resume the instant someone clicks approve.

Durable functions vs Step Functions: the honest trade

Since durable functions cover ground AWS Step Functions has owned for years, the fair question is which, and when.

The Step Functions version of this exact gate needs a state machine definition in Amazon States Language, three separate Lambda functions, the human wait implemented as a .waitForTaskToken task, and JSONPath plumbing to thread data between states. The logic lives in two places at once: orchestration in the state machine, work in the Lambdas. To understand the whole thing you read both.

The durable version is one file you read top to bottom.

But, and this is the honest part, the durability comes from replay, and replay is a contract you now carry in your own code. Step Functions never replays your task code. Each task runs exactly once, its result stored externally, orchestration handled outside your logic. It makes you write more plumbing, but it will never surprise you with a silent re-run. Durable functions make the code simple and hand you the replay discipline to uphold yourself.

So the trade is not "new thing beats old thing." It is: Step Functions, more ceremony, no surprises. Durable functions, less ceremony, you own replay-safety. Pick the failure mode you would rather be responsible for.

What I am taking away

A few things from this one:

• Durable functions are a genuine joy to write and a genuine footgun if you skip the manual. The replay model is not an edge case, it is the whole model, and it deserves ten minutes of your attention before you build anything real.
• The dangerous bugs here are silent. The unsafe workflow does not crash. It runs, returns success, and does the wrong thing. Those are the worst kind, and the only defense is understanding replay well enough to wrap the right lines.
• When something fails and hides its reasons, stop guessing and read the execution history. It told me exactly what was wrong the moment I actually looked.
• Put every non-deterministic or side-effecting call inside a step. If you remember one sentence from this article, that is the one.

The whole thing is on GitHub: both workflows, the passing tests that assert the divergence, the Step Functions comparison, and the local runner so you can watch it betray the human yourself in about a minute. MIT licensed, no AWS account needed to run the demo.

Repo: https://github.com/mursalfk/agent-approval-gate

Happy Coding! 👋