The cost of undoing: partial failures and the cleanup bill

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

An agent I was running had a simple job: register a customer in three systems, then send a welcome email. It succeeded at steps 1–3 (customer record in the CRM, account created in the billing system, subscription set in the feature service), then failed at step 4 (email send gateway returned a 429).

The agent's error-recovery logic kicked in: retry the email, wait, retry again. After the retry budget was exhausted, the agent reported failure and stopped.

What it didn't report: three systems now had a customer record that didn't exist in any of the others. The email never sent, so the customer never knew they had an account. The CRM and billing system had data that was out of sync with the feature service. A week later, when a customer support ticket came in ("I thought I signed up but nothing works"), someone had to manually find the orphaned records across three systems, decide whether to roll them back or forward-fix them, and clean up the inconsistency.

The cost wasn't the failed email send. It was finding and fixing the orphaned state.

This is the cleanup bill: the hidden cost of partial failures that standard retry logic doesn't touch.

Why partial failures are expensive to undo

A partial failure — where some steps in a multi-step operation succeed and others fail — creates an asymmetry in cost:

1. The original operation was cheap. One agent, one pass, straightforward execution. Cost: 5 API calls, ~300 tokens, ~$0.01 in LLM spend.
2. The cleanup is not. Now you have to:
   • Detect that cleanup is needed (a human notices, or an alert fires, or a test fails)
   • Find which records were written (dig through logs, query across three systems)
   • Decide the recovery strategy (rollback vs. forward-fix)
   • Implement the fix (write a script, run it carefully, verify it worked)
   • Communicate the fix (notify the customer, update internal docs)

For a single incident, this costs $200–500 in human time. For a fleet of 10,000 agents, if the failure rate is even 0.1%, that's 10 orphaned-state incidents per day. The cleanup bill scales faster than the operations do.

The patterns that create cleanup debt

Pattern 1: No rollback plan baked into the agent

The naive approach:

async def onboard_customer(agent, customer_id):
    # Step 1: Write to CRM
    crm.create_customer(customer_id, data)

    # Step 2: Create billing account
    billing.create_account(customer_id, plan="starter")

    # Step 3: Set feature flags
    features.enable(customer_id, ["core", "api"])

    # Step 4: Send email
    email.send(customer_id, "welcome.html")

    return "success"

If step 4 fails, steps 1–3 are committed. There's no automatic rollback because there's no rollback mechanism defined. The agent doesn't even know that a partial success is a problem — it just knows that step 4 failed and retried.

The fix is to define a rollback strategy and bake it into the flow:

async def onboard_customer(agent, customer_id):
    try:
        # Step 1: Write to CRM
        crm_id = crm.create_customer(customer_id, data)

        # Step 2: Create billing account
        billing_id = billing.create_account(customer_id, plan="starter")

        # Step 3: Set feature flags
        features.enable(customer_id, ["core", "api"])

        # Step 4: Send email
        email.send(customer_id, "welcome.html")

        return {"status": "success", "crm_id": crm_id, "billing_id": billing_id}

    except EmailSendError as e:
        # Rollback: remove from feature service
        try:
            features.disable(customer_id)
        except:
            pass  # Best-effort rollback

        # Rollback: mark account as pending in billing
        try:
            billing.mark_pending(customer_id)
        except:
            pass  # Best-effort rollback

        # CRM stays (it's the source of truth; manual fix if needed)
        # Re-raise so caller knows to retry or escalate
        raise

The cost of defining the rollback and executing it on error is ~50 tokens and one extra API call per failure. The cost of not defining it is $200+ per incident, times the number of incidents. If your fleet has a 1% failure rate, you're paying the cost of cleanup hundreds of times more than you'd pay for preventive rollback definition.

Pattern 2: Rollback is defined but not tested

This is the silent killer. You have a rollback path, you've written it, and it's never been exercised until the day you actually need it.

def rollback_customer(customer_id):
    # Remove from feature service
    features.disable(customer_id)

    # Revert billing status
    billing.revert_to_pending(customer_id)

    # Delete from CRM
    crm.delete_customer(customer_id)  # <-- This call might not exist

When the real failure happens and you call the rollback:

• features.disable() works fine.
• billing.revert_to_pending() works, but turns out this endpoint has a bug where it creates a new pending record instead of reverting the old one. Now you have two billing records.
• crm.delete_customer() fails because the endpoint doesn't support deletion — it only allows marking as inactive.

The rollback itself is partially failed, and now you have a bigger mess than the original problem. The cost balloons: not just cleaning up the original orphaned state, but untangling the broken rollback too.

The fix: test the rollback paths under failure conditions before deploying.

def test_rollback_after_failed_email():
    # Setup
    customer_id = "test_customer_123"
    crm.create_customer(customer_id, {})
    billing.create_account(customer_id)
    features.enable(customer_id, ["core"])

    # Simulate email failure
    email.fail_next_send()

    # Call the real onboarding
    with pytest.raises(EmailSendError):
        onboard_customer(agent, customer_id)

    # Verify rollback happened
    assert not features.is_enabled(customer_id)
    assert billing.get_account(customer_id)["status"] == "pending"
    assert crm.get_customer(customer_id)["marked_inactive"] == True

Running this test before production means you catch the "delete doesn't exist" problem in staging, not after 100 customers are partially onboarded.

Pattern 3: Observability doesn't surface partial failures

If you only monitor "how many onboarding requests succeeded/failed," you miss the partial-success case. The request failed, so it's counted as a failure. But three systems have data for that customer anyway.

# This is insufficient:
@app.post("/onboard")
def onboard(customer_id):
    try:
        result = onboard_customer(customer_id)
        metrics.increment("onboard_success")
        return result
    except Exception as e:
        metrics.increment("onboard_failure")
        raise

You need a health check that detects orphaned records:

def detect_orphaned_customers():
    """Find customers in some systems but not others."""
    crm_customers = set(crm.list_customers())
    billing_customers = set(billing.list_accounts())
    features_customers = set(features.list_enabled_users())

    # Find inconsistencies
    in_crm_not_billing = crm_customers - billing_customers
    in_billing_not_features = billing_customers - features_customers

    if in_crm_not_billing or in_billing_not_features:
        metrics.gauge("orphaned_customer_records", 
                     len(in_crm_not_billing) + len(in_billing_not_features))
        alerts.fire("orphaned_state_detected", 
                   crm_not_billing=in_crm_not_billing,
                   billing_not_features=in_billing_not_features)

A daily or hourly run of this check gives you a leading indicator of partial failures. The orphaned records are found quickly, not days later when a customer complaint arrives.

Why partial failures cost more than full failures

Here's the cost comparison:

Full failure (all steps fail, none commit):
├── Detection: Automatic (agent error on first step)
├── Recovery: Retry the operation (no cleanup needed, state is clean)
├── Cost: One LLM pass + retry logic
└── Total: ~$0.05

Partial failure (steps 1-3 succeed, step 4 fails):
├── Detection: Manual (customer complains, or health check fires)
├── Investigation: Find which records exist (query logs, query systems)
├── Decision: Rollback or forward-fix? (human judgment call)
├── Execution: Write and test fix script (developer time)
├── Verification: Confirm fix worked across systems (manual testing)
├── Communication: Notify customer (email or support ticket)
└── Total: $200–500

The multiplier is real. I've seen single partial-failure incidents cost more to clean up than the LLM spend on 10,000 successful operations.

The cost calculus for different recovery strategies

When a partial failure happens, you have three choices:

Strategy 1: Rollback everything

Undo all the writes that succeeded, return to the clean state before the operation started.

Cost:

• Execution: One API call per system that was written (low cost, ~5–10 calls)
• Verification: Check that rollback succeeded across all systems (medium cost, ~1 human minute to verify logs)
• Retry: Re-run the operation from scratch (medium cost, same as original)
• Risk: Rollback itself might fail (low risk if tested, high risk if not)

Benefit: System is in a known-clean state; no orphaned data.

When to use: Operations where the output isn't consumed immediately (e.g., onboarding, data import). Safe to retry.

Strategy 2: Forward-fix (complete the partial operation)

Decide that the three succeeded writes are good; just finish the missing step.

Cost:

• Execution: One API call to complete the missing step (low cost)
• Verification: Check that the systems are now consistent (low cost)
• Risk: If the missing step fails again for the same reason, you're back to partial failure

Benefit: No rollback risk; faster than rollback-and-retry.

When to use: The partial state is valid and useful (e.g., you wrote a customer record; email can retry later). The missing step is independent of the others.

Strategy 3: Quarantine and manual fix

Mark the records as "needs manual review," don't rollback or complete, escalate to humans.

Cost:

• Execution: Add a flag to the record, send an alert (low cost)
• Human review: Someone investigates and decides rollback or forward-fix (high cost, ~30 min per incident)
• Risk: Human makes a mistake, or the fix gets delayed

Benefit: Gives you time to understand what went wrong before committing to a fix.

When to use: When you're unsure whether rollback or forward-fix is correct, or when the partial state is dangerous.

For the customer onboarding example:

async def onboard_customer(agent, customer_id):
    try:
        crm_id = crm.create_customer(...)
        billing_id = billing.create_account(...)
        features.enable(...)
        email.send(...)
        return "success"

    except EmailSendError:
        # Email is not critical; forward-fix is safe
        # Mark the account as active anyway (customer can resend email)
        return {"status": "email_pending", "customer_id": customer_id}

    except BillingCreateError:
        # Billing failure means we can't charge; rollback entirely
        try:
            features.disable(customer_id)
            crm.rollback_to_pending(customer_id)
        except:
            alerts.fire("rollback_failed", customer_id=customer_id)
            raise
        raise  # Let caller retry

    except CrmError:
        # CRM is source of truth; if it fails, nothing is written yet
        # Just fail and retry
        raise

Different failures call for different recovery strategies. The cost of choosing wrong is paid in cleanup.

What I'd do

1. Define rollback before failure. For each multi-step operation, decide the recovery strategy before deploying. What's the rollback order? Which steps are critical vs. safe to leave orphaned? Which steps can be retried safely? Bake this into the agent logic.

2. Test rollback paths under realistic failure conditions. Don't test "happy path with manual rollback after the fact." Test "operation fails at step 4, automatic rollback runs, system is clean." Use chaos engineering or fault-injection tests to exercise rollback.

3. Surface partial failures in observability. Add a periodic health check that detects customers/records/state inconsistent across systems. Alert on orphaned records immediately, don't wait for customer complaints.

4. Classify failures by rollback cost. On each exception, decide: can we rollback safely, or should we forward-fix, or should we quarantine and escalate? Different decisions for different error types.

5. Measure cleanup cost separately. Track "incidents that required manual cleanup" separately from "operations that failed." The cleanup cost per incident is often your largest LLM-operation cost lever — it's where single-incident fixes save the most money.

6. Prefer idempotent operations at the boundary. If your four-step operation is idempotent (safe to run multiple times and get the same result), then partial failure is less dangerous — you can just retry the whole thing. Idempotency is expensive, but rollback is more expensive.

In a fleet, partial failures don't scale linearly — they scale with the product of fleet size and failure rate. A 0.1% failure rate sounds safe until you realize it means 10 incidents per day on a fleet of 10,000. Each one costing $200 to clean up. That's $2,000/day in cleanup overhead that doesn't appear in your LLM token spend. Preventing it with rollback strategy, testing, and observability is usually the highest-ROI reliability investment.

The numbers here are from composite incidents I've traced — single customer onboarding scenarios in B2B SaaS. Your cleanup costs may be lower (internal APIs are faster to fix) or higher (if data corrupts customer-facing reports). The principle holds across domains: partial failures that go undetected are the cleanup bill on your unexpected invoice.

ref: phase-2-content-2026-08-15