Originally published on AIdeazz — cross-posted here with canonical link.
A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. August 19, 2026.
A disabled workflow accepted every message and read none of them.
What it looked like from outside
Enquiries arriving through the website chat widget were recorded correctly but received no reply draft for twenty minutes, while the application log stated the handoff had succeeded.
What was actually happening
The chat path handed each lead to an external workflow endpoint and drafted locally only if that handoff failed. The endpoint returned HTTP 200 even though the workflow behind it was switched off, so the failure branch never executed. The lead was neither lost nor answered: it waited for a timeout-based safety net designed as a last resort, not as the primary path.
The fix
The local draft is now produced unconditionally rather than as a fallback, and the external handoff is fire-and-forget. Duplicate suppression absorbs the overlap, so both paths running costs nothing.
How I know it worked
Duplicate suppression confirmed in production -- a second identical submission returned a duplicate result referencing the first draft rather than producing a second approval card.
The rule this earned
Never branch on an acknowledgement. If the fallback logic reads "if the handoff failed, do it myself", it will never run, because the handoff reports success.
The named concepts behind it
Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.
Acknowledgement is not completion
A receipt proves delivery. It never proves processing.
When you hand work to something asynchronous -- a queue, a webhook, a workflow tool, a background job -- the response you get back means "I have received this". It does not mean "I have done this", and very often it does not even mean "I intend to do this".
This is the trap behind a large share of "the data just vanished" incidents. The sending side logs a success, the receiving side never processes anything, and both halves look healthy in isolation. A queue that accepts your message and never reads it looks exactly like one that works.
Defences, in order of strength:
- Do not branch on the acknowledgement. If your fallback logic reads "if the handoff failed, do it myself", it will never run, because the handoff reports success. Make the local path unconditional and let idempotency absorb the duplicate.
- Confirm from the other side. Check that the work actually completed -- a status endpoint, a result record, a callback -- rather than trusting the receipt.
- Set a deadline. If the expected outcome has not appeared within N minutes, treat it as failed and act, rather than waiting forever.
Idempotency
Doing it twice produces the same result as doing it once.
An operation is idempotent if repeating it changes nothing beyond the first time. Setting a value to 5 is idempotent. Adding 5 is not.
This is the property that makes reliability affordable. Networks time out, retries fire, and redundant paths overlap -- so in any real system some operations will happen more than once. If those operations are idempotent, that is a non-event. If they are not, your safety net becomes the thing that corrupts the data or spams the customer.
The usual implementation is a fingerprint: a hash of the inputs that identify the work. Before acting, check whether that fingerprint was already handled inside some window; if so, do nothing, and say so in the log.
The detail that separates a junior implementation from a senior one is what goes into the fingerprint. Too narrow and genuine repeat work gets swallowed; too wide and duplicates slip through. Hashing only "who" would silently discard a real follow-up message from the same person an hour later. Hashing "who plus what they said" collapses the duplicates while letting a genuine second message through.
This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.
No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.
