The symptom
I run an autonomous pipeline that queues tasks, executes them, and writes a status back: pending, running, done, or failed. One morning I was scanning the day's task log and saw this:
{
"task_id": 28,
"status": "done",
"notes": "JSON parse failed",
"output": { "title": "...", "single_file_html": null }
}
Read that again. The status says done. The notes say the parse failed. The field the whole downstream report depends on — single_file_html — is null. This isn't a task that failed loudly. It's a task that failed quietly, while reporting success, and every consumer downstream of it (a daily digest, a deploy step, a human skimming a dashboard) had every reason to trust the green checkmark and move on.
If you've worked on any pipeline — CI, ETL, an agent framework, a batch job — you already know how this story ends. But the specific way it happened is worth walking through, because the fix is boring and the mistake is not.
Wrong theory #1: "It's a downstream parsing bug"
My first instinct was that the consumer of this task's output was broken — that something reading single_file_html later in the pipeline was choking on valid-but-unusual JSON and null-ing it out on the way through. I went looking in the wrong file for twenty minutes before I noticed the notes field was written by the producer, not the consumer. The task itself was telling me, in plain text, that it already knew something was wrong. I'd just built a system where that knowledge had nowhere to go.
Wrong theory #2: "It's LLM flakiness, retries will fix it"
The task in question calls an LLM to generate a JSON payload, including an HTML string field. LLMs occasionally truncate long string fields or emit malformed JSON under token pressure — that part isn't surprising. My second theory was that this was just noise: a probabilistic failure mode that a retry-with-backoff would paper over. That's a reasonable mitigation, and I added it. But it's a mitigation for the symptom (LLM sometimes returns bad JSON), not the defect (bad JSON silently became a successful task). Retries reduce how often you hit the bug. They don't fix the bug.
The actual root cause
Here's the code shape that caused it, simplified:
def run_task(task):
try:
raw = call_llm(task.prompt)
payload = json.loads(raw)
except json.JSONDecodeError as e:
log.warning(f"JSON parse failed: {e}")
payload = {"title": task.title, "single_file_html": None}
write_output(task.id, payload)
task.status = "done" # <-- this line doesn't know or care what happened above
return task
The bug is that one line. task.status = "done" runs unconditionally after write_output, because in the mind of whoever wrote it (me, three weeks earlier), "done" meant the function reached the end without raising an unhandled exception. It never meant the output is usable. Those are two different claims, and the code only ever tracked the first one. The except block was written defensively — don't crash the whole pipeline over one bad LLM response — but defensiveness at the exception layer quietly became a data-integrity hole at the status layer.
This is the same failure mode as a CI job that runs pytest || true to keep the build from going red when a flaky suite fails, or a build step that logs "0 files matched, skipping" and still exits 0. The mechanism for surviving a failure and the mechanism for reporting success got fused into the same code path, and nobody separated them back out.
The fix
The fix has to make "done" a claim about the data, not the control flow:
def run_task(task):
raw = call_llm(task.prompt)
try:
payload = json.loads(raw)
except json.JSONDecodeError as e:
task.status = "failed"
task.notes = f"JSON parse failed: {e}"
return task
if not is_complete(payload, required=["title", "single_file_html"]):
task.status = "failed"
task.notes = "schema incomplete: missing required field(s)"
return task
write_output(task.id, payload)
task.status = "done"
return task
Two changes matter more than the line count suggests. First, validation happens before write_output, not after — so a bad payload never even reaches storage under a name that implies it's trustworthy. Second, and more importantly, a parse or schema failure now sets status = "failed" instead of quietly falling through to done with an apologetic note. The pipeline loses nothing: it still doesn't crash, it still logs exactly what went wrong, and now every downstream consumer that filters on status == "done" gets what that filter has always implicitly promised — a payload it can actually use.
The lesson
"Done" is not a neutral word. It's a contract, and every piece of code that sets it is making a promise to every piece of code that reads it. The bug here wasn't that the LLM returned bad JSON — that was always going to happen sometimes, and no amount of prompt engineering will get that to zero. The bug was that I'd let "the code finished executing" and "the output is valid" collapse into a single status value, so the moment the first one became true, the second one got assumed for free.
The audit worth running on your own pipeline isn't "do we have error handling" — you probably do. It's narrower: for every place that sets a status to something meaning success, trace backward and ask what claim that status is actually making, versus what claim your downstream code thinks it's making. If those two things differ even slightly, you have a task 28 waiting to happen — a green checkmark sitting on top of a null, quietly making everyone's life easier right up until it doesn't.
