Do you trust AI code? Of course you don't. Not fully.
And here is the part that's easy to miss: it usually isn't wrong. It's incomplete. The code compiles, the happy path works, the demo is clean. What's missing is the boring 20%: the dependency nobody verified, the step that runs before its prerequisite, the rollback that exists on paper and doesn't actually undo anything.
I learned this building agent systems, not by reading about them. I ran 170 agent goals and watched a planner repeat the same three planning defects predictably. I ran 83 real agents through a tool-call gate. I spent a week fixing a matcher when the bug was six lines in the evaluator.
If add/commit/push and "review the diff" are second nature, this list is for you. If you're new, bookmark it. These are the checks that don't show up in a green CI run.
Each item is one thing I now make a gate instead of a guideline. A guideline is a hope. A gate is a test that fails the build.
1. Verify every dependency before the step that needs it
The most common defect I measured: a plan declares a precondition that no earlier task establishes. The planner knows what should be true. It doesn't arrange the steps to make it true.
[BLOCKER] unverified_dependencies, task=cutover_traffic_100
"Cutover to 100% is dependent on prior stages being established
but lacks confirmation of stability before proceeding."
Across 63 failing plans I collected 57 blockers in this one family. The fix is a deterministic precondition closer that runs after the model produces a draft and refuses to pass until every precondition maps to an earlier task.
If AI says "after verification," ask which task does the verifying. If the answer is "it goes without saying," that's the bug.
2. Order tasks after their hard prerequisites
46 blockers were pure sequencing errors. The cutover ran before the check. The backfill ran before the index was validated. The database was dropped before the backup finished.
Models are good at listing the right steps and bad at ordering them. That's a graph problem, not a language problem. Enforce it with a topological check, not a prompt.
[BLOCKER] unsafe_sequencing, task=backfill_vectors
"Cannot proceed until the index is verified for quality;
it is ordered incorrectly in the sequence."
Full breakdown: The Planner Made the Same 3 Mistakes Every Time.
3. Require a credible rollback on high-blast-radius steps
Not just a rollback. A credible one. The planner put rollback on routine steps and skipped it on cutover, teardown, and failback, for 18 blockers.
[BLOCKER] weak_rollback, task=dual_write_setup
"Rollback only switches to single-write mode without addressing
the inconsistencies dual-write may have introduced."
A rollback that doesn't restore the prior state is decoration. Gate it by asking one question: if this step fails, does the declared rollback return the system to the known-good state?
Rollback coverage is not the metric. Rollback reachability is.
4. Gate tool calls, not prompts
Every agent safety talk ends at the prompt. Prompt injection defense via prompt is a suggestion. The agent's tool call is the last place you can be deterministic, so make it a gate.
I built Agent ToolTrust around four decisions, not two: allow, audit, escalate, deny. Binary allow/deny forces a choice between over-privileged agents and approval fatigue. Four states give you a middle ground. And because the engine sits outside the model, no amount of prompt engineering can override a deny.
@adapter.guard(tool_name="deploy_service", action="deploy",
environment="production", data_class="restricted")
def deploy_service(service: str) -> str: ...
# -> escalate: write action in prod on restricted data requires approval
Tested against 83 real agents across 10 frameworks, zero mocks. Write-up: I Stopped Trusting AI Agents With Tools.
5. Fail closed on unknown input
The cheapest safety property to get wrong. Unknown tool, deny. Malformed input, deny. Engine crash, deny. The alternative is fail-open: an attacker who can crash the gate gets unrestricted access.
def decide(request):
try:
return policy.evaluate(request)
except Exception:
return DENY # fail closed, never fail open
I wrote this as design decision DD-14 before the first line of code, because a near-miss teaches it more expensively than a spec does.
If your gate's failure mode is "allow," it isn't a gate. It's a suggestion with logging.
6. Count refusals and escalations as output, not failure
A system that approves everything isn't safe. A system that refuses 96 of 97 strict goals and hands them to a human is doing exactly what it should.
The number that matters isn't approval rate. It's whether the escalate path actually reaches a human with enough context to decide.
if goal.posture == STRICT and goal.critical_blockers > 0:
result = ESCALATE # a correct output, not an error
I wired that as a first-class output: My Agent Refused 96 Times. That Was the Right Output.
7. Canary your safety gate
The failure mode I didn't see coming: a deterministic gate quietly stops firing. No crash, no error. The blocker count drops from 226 to 40 and it reads like plans got safer.
A reader pointed out the shape: a Kubernetes incident where four releases never ran because an old ReplicaSet kept one pod green, and every health check passed.
The fix is a canary. A known-bad plan per gate, asserting the gate still fires, run before every sweep.
plancritic gates canary --check # 10 fixture pairs, all must fire
# 10/10 in dev and Docker
Full story: The Gate That Stayed Silent.
What component of your system, if it stopped working, would make your metrics look better?
8. Don't let the model grade itself
My local 3B model produced a trigger that was literally "step_1". It matched every trajectory, scored precision 1.00 / recall 0.02, and my benchmark said pass. The model wasn't misbehaving. It was optimizing the reward I defined.
_DEGENERATE_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)
# reject "step_1" before it ever reaches the matcher
If your matcher rewards surface similarity, your model will produce surface similarity. That's a benchmark design bug, not a model bug. My 3B Model Found a Shortcut.
9. Field test with real agents, not mocks
Mock agents always call the tool. Real 4B models sometimes answer textually instead. On an earlier project I added field testing late and discovered the pass rate was 9%. Not because the tool was bad, but because mocks had hidden every integration problem.
pytest tests/ -q # unit tests: green, and they prove nothing about integration
python field_test.py # real agents: the run that actually counts
This time the rule was in the spec: no ship until real agents prove the policy works. The integration cost is visible from the first run. The cost of mocks is invisible until production.
10. Re-run the eval after every fix
I spent a week fixing a matcher. Four fixes, 359 green tests, golden pass rate moved 10% to 20%. Then a six-line classification fix moved it 20% to 50%. The report had named the matcher as the top target, and it was right, for the previous release. I was optimizing a stale diagnosis.
fix -> rerun field test -> re-diagnose # do not trust the old report
A decisive verdict is not the same as a correct verdict. Re-run the eval before you trust the direction.
Full post-mortem: The 6-Line Fix That Outperformed My Entire Matcher Week.
The point
None of these are exotic. They're the checks a careful human does without thinking: verify the dependency, order the steps, know how to get back, refuse when unsure. The problem is that AI generates volume, and unmanaged volume amplifies every skipped check at the exact moment review attention is thinnest.
The move that worked for me wasn't a better model. It was turning each check into a deterministic gate that fails the build.
So what is your #11? What check did AI skip on you that you now enforce automatically?
Repo: planner-critic-engine (deterministic gates over LLM plans) · agent-tooltrust (four-state permission engine) · CauterRule (agent failures into standing rules)
