Every agent I have ever shipped was qualified the same way: someone watched it work once, nodded, and called it production-ready. I did this for years. I trusted my own demos.
So I built the thing that stops me. It's called HivePlane — an open-source control plane where an agent cannot touch a production context until it has passed a reproducible benchmark and holds a signed attestation. Last week I ran its first full field test against real agents: 10 scenarios, 20 acceptance criteria, all pass. The most interesting moments were the refusals.
If you run more than one agent in anything resembling production, this story is for you. And if your agents already benchmark-gate their promotions, tell me where — I looked for that tool and couldn't find it, so I'd genuinely like to hear about yours.
The sentence that started it
I kept writing the same note while building this thing:
An agent is production-ready because someone watched a demo.
That sentence is true at every company I've worked for, and it's true for a worse reason than laziness: there is nothing to check. Agent frameworks solve orchestration inside one workflow. Nothing solves the fleet-level questions — who owns this agent, what may it spend, which tools may it call, and the one nobody answers: has it proven itself?
SWE-bench gave coding agents a reproducible benchmark and clear pass/fail, and coding agents got dramatically better. Production agent fleets have no equivalent. Teams swap prompts, change models, and ship to production with zero benchmark evidence — then act surprised when a regression reaches a customer.
The loop, and the ladder inside it
The control plane enforces one loop:
register → certify → gate → run → intervene → deliver → observe
The part I care about is the ladder. Every workload carries a certification status, and admission is enforced against it:
| Status | Meaning | Where it can run |
|---|---|---|
uncertified |
Registered, never benchmarked | Sandbox only |
provisional |
Passed the staging threshold (0.80) | Staging |
certified |
Passed the production threshold (0.90) | Production |
quarantined |
Failed re-certification or drifted | Runs blocked |
The ladder is the product. Budgets, policies, and dashboards exist to make the ladder real.
tip: Production certification is not the staging threshold with a bigger number. It's a separate benchmark run, and a manifest change — prompt, model, tools — invalidates the old certification. There is no silent path back into production.
What the field test proved
The receipts are committed in the field test report: 10 scenarios against the live Docker stack, 10/10 pass, all 20 acceptance criteria holding. The subjects were real agents — a raw-Python support agent and a LangGraph judge graph — plus four deliberately bad fixtures I'll cover in article 3.
Certification is not a simulation, which is the part I'd underline twice. The benchmark executes every corpus task as a real run — through the runtime adapter, the policy boundary, and (where used) the governed model seam:
| Workload | Context | Status | Pass rate | p95 latency |
|---|---|---|---|---|
| support-agent | staging | provisional |
1.00 (6/6) | 92 ms |
| support-agent | production | certified |
1.00 (6/6) | 67 ms |
| eval-judge | staging | provisional |
1.00 (4/4) | 215 ms |
| eval-judge | production | certified |
1.00 (4/4) | 126 ms |
Every certification produces a signed Ed25519 attestation bound to the exact model identity, verified on every read. The same sweep refused an uncertified agent with a 403, blocked a model swap with a 403, quarantined an agent that looked fine and wasn't, and killed a run the moment it went over budget.
The container layer separately went 25/25 — image build, API contract, the control loop, restart durability, the UI — because a control plane that only works on my laptop is not a control plane.
Why "Kubernetes for agents" is the honest analogy
Kubernetes is not a container runtime; it operates many of them behind one contract. This is not an agent framework; it operates many agents behind one contract — the workload manifest: owner and team, runtime adapter, allowed tools, model identity, budgets, sandbox and egress rules, certification corpus and thresholds, fan-out destinations.
Change anything in that manifest and the agent re-certifies before it can touch production again. That last sentence is the whole thesis, and the rest of this series is what it cost to make it true.
What I learned building it
The refusals are the product. Any platform can start runs. The gate that says 403: certification status 'uncertified' is insufficient for production; requires 'certified' — named, attributed, actionable — is what makes the platform trustworthy. A refusal an operator can act on is a workflow; "Forbidden" is a dead end.
Certification is a security control, not a quality metric. The moment production admission depends on a signed attestation, a whole attack class becomes blocked and auditable: swap the model, edit the manifest, quietly regress the agent. Article 3 is the four ways my own gate said no.
The operator surface has to be fast or it won't be used. Inspect-plus-stop measured at 0.04 seconds; scaffolding a new project at 0.24 seconds. Slow governance tools get bypassed, and a bypassed control plane is a expensive dashboard.
What it doesn't do yet
- v0.1.0 certifies two adapters: raw Python workers and LangGraph. The adapter contract is the seam; broader framework coverage is deliberately deferred.
- The drift detector ships next. Today, re-certification is scheduled and change-triggered — it catches what changes, not what fades.
- One model identity was validated this cycle; the cloud-profile run with real prices is the next field test.
References
- Field test report (v0.1.0) — source for every number above
- Docker test report — the 25/25 container layer
- Security audit · Release notes · Changelog
Next in this series: why my three "real" agents failed the field test on day one, and what I certified instead.
What's the last agent you shipped on demo evidence alone — and what did it cost you when it mattered?
