In the previous article, I argued that AI isn't simply changing the tools we use to develop software, but shifting our work to a new level of abstraction. In this one, I want to address the problem that immediately follows: if we're going to write less and less code directly and agents are going to produce an increasingly larger part of it, how the hell do we know whether what they code is actually right?
Because the answer obviously can't be “trust the AI, it's very smart”.
Even though I personally develop code with AI today with practically no review, I don't blindly trust AI. Just as I don't blindly trust an engineer on my team. I don't even blindly trust myself.
Blind trust is a security hole. And not blindly trusting someone doesn't mean distrusting them, it means having mechanisms to prevent their mistakes, or mine, from causing problems.
That's why we've spent decades building mechanisms and methodologies around software development to detect, and avoid as much as possible, our mistakes. XP. Scrum. Tests. Code reviews. Pair Programming. CI. Static analysis. Permissions. Observability. Environments. Containers. Auditing...
The question, therefore, shouldn't be whether we can trust an AI. The question should be what system do we need to build so we can use it without needing to blindly trust it?
It's not deterministic
One of the first objections is usually that if you ask it the same thing twice, it generates two different pieces of code.
True.
But if you give the same task to two different programmers, or to the same programmer with enough time in between, we'll very probably get two different implementations too, depending on the complexity of what we're asking.
And if we've never required two developers to produce exactly the same code, why do we expect AI to produce exactly the same code from the same request?
Isn't it enough for the result to satisfy the requested requirements?
That it does what it's supposed to do. That it passes all kinds of tests. That it satisfies the acceptance criteria. That it meets security requirements. That it respects certain constraints.
The problem appears when we keep evaluating the writing process while this new level of abstraction increasingly forces us to evaluate the result against a contract.
And when I say contract, I'm not talking about a huge spec telling the agent how it has to build every single thing. I'm talking about the conditions the result must satisfy and having mechanisms that can verify them.
We haven't reached compiler-level reliability yet, and I don't know whether we ever will, or even whether we need to. What is happening is that agents are becoming more capable and we're giving them more and more autonomy.
And the more autonomous they become, the less sense it makes to think we'll be able to control them by constantly looking over their shoulder.
Rules can't live inside the agent
This is, for me, one of the important parts.
A rule written only in a prompt isn't a rule. It's a request.
You can have an AGENTS.md, a skill or whatever you want telling it never to do a certain thing. It helps enormously. I use them, obviously. But we've already seen that the agent forgets them more often than we'd like, sometimes interprets them incorrectly, can even lose that information among the ever-growing amount of context it's handling, or simply finds another way to achieve what it was trying to do.
And that last one isn't theoretical.
While experimenting with all this, I configured a mini PC with Ubuntu to run a model's CLI. I removed the rm command from the system so it wouldn't be able to delete anything.
Then one day I discovered that it had deleted files. How? I couldn't believe it. I asked, and it explained that since rm didn't work, it had used rsync --delete.
I was surprised and scared at the same time.
I had prevented it from using rm. And even though I had explicitly told it never to delete a file without my approval, at one point it realized it had made a mistake and needed to rebuild a structure, so it decided it needed to delete things. It interpreted my prohibition as meaning it shouldn't delete anything critical and that, given the context it had, it could decide that deleting those files wasn't going to cause any problem. After all, it was code that hadn't existed before and that it had created itself, right?
It disobeyed an instruction in order to achieve its goal.
That's why important restrictions have to live outside the agent, in mechanisms it doesn't control. Hooks. CI. Permissions. Tests. Mutation tests. Policies. Gates...
If I want every change to have tests, I can tell it to do TDD. But I also have to verify that those tests exist, that they run and that they actually pass.
If I don't want any change to get in without review, I can't let the same agent decide that “this time” the review isn't necessary.
If I want a security vulnerability to block a change, no second agent should be able to casually decide “well, it doesn't look that serious”.
If I want all work to be traceable, it can't depend on an agent remembering to create a ticket afterwards.
Important rules have to become mechanisms.
Not good intentions.
The agent can decide how to solve the problem. What it shouldn't be able to decide is whether it can bypass the mechanisms that determine whether that solution is acceptable.
Orchestrating isn't governing
And here I think we're usually mixing two different problems.
One is how we orchestrate agents. Who writes. Who researches. Who reviews. How we share context. How several agents work in parallel. Which model we choose for each task.
We can have one writing, another from a different family reviewing, and a third arbitrating when they disagree. We can even launch the same task to several models and then choose between the solutions that satisfy our conditions.
This already looks much more like building an artificial team than using a programming assistant.
But there's another different problem: who governs those agents?
Who establishes the rules? Who decides which actions are allowed? Who checks that they were actually followed? Who prevents an agent from granting itself an exception?
Orchestration gives capability.
Governance gives guarantees.
And we need both.
What about specs?
For a while it seemed that the answer to AI development was the prompt engineer.
It lasted months.
Then the idea emerged that development with agents was going to be about writing better specs. Right?
If the AI made mistakes, it was because we hadn't specified clearly enough what we wanted. So we started writing increasingly detailed documents explaining which endpoints to create, which classes to use, which structure to follow, which cases to handle...
I think that's also starting to become “the past”.
If today I tell a good frontier model “build me a REST API to access my recipe database”, and it has access to the repository, the database and the project context, I don't need to explain that I probably need to list recipes, retrieve one, create one, modify one or delete one, or how to do any of that.
It will explore. It will understand the schema. It will look at how the rest of the project is built, its stack, its style, the conventions we use, even previous conversations we've had, and it can make many of those decisions by itself surprisingly well.
If I've also told it that we work with TDD, that APIs must be documented and that it should generate a Postman collection, I don't have to repeat that in every request either.
But, and here we go back to the beginning of the article, telling it something is one thing and guaranteeing that it does it is another.
Instructions guide.
The environment governs.
That changes where we put our effort again.
I don't need to translate my need into an almost technical specification first so that the AI can then translate that specification into code.
I can express the intention at the highest possible level, my need, my desire, my idea, nothing more: “I want to access my recipes through a REST API”.
It's practically the same thing I do as a manager when I ask a senior engineer. I don't have to specify the details, because they know how we work, they know the conventions, the rules and the context.
The detailed spec was, in a way, another layer of human translation between the idea and the code. And AI is starting to eat that layer too.
What doesn't disappear, and actually becomes more important, are the rules, the constraints and the definition of what really matters.
Are we moving to RDD? Rules-Driven Development? Are they going to start selling us courses on how to govern agents?
Of course 😝
Jokes aside, one thing is telling the agent exactly how to build the solution, which is something I don't do with a senior engineer either, and another very different thing is giving it enough context to make good decisions. A senior engineer can work from a high-level request because they know the domain, understand the product, know how the team works and have references they can use to compare their decisions. And if they don't know something, they know where to look or who to ask.
The same should happen with an agent. If it has access to the domain, the code, the documentation, previous decisions, examples and the team's rules, I increasingly need less to turn my need into a detailed implementation spec.
What is still necessary is defining the conditions that any solution must satisfy. Those can live in team agreements, a playbook, a developer kit, ADRs, architecture rules, security criteria or verification mechanisms.
Fewer specs doesn't mean less context. It means stopping using the spec as a manual translation layer between the need and the implementation.
That's where we're getting to, if we're not already there.
Less implementation specification. More intention. More available context. More mechanisms around it.
Governance also has a cost
But here's another problem I don't think we've solved yet.
Guardrails work, but many times they work after the agent has already smashed into them.
It already happens to me.
One of my rules may say that I don't want PRs larger than 200 lines.
The agent knows it.
And even so, far too often, it ends up trying to create a PR with more than 200 lines.
The hook blocks it.
Perfect.
Governance worked.
But now the agent has to go back. Redo part of the work. Split it. Maybe create another file. Reorganize imports. Divide the PR. Run things again.
And all of that consumes tokens.
A lot of them.
The final result may comply perfectly with the rules, but we got there in an obviously inefficient way.
The system is safe, but the agent has been banging its head against the walls until it found the right path.
And this happens because the rules that live in the model's context don't have the same weight as the real restrictions that live outside it. The model may “know” that a rule exists and still forget it during execution.
I use tools to try to reduce token consumption, such as RTK or Squeezr, but you're only optimizing one part of the problem.
The underlying problem is still there.
Building better guardrails isn't enough. We need agents to get better at understanding and maintaining those constraints throughout the work so they don't waste time, tokens and money attempting solutions that we already know the system is going to reject.
I think this is one of the areas where companies building models and agents have a huge amount of room for improvement.
Because fully delegating development doesn't just require the result to be correct.
Getting there also has to be reasonably efficient.
Do we have to do everything with frontier models?
And this efficiency problem leads me to something else.
If we're going to have agents working, reviewing, arbitrating, retrying after hitting a guardrail and handling increasingly large amounts of context, always using the most powerful frontier model for absolutely everything doesn't seem particularly smart either.
Or cheap.
I think companies should seriously start considering running their own models in their infrastructure too.
And individuals as well.
Not necessarily to replace Claude, Codex or whatever frontier model happens to be the one next week. But to complement them.
There are tasks a local model can do well enough. Repetitive tasks. Classification. Context retrieval. Some reviews. Simple generation. Work involving code that, for security reasons, you don't want leaving your infrastructure.
And when the task requires more capability, when the local model can't handle it, when it gets stuck or when you need someone to come to the rescue, you escalate to a frontier model.
Just as I wouldn't put my best engineer on absolutely every task in the team simply because they're the best engineer.
We have to find the balance.
Which model we use for what. What stays inside. What goes outside. When it's worth paying more and when a smaller model is enough.
And this reinforces again the idea that we're not heading towards “using an AI”.
We're heading towards building teams of AIs, each one doing what makes the most sense for it, under a common set of rules that none of them controls.
How do I see this governance?
After thinking about it a lot, experimenting and taking a few hits along the way, I more or less know what my ideal framework looks like for developing software with AI without having to blindly trust it, but still having enough governance guarantees.
I want to start by telling the model what I'm trying to do, at the highest level possible. My need, my idea, the problem I want to solve. Sometimes I may even give it a pile of documents that have been written about the idea, for example, a role-playing game with its rules, its objective, its cards... And once I give it that high-level instruction and some details, I want it to generate a first plan.
That plan I do want to review. Me as an engineer, other engineers if necessary, or even other agents from different families that challenge it, look for problems, suggest alternatives and debate the decisions with it.
Once the plan is approved, I want the system itself to turn it into an actual working plan. Epics. Tasks. Dependencies. Estimates. Acceptance tests. Everything initially generated by AI, but something we can discuss, correct and complete with it before starting.
In other words, I don't want to write a huge spec myself telling it how to do things. I want to express what I need and discuss with the AI the plan it proposes to achieve it.
And once that plan is approved, then yes, let the machinery start.
Let an orchestration of agents pick up the tasks. Let them architect when architecture is needed. Let them code. Let other agents review. Let them debate when they disagree. Let us pit several models against the same problem and run tournaments if it makes sense. Let there be specific security reviews. Let tests, mutation tests, static analysis, SonarQube or SonarCloud, linters and all the tooling we consider necessary run.
But all of that inside an environment that doesn't depend on the agent remembering our rules.
Pre-commit and pre-push hooks. CI gates. Permissions. Real limits. Policies that prevent certain things from happening even if the agent wants to do them.
The agent can decide how to get there.
The system decides where it is allowed to go.
And I want incremental updates on every merge to main to keep the codebase RAG up to date, plus a full reindex on every release so that when we deploy software, the system's own knowledge is refreshed as well. And also update the wiki and the documentation so that the next agent that arrives isn't working from how the system looked three months ago, but from how it looks now.
And that RAG, that wiki, that task board, all of it should be inspectable by the engineer, so it can be corrected or understood.
For me, that's the framework.
Human intention. A generated and debated plan. Decomposed and verifiable work. Teams of specialized agents executing it. Other agents reviewing and challenging it. Deterministic tools checking it. External guardrails preventing them from stepping outside the rules. And knowledge updated after every change.
Not trusting that the agent will do it right.
Building an environment where it's very difficult for it to do it wrong and even harder for an error to reach production.
And even then, it will make mistakes.
That's why we need to build a governance system for our agents.
The compiler from decades ago
Today's AI still isn't that modern compiler we trust without practically ever looking at what it generates.
It's more like the compiler from decades ago. Good enough to change the profession. Not good enough to blindly trust.
And that's precisely why I think we're making a mistake when we conclude that we should keep manually reviewing every line.
The answer should be something else: learning to build better systems around it.
Systems that limit. And verify. And block. That allow mistakes without causing a disaster.
But also systems that are efficient and don't force the agent to discover the rules by smashing into them again and again.
Because the goal shouldn't be to make AI never make mistakes.
We can't even achieve that with humans. I wish.
The goal should be that when it makes a mistake, the system detects it before it matters. And, if possible, before it burns through a few million tokens discovering it.
And even if we solved all of this, we'd still be left with probably the hardest problem.
Getting an engineer to actually want to work this way.
Because changing the tools is one thing.
And telling someone who has spent years programming that part of what defines their profession, and something they probably love doing, is no longer going to be done by them is something very different.
And CI doesn't fix that.
This new path has to be led.
