The industry's currently obsessed with how fast we can generate code. Every morning in our engineering general channel on Microsoft Teams, someone's posting a link to a new agent or a model that can write a thousand lines of perfect frontend or backend code in seconds. We're treating implementation speed like it's still the primary obstacle to shipping software.
It isn't. Implementation's basically a solved problem.
If I want a new microservice to handle regional tax calculations for our checkout flow, I can have the scaffold, the logic, and the unit tests generated before my coffee gets cold. The bottleneck's shifted entirely. The bottleneck isn't "How do we write this?" but "How do we know this is actually right?"
The evaluation bottleneck
Evaluation's replaced implementation as the hardest part of our jobs.
I saw this play out a few weeks ago with one of our teams. They were trying to optimize the way we index product metadata into the search database. A senior engineer used an automated workflow to refactor a particularly gnarly part of the ingestion pipeline. On paper, it was a miracle. A task that usually takes a full sprint was done in an afternoon. The code was clean, followed our patterns, and the unit tests were green.
But then the mess started.
When we pushed it to our staging environment, the search results for our United Kingdom region started behaving strangely. It wasn't a crash. The monitoring dashboards looked fine. But the relevancy of products was just off. It took us six days of manual digging through logs in the log viewer to realize the generated code would subtly misinterpreted how we handle VAT-inclusive pricing in the search index.
We saved sixteen hours of coding only to spend sixty hours in the Microsoft Teams incident channel trying to figure out why the implementation was technically correct but logically wrong. Actually, it's not even that the code was bad. It just lacked the years of tribal knowledge baked into our legacy order management monolith that it was supposed to interact with.
I've realized that when we're using AI at scale, we have to treat evaluation as a first-class engineering discipline. Most of us are still pretending that a few unit tests and a quick review are enough.
We're measuring the wrong thing. We're still measuring velocity by how many tickets we move to "In Review" and how many lines of code we ship. But in an era where code's free, those metrics are worse than useless. They're dangerous. They're encouraging us to flood our repositories with volume that we don't have the capacity to verify.
The real work now's building the rigs to test these systems. It's about shadow deploys, complex regression suites, and observability that can catch logical drifts before they hit the Microsoft Teams deploy channel. It's about having the discipline to say that even if the code was written in five seconds, it still needs five days of rigorous evaluation before it's allowed anywhere near our customers.
I'm as guilty as anyone. Last Tuesday, I was looking at a demo for a new internal tool that was built almost entirely with agents. I was so impressed by the speed of the implementation that I let it go and pushed it through for a pilot without even looking at a test plan. I wanted the win. I wanted to show that we're being productive and innovative with the new stack.
I let the deadline make the decision for me. By prioritizing that raw implementation speed over a proper evaluation framework, I just added another layer of cognitive debt to a team that's already struggling to keep up with their existing services. I'm complaining about the firehose of unverified code, but I'm the one holding the nozzle.
