I have been building an interview practice tool for the last two months, writing it alongside Claude. Not "generate me an app" — ordinary daily work where I set a task, look at the result, argue with it and rewrite.
Plenty has been written about how fast AI writes code. I find the other question more interesting: what does it break in a way you cannot see?
All four bugs below are real, from my own project. What they have in common is that none of them throws. No exception, no 500, no red line in the logs. Every one of them looks correct in review. You find out when a user tells you — or you never find out at all.
1. The counter that never counted
Free plan, three interviews a month. Simple logic: questions generated, counter goes up.
Roughly this:
if (!isPaid) {
incrementInterviewsUsed(profile.userId).catch(() => {})
}
return NextResponse.json({ questions })
That reads fine. The error is swallowed deliberately, so a database hiccup does not break somebody's interview. The limit is checked above, the write happens below.
Except the write does not happen.
This is Vercel — serverless. The function returns its response, and the platform is entirely within its rights to freeze it immediately afterwards. A promise nobody awaited simply never reaches the database. Sometimes it lands, sometimes it does not. No pattern to it.
Here is how it surfaced: a user runs their fourth interview and the dashboard says "1".
The fix is one line:
if (!isPaid) {
try { await incrementInterviewsUsed(profile.userId) }
catch (e) { console.error('[Quota] increment failed', profile.userId, e) }
}
That is not the interesting part. The interesting part is why it survived for weeks.
I had tests on the quota. They checked that a request is rejected when used = 3. In other words they tested the read path. The broken thing was the write path — the one responsible for used ever reaching three in the first place.
The rule I took away and now keep written down: anything that enforces a limit or touches money gets tested on the write path, not on the logic surrounding it.
2. The model obeys the last thing it read
The product generates questions for a role and a seniority level. The complaint: "I picked senior and I am getting junior questions."
The prompt was assembled from blocks, roughly like this:
const prompt = `
${roleGuide}
${mixRule} // "60% theory, 40% practice"
${levelGuide[difficulty]} // "for senior: architecture, trade-offs..."
`
And mixRule said something along the lines of "these proportions override the block above."
The level block was interpolated after it.
A model does not read a prompt as a specification with a hierarchy. It reads it as text. When one place says "this overrides the block above" and something further down says otherwise, the thing closest to the end wins.
There was a second layer to the same problem. The proportions rule historically lived inside the branch for QA roles. Twenty-one roles out of twenty-three never saw it at all. That one was not the AI's mistake — it was mine, faithfully replicated when it added new roles by following the existing pattern.
Fixed by reordering the blocks and lifting the rule out of the branch. The broader lesson: a prompt is code with no compiler. Nothing will tell you two instructions contradict each other. The only way to find it is to read the assembled prompt end to end, with your eyes, the way the model sees it.
3. Zero interviews for every paying customer
There is a B2B tier: a team, with an admin dashboard showing each member. Among other things, how many interviews each person has done.
That number came from interviews_used on the profile.
The same field from bug one. The free plan quota counter, which is deliberately never incremented for paid plans — why count something that is not limited.
The team plan is a paid plan.
So on the single screen a company pays for twenty seats to look at, every employee showed zero, permanently. The data existed — interviews were happening, scores were being saved. The dashboard was simply reading the wrong column.
This one is not a technology mistake. It is a mistake about one field meaning two different things, with nothing in the name to say so. interviews_used reads as "interviews this person has done". It actually means "how much has been drawn down from the free quota".
Everything is now counted from the sessions themselves, and the quota field stayed a quota field.
4. The schema did not exist
The worst of the four, even though it broke nothing.
The product runs on two tables — interview sessions, and per-answer feedback. I created both by hand in the Supabase dashboard at some point. Fast, convenient, works.
The migrations folder in the repository described different tables. The ones the project started with and stopped using long ago.
Nothing failed. Nothing could fail: production has the tables, the code queries them, all good. It would have diverged at exactly one moment — standing up a staging environment, or restoring the database after an incident. Then the application would go looking for tables that are not there and fail to come up.
I went a long time without noticing precisely because there are no symptoms at all. It turned up by accident while I was writing team reports and went to check which columns a table actually had.
One more detail from the same dig. The original migration had:
check (plan in ('free', 'basic', 'pro'))
The team tier came later. I fixed the constraint in the dashboard and never carried it back to the repository. On a fresh database every B2B purchase would have been rejected by Postgres.
The fix: a migration describing what actually runs, plus a test that reads every table reference out of the source and requires each name to be created in a migration. The next hand-made table fails CI.
for (const [table, files] of queriedTables()) {
it(`creates ${table}`, () => {
const created = new RegExp(`create table (if not exists )?${table}\\b`)
.test(migrationSql)
expect(created, `${table} is queried in ${files[0]} but no migration creates it`)
.toBe(true)
})
}
There is a nice epilogue. The first time I ran that migration against the live database it failed with cannot change return type of existing function. Two columns in my version of a function were in a different order from the real one. The application reads fields by name and does not care — but Postgres treats the set of output parameters as a row type and refuses outright. The database itself showed me how far the description had drifted from reality.
The one that got a real user
While writing this I found a fifth, and it is the one that actually cost something.
The interview screen requested its questions on every mount. Generating questions charged a free interview. So a refresh, a back-and-forward, or the installed app being relaunched quietly spent another of somebody's three.
The first stranger who ever found this product did exactly that: two sessions, forty seconds apart, zero answers in either. She left having lost two thirds of her free quota without seeing the tool work once.
Generating questions is now free, and the interview is only charged when the first answer is sent. A reload picks up where it left off.
And when I wrote the test for it, the test failed — because the database mock returned nothing from a write, while real PostgREST returns the rows it touched. The product code was right; the thing meant to verify it was wrong, and it failed in the most convincing way possible, by quietly reporting that nothing had happened.
What I take from this
What got me was not that the AI makes mistakes. Everyone does.
It was how it makes them. A person who does not understand the task writes code you can see is wrong — it is clumsy, it does not compile, it falls over on the first run. Code written by a model looks like code written by someone experienced. Sensible names, tidy structure, thoughtful error handling. And exactly where it lacks context — how a serverless function gets frozen, what a particular column has historically meant — it will confidently write something that looks reasonable and is wrong.
Three things that actually work against this, in my experience:
Test the write path, not the logic around it. My most expensive bug was in a line I wrote myself and verified myself — from the reading side.
Write tests that compare the code to the outside world. Not "this function returns the right value", but "what the migrations declare matches what the code asks for". Those catch an entire class of problem that unit tests cannot see by definition.
Check it on the running product. Four of the five above were impossible to spot in code. You only see them by opening the screen and looking at the number.
None of this is new advice. What is new is how much it matters now, when the volume of plausible-looking code you did not write yourself goes up by an order of magnitude.
