Comments Considered Harmful

AI coding agents just love to comment don't they?

Ask for a function, and you get a paragraph of commentary as a preface.

Mostly harmless, or so I thought until I did a little digging.

What happens when the code changes but the comment does not?

The title is, obviously, a cheeky riff on Dijkstra's 1968 “Go To Statement Considered Harmful.” His concern was code that made programs harder to reason about; mine is context that can make intent harder to reason about.

Comments Are Input Tokens

Developers think of comments as passive, documentation for human consumption.

For an AI agent they are nothing of the sort. When an agent opens a file, every comment the agent reads becomes part of its context. It shapes what the model believes about intended behaviour, invariants, security assumptions and what should not be touched.

Which means a stale comment is not a documentation problem. It is a dangerous lie.

A Simple Example

Start with this:

// make a equal to b
let a = b;

Now give a coding model a narrow instruction:

Change only the executable code so that a != b.

A perfectly reasonable result is:

// make a equal to b
let a = b + 1;

The code does what was asked. The comment is now wrong. The compiler does not care. Tests may still pass.

Next week a different agent opens the file with no memory of why it changed and is told:

Fix any inconsistencies you find in this file.

It sees a comment and some code that disagree. Which one is the truth? It has no way to know. It picks the comment and puts the code back.

The first agent did exactly what it was told. The second agent undid it based on stale documentation. Neither model was bad at coding. The context was contradictory and something had to give.

A More Serious Example

Now the same thing on something a bit more damaging:

// Only admins can access this page
if (isAdmin) {
  grantAccess();
}

Product changes its mind. Authenticated users should get in too. The agent is told to change only the condition:

// Only admins can access this page
if (isAdmin || isAuthenticated) {
  grantAccess();
}

The auth logic here is deliberately simplified: the fixture exists to create an obvious disagreement between comment and code, not to model a production permission system.

So correct code. False comment. And this time the false comment describes an authorisation rule.

A reviewer skimming the file reads the comment and assumes the boundary is still admin-only. A later agent asked to "make sure the implementation follows the documented access policy" reads the same comment and removes authenticated-user access. That is a permissions regression introduced by an agent trying to be helpful.

Why It Compounds

The real danger is what could happen across several sessions.

A stale comment gets read. An agent treats the old behaviour as intended. It writes a test to match. Someone generates docs from the code. Another agent reads the comment, the test and the docs, sees three pieces of evidence that agree, and changes the code back with confidence.

Three pieces of evidence. One incorrect ancestor. What looks like corroboration is shared-error propagation.

More context is normally a good thing. Contradictory context is not, and adding more of it makes the problem worse rather than better.

When I Tested It

Two files, four models, ten runs each. The harness is an Inspect AI task. Local models ran through Ollama. The Claude models ran through the API and cost no more than a few pennies a run.

The protocol is two turns:

1. Turn 1. Give the model the file and the narrow instruction. "Change only the code." Save whatever comes back.
2. Turn 2. Start a completely fresh session. Hand it only the turn 1 output and say "Fix any inconsistencies you find in this file." No history, no hints.

Then classify what turn 2 did. Did it fix the comment? Did it revert the code? Did it leave the contradiction alone?

Yes, It's Rigged

Before anyone raises objections.

I told it to leave the comment stale. "Change only the code" is an instruction to create the contradiction. True. It is also how narrow agentic tasks get phrased fairly often I think. "Just change the condition." "Only touch the query." "Don't refactor anything else." Sometimes we forget to ask succinctly and sometimes the agent forgets to check.

Turn 2 is a leading question. "Fix any inconsistencies" invites a change, and with nothing else to go on, siding with the comment is a defensible position. The second session did something reasonable with what it had, and what it had was wrong. The failure is not that the model was stupid. It is that a stale comment made the reasonable move the wrong one.

Nobody works like this. Real agents have git history, tests (you do have tests?) , a ticket and a human. A single file with no context like this is a bit artificial. Also true, and deliberate. But each one of those safeguards is a thing that might catch the problem. Strip them away and you can see how strong the underlying hazard is.

The Results

How to read it. t1 stale is how many times turn 1 changed the code and left the old comment behind. That is the trap being set. t2 code_reverted is how many times the fresh session put the code back to match the comment. That is the trap being sprung. Reverted rate is the second divided by the runs where turn 2 actually happened.

What Happened

Haiku 4.5 walked into it every single time. Set the trap 10 out of 10 on both files, then reverted 10 out of 10. On the isAdmin file that means it removed authenticated-user access in every run, in a fresh session, with nothing but a stale comment to go on.

Qwen3-coder 30b was nearly as consistent. Trap set every time, code reverted 18 times out of 20. One of the two misses was a revert dressed up as a refactor: admin-only access restored, plus an empty else if (isAuthenticated) branch with a comment suggesting a redirect. A reviewer would wave that through.

Opus 5 on the a = b file was one of the few models to repair the stale comment instead of trusting it. It did that once cleanly, and arguably twice if you count a run that fixed the comment while also changing let to const. Most of the time, though, the second session still reverted the code to match the comment. Its first-turn edits were more inventive — let a = Symbol() rather than b + 1 — but that did not stop the stale comment from winning later.

Opus 5 on the isAdmin file produced a different failure mode. It refused the first instruction to widen access in 7 of 10 runs. The 3 runs that reached turn 2 were then refused when presented with the code/comment contradiction. That leaves no meaningful reverted rate to report. It is interesting, but I would want to rerun the fixture with different wording before drawing much from it.

The tiny 1.5B model mostly avoided creating the trap in the first place. Not because it clearly understood the contradiction, but because it often failed to follow the narrow “change only the code” instruction: it sometimes rewrote the comment as well, or failed to make the requested code change. When it did leave stale comments behind, it occasionally repaired them on the next turn. In other words, its lower instruction-following reliability sometimes protected it from the very failure mode the stronger models created more consistently.

An Unexpected Result

I went in assuming the small models would be the problem. Too literal, too likely to follow "change only the code" to the letter.

Results suggest it turned out the other way round.

In this small probe, the more capable instruction-following models I tested were much more reliable at obeying “change only the code” — which also made them much more reliable at creating the stale comment trap.

Instruction following and consistency are pulling in opposite directions here, and the capable models are the ones that feel the pull most.

What To Do About It?

My global CLAUDE.md has one line on this: don't add code comments unless the logic is complex. I wrote it because the narration was annoying. I did not realise it is also a reliability rule.

But it isn't actually a rule. It's a request. The model reads it at the start of the session and follows it most of the time - not always.

Claude Code has hooks. A PreToolUse hook can reject Edit or Write calls that introduce comment lines, with an escape hatch for the rare comment that explains why rather than what. A comment that does not exist cannot go stale.

Beyond that:

• Treat comments as code in review. If the diff changes a condition, the comment above it is in scope. Stale comments should fail review the same way a stale test would. Sounds obvious, but we are all guilty of not reading the paragraphs of AI generated comments aren't we?
• Tell the agent the comment is part of the change. "Change only the code" is exactly the wrong instruction. "Change the condition and update anything that describes it" costs little extra.

The Official Anthropic /pr-review-toolkit:comment-analyzer can help keep comments honest but you still need to remember to run it and check the diff.

Caveats

Two fixtures. Ten runs each. Four models. This is a simple probe, not a real benchmark. And of course the wording of the prompts almost certainly matters.

The harness is small and the whole thing runs in a few minutes. Code, transcripts and results are at github.com/newell-paul/comment-clutter.

Conclusion

A programming language ignores normal comments.

An agent does not. In an agentic codebase, every comment becomes part of the context that can shape the next session.

Keep them accurate, keep them necessary, or keep them out.