TL;DR
I let my AI coding agent auto-generate changelog entries for every merged PR, and the first month of output was unreadable — diff-summaries no human wanted to read. I fixed it with a two-pass process (draft + "would a user care?" review) and a small set of rules that cut noise by more than half. Here's what actually worked, what didn't, and the prompt structure I landed on.
The Problem
I run an autonomous coding agent that ships small changes daily — bug fixes, refactors, dependency bumps, the usual grind. Early on I had it write the changelog entry for every PR automatically, figuring "it already knows what changed, just summarize it."
The first batch of entries looked like this:
- Updated `retry_handler.py` to add exponential backoff logic in the
`_should_retry` method and modified the `RetryConfig` dataclass to
include a new `jitter` field with default value 0.1
Technically correct. Useless to anyone who isn't reading the diff. Real users didn't care about _should_retry or a dataclass field — they cared that "flaky network requests now retry automatically instead of failing outright." My changelog was a commit log with worse formatting.
This mattered because I was pushing these changelogs into a public release feed. A few subscribers pinged me asking what half the entries even meant. That's the moment I realized "summarize the diff" and "write a changelog" are not the same task, and I'd been asking the agent to do the first while expecting the second.
How I Solved It
Failure mode #1: the agent summarizes code, not impact
My first prompt was basically:
Here's the diff for this PR. Write a one-line changelog entry.
This produces exactly what you'd expect: a compressed diff description. The agent has no signal about what actually matters to a reader, so it defaults to describing what changed in the code instead of what changed for the user.
The fix was forcing a translation step. I split the job into two prompts instead of one:
Prompt 1 — extract the user-facing effect:
You are not summarizing code. You are answering: "If I didn't read this
diff, what would I notice differently about this product tomorrow?"
Diff:
<diff>
If there is no user-facing effect (internal refactor, test-only change,
CI config), respond with exactly: NO_USER_IMPACT
That NO_USER_IMPACT escape hatch turned out to be the single highest-leverage line in the whole prompt. Before I added it, the agent would invent a user-facing angle for pure refactors just because I asked it to find one. Giving it permission to say "nothing to report" cut fabricated entries almost entirely.
Prompt 2 — turn the effect into a changelog line:
Turn this user-facing effect into one changelog line, in the style of
[Stripe / Linear / your product's changelog].
Rules:
- Lead with the verb (Fixed / Added / Improved / Removed)
- No file names, function names, or internal module references
- No implementation detail ("using X algorithm", "via Y library")
- Under 20 words
- If the effect is genuinely minor, it's fine to be blunt about that
Effect: <output from prompt 1>
Splitting "what happened" from "how to phrase it" mattered more than I expected. When I tried to do both in one prompt, the model would anchor on the diff's vocabulary no matter how I phrased the instructions — old technical terms kept leaking into the final line. Forcing an intermediate, diff-free representation broke that anchoring.
Failure mode #2: no verification, so garbage ships
Even with the two-pass prompt, maybe 1 in 8 entries was still bad — either too vague ("Improved reliability") or still leaking an internal term. I added a lightweight verifier as a third pass, run by a separate agent call with no access to the original diff:
Read only this changelog line: "<line>"
Would this make sense to someone who has never seen the codebase?
Fail it if it: references internal names, is vague to the point of
being meaningless, or assumes context the reader doesn't have.
Respond PASS or FAIL with a one-sentence reason.
Running the verifier "blind" (diff-free) was deliberate — if it can see the diff, it starts grading the summary against the code again instead of against a naive reader's understanding, which defeats the point.
flowchart LR
A[Merged PR diff] --> B[Pass 1: extract user-facing effect]
B -->|NO_USER_IMPACT| X[Skip entry]
B -->|effect found| C[Pass 2: phrase as changelog line]
C --> D[Pass 3: blind verifier]
D -->|FAIL| C
D -->|PASS| E[Publish to changelog]
Anything that fails goes back through Pass 2 once with the verifier's reason attached as feedback. If it fails twice, I drop the entry entirely rather than ship something bad — a missing changelog line is much cheaper than a confusing one.
Failure mode #3: grouping and ordering made the whole changelog feel noisy, even when individual lines were fine
Once individual lines got good, I still had a readability problem: a day with six merged PRs produced six flat bullet points in arbitrary merge order. A security fix sat next to a copy tweak next to a performance improvement, all weighted the same. Nobody scans six identically-formatted bullets carefully — they skim the first two and stop.
I added a fourth, batching pass that runs once per release instead of once per PR:
You have N changelog lines for today's release. Group them under
these headers, in this order, and omit any header with no entries:
Security
Fixed
Improved
Added
Removed
Within each group, order by how much a typical user would care,
most important first. Do not edit the wording of any line, only
group and reorder them.
Constraining this pass to "group and reorder only, don't touch wording" was intentional — I'd already spent two prompts getting the wording right, and letting a fourth pass rewrite things risked reintroducing the exact problems Pass 2 and Pass 3 had just fixed. It's tempting to let a later step "polish" everything one more time, but each additional pass that's allowed to rewrite is another chance to drift back toward diff-speak.
This one change did more for perceived quality than either of the wording fixes. A five-line changelog with a "Security" header at the top reads as trustworthy in a way six flat bullets never did, even when the underlying content was identical.
What I tried that didn't work
For completeness, two things I abandoned:
- Giving the agent the PR title and description as extra context. I assumed this would help it infer user impact faster. Instead it usually just copied the PR title's phrasing verbatim, including internal terminology, which defeated the whole point of Pass 1. Diff-only input, no metadata, produced more honest output.
- Asking for changelog entries at PR-open time instead of merge time. The idea was to save a pass by writing the entry once. In practice, PRs change enough between open and merge (scope shrinks, edge cases get cut) that the entry was stale about a quarter of the time. Generating from the final merged diff, after the fact, was slower but far more accurate.
Results after a month
- Entries dropped as
NO_USER_IMPACT: ~35% of merged PRs (mostly refactors, test additions, CI tweaks — correctly excluded) - Verifier fail rate on first pass: ~13%, dropped to ~3% after I tuned Prompt 2's rules based on recurring failure reasons
- Zero reader complaints since switching, versus multiple per week before
Lessons Learned
- "Summarize the diff" and "write a changelog" are different tasks — treat them as different prompts. Trying to get one prompt to both understand the code and write good user-facing copy consistently under-performs splitting the work.
- Giving the model an explicit "nothing to report" escape hatch prevents fabrication. Without it, the agent will manufacture a user-facing angle for changes that don't have one, because it's trying to satisfy your request rather than tell you the truth.
- A verifier that can't see the source input catches a different class of error than one that can. A diff-aware reviewer grades against the code; a diff-blind reviewer grades against the reader's actual experience. You want the second one for anything user-facing.
- Vocabulary leaks unless you force an intermediate representation. If the final output is generated directly from technical input, technical words sneak through no matter how firmly you instruct otherwise. Add a translation step in between.
- A dropped changelog entry is a better failure mode than a bad one. Optimizing for "never publish something confusing" over "always publish something" was the right tradeoff for anything public-facing.
What's Next
I'm extending the same three-pass pattern to PR descriptions and release summaries — same failure mode (technical leakage, missing "why should I care"), same fix. I'm also experimenting with letting the verifier's rejection reasons accumulate into a running list of house style rules, so Prompt 2 gets a little sharper every week instead of repeating the same mistakes.
Wrap-up
If you're auto-generating anything user-facing from a diff, split "understand the change" from "phrase the change" into separate calls, and don't skip the blind verification pass — it's cheap and it catches a real category of bad output the first two passes miss.
If this was useful, follow me here for more of these build-in-public lessons, and let me know in the comments if you've solved changelog generation differently — I'd like to compare notes.
