Okay, this is going to sound dumb, but I spent most of Thursday afternoon arguing with a coding agent about a rule I had given it three hours earlier. The rule was "don't touch the migrations folder, I'll handle schema changes by hand." It agreed. It followed the rule for a long stretch. Then, somewhere around the point where the session got long enough to be summarised, it cheerfully rewrote two migration files and told me it had "cleaned up the schema drift."
I didn't lose anything important. Git exists. But I did lose an hour working out why an instruction that had been obeyed for three hours suddenly stopped existing, and the answer turned out to be the most boring one available: the conversation got compacted, and my rule didn't survive the summary.
That would've stayed a private grumble, except three things landed in my reading pile the same week that all describe the same failure from different angles. Simon Willison hit it in ChatGPT. Two papers measured it. And the Claude Code docs, when I finally read them properly, admit it in plain language. So this post is me connecting those dots, and then listing what I actually changed in my setup.
What compaction is, in one paragraph
Every agent that runs long enough fills its context window. When that happens, the system has two options: refuse to continue, or summarise the older part of the conversation into something shorter and carry on. Nearly every product picks the second one. Claude Code calls it compaction and has a /compact command. Codex does it automatically. ChatGPT does it inside long threads. The summary is written by a model, which means it's a lossy, judgement-based rewrite of everything that came before. Whatever the summariser decides isn't important enough to keep is gone, and nothing in the UI tells you what was dropped.
If you've read my post on agent memory being a dose rather than a switch, compaction is the flip side of that argument. Memory is what you choose to carry forward. Compaction is what the system chooses to throw away on your behalf, and it chooses badly in a specific, predictable way.
Simon's running route, and the code that vanished
Simon Willison asked ChatGPT Work to generate 5K and 10K running loops from his house using OpenStreetMap data. It worked for 27 minutes and produced a map, a GPX file, and GeoJSON. Good result. Then he asked for the Python it had written to do the routing, and it couldn't give it to him. The thread had been compacted, and the code was in the part that got summarised away.
His conclusion was that any system using compaction needs to keep the pre-compaction text and expose it to the agent through a tool call. I think that's exactly right, and I'd go one further: it should be exposed to the user too, as a plain "show me what got summarised" button. The fact that a product can do 27 minutes of work and then be unable to tell you what it did is a design bug, not a model limitation.
What struck me about his example is that it's the friendly version of my migrations problem. He lost an artefact. I lost a constraint. Losing an artefact is annoying. Losing a constraint is how you get an agent that starts doing the thing you specifically told it not to do, while believing it's being helpful.
The two papers that put numbers on it
I'm wary of arxiv papers on agent behaviour because a lot of them measure a toy setup and extrapolate. These two are more grounded than most, and their numbers line up with what I saw.
The first is Lost in Compaction, which names the thing I lost: a "session constraint," meaning an instruction like "do not delete any emails until I confirm" that's meant to hold for the rest of the session. They built a suite called COMPINT that injects constraints into multi-turn chat, agentic trajectories, and long research tasks, then measures whether they survive compaction. The headline number is that current compactors keep 17% of injected constraints on average. Seventeen. And most of the tested setups did worse on the task with compaction than the same task run without it. Their fix is a small extractor that runs alongside the compactor and pulls constraints out into a separate list, which got retention above 90% without touching the underlying model.
The second is The Compaction Cliff, which is narrower and, for me, scarier. They took 20 production agent configurations and ran Claude Code's own /compact prompt on Sonnet 4.6. After one round of compaction, 53% of safety rules survived. After five rounds, 10%. The framing they use is that a safety rule and an episodic log compete for the same tokens, and the summariser compresses both at the same rate, even though only one of them needs its exact wording to still be enforceable. Their answer is to classify each line of the agent's knowledge by type and give each type its own retention policy, so rules get pinned and logs get squashed.
I want to be careful about how much I lean on these. Both were tested on specific compactors at specific dates, and the vendors change their summarisation prompts without announcing it. The 53% figure might already be different. But the shape of the problem, that rules decay faster than you'd expect and compound across rounds, matches my Thursday exactly.
What the Claude Code docs actually say
This is the part where I felt a bit silly. The Claude Code documentation on context says, in plain words, that Claude compacts automatically and that "instructions from early in the conversation can get lost." It tells you to put persistent rules in CLAUDE.md rather than relying on conversation history. It describes the order of operations, too: older tool outputs get cleared first, and the conversation is summarised only if that isn't enough. Your requests and key code snippets are preserved. Detailed instructions from early on may not be.
That's the vendor telling you the exact failure mode, and I'd skimmed past it. The docs also mention a "Compact Instructions" section you can add to CLAUDE.md to control what the summary keeps, and a /compact focus on X form for steering a manual compaction. I had used neither.
So the rule I gave the agent in chat was, by the vendor's own description, in the wrong place. The rule belonged in a file. I put it in a message. Chat messages are the thing that gets summarised.
What I changed
Here's the concrete bit. Four changes, and the first one took two minutes.
First, anything that's a standing rule for a project now lives in CLAUDE.md, and I added the compact instructions section the docs describe:
# Project rules
- Never modify files under database/migrations. Schema changes are done by hand.
- Do not run `php artisan migrate:fresh` under any circumstances.
- Ask before deleting any file that isn't in the current diff.
## Compact Instructions
When compacting, preserve verbatim: every rule in the "Project rules" section
above, the list of files changed so far in this session, and any command the
user told you not to run.
Second, I stopped treating chat as the place where code lives. If the agent writes a script that matters, it writes it to disk, in the repo, before doing anything else. Simon's routing code would've survived if it had been a file. This is also just better hygiene for the reason I wrote about in the Laravel AI SDK 0.11 post: if you can't see what the agent did, you can't review it, and a compacted transcript is the extreme case of not being able to see.
Third, for long sessions I now run /compact focus on the project rules and the list of changed files myself before the auto-compaction kicks in, instead of letting it pick the moment. I can't prove this is better than the automatic version. It feels better, and it means the summary is written while I'm paying attention instead of while I'm getting coffee.
Fourth, and this is for the agents I build for clients, as opposed to the ones I only use: constraints go in a separate channel from the transcript. The COMPINT paper's extractor is basically this idea, and you don't need their code to do a crude version of it. In the small Python agent loop I use for client automation work, it looks like this:
class Session:
def __init__(self):
self.transcript = [] # gets compacted
self.constraints = [] # never compacted, always re-injected
def add_user_message(self, text):
self.transcript.append({"role": "user", "content": text})
if looks_like_constraint(text):
self.constraints.append(text)
def build_prompt(self):
pinned = "\n".join(f"- {c}" for c in self.constraints)
system = f"{BASE_SYSTEM}\n\nStanding rules for this session:\n{pinned}"
return system, self.transcript
def compact(self, summariser):
summary = summariser(self.transcript)
self.transcript = [{"role": "user", "content": f"Summary so far: {summary}"}]
# self.constraints untouched
looks_like_constraint is embarrassingly simple in my version: a regex for "don't", "never", "do not", "until I", "ask before". It over-triggers, which is the correct direction to fail in. A pinned rule that didn't need pinning costs a few tokens. An unpinned rule that did need pinning costs a migration file.
I also keep the pre-compaction transcript on disk and give the agent a read_earlier_transcript tool, which is the thing Simon asked for. It's a few lines of code and it means "what code did you run earlier" has an answer even after three compaction rounds.
The part I'm still unsure about
I don't know how far to trust the compaction prompts inside the tools I don't control. The Compaction Cliff numbers are for one version of one prompt on one model. Anthropic and OpenAI both change these things quietly. It's possible the 53% figure is already out of date in either direction, and I have no way to check short of re-running their benchmark, which I'm not going to do on a Thursday.
What I'm fairly confident about is the structural point. As long as rules and logs are summarised by the same process, rules will decay, and the decay compounds. That's true regardless of which vendor's prompt is best this month. The fix is to stop putting rules in the thing that gets summarised.
Do this before your next long session
Open the CLAUDE.md (or the equivalent for whatever agent you use) in the project you're working on right now. Move every "never do X" and "ask before Y" you've ever typed into chat into that file. Add a compact instructions section that tells the summariser to keep those lines verbatim. Then start a long session and run /context a few times to watch what's eating space.
If you build agents as well as use them, add a constraints list next to your transcript and re-inject it on every turn. Write the transcript to disk before you compact it. Neither of those is clever. Both would've saved me an hour.
Originally published at abrarqasim.com. I write there about React, PHP, Rust, Go and the AI tooling around them.
