Summary
When I point something out to a conversational AI agent, the feedback disappears once the session ends, and in the next session I end up pointing out the same thing all over again. I wanted to find where to record that feedback so it would take effect in the next session too, so across two projects that differ completely in field and in build, a desktop tool I built myself and a repository for research drafts and research operations, I reread the last few weeks of commit history and design documents. What I found was that the two, while proceeding separately, had sorted where feedback should live into exactly the same three layers.
The three layers are these: the instruction file the agent loads every session together with the documents that hold the operating rules; skills that gather up fixed procedures; and a memory outside the session that keeps only the history of decisions. Which layer something goes into is decided by a single criterion, whether another session, looking only at the repository, can reproduce the same operation and the same quality. The conclusion I reached was this: choose, every time, which layer a piece of feedback belongs in, instead of fixing it on the spot.
There were failures along the way. I kept writing general rules into memory, and the discipline did not reproduce in other sessions. After that, I moved a summary of the operating rules into the top-level instruction file, and I put in place a hook that prompts a reminder just before an edit without stopping the operation, along with a two-layer test that separates what a machine can judge from what requires judging meaning. These happened on separate days over several weeks.
Feedback disappears when the session ends
For the past two or three weeks I have been running two projects in parallel, a desktop tool I built myself and a repository for research drafts and research operations. The whole time, I kept finding myself pointing out the same things to a conversational AI agent over and over. Whatever I point out, the feedback does not survive once the session ends, and in the next session I end up pointing out the same thing from scratch. What I write about in this article is how to choose where to record that feedback so it takes effect in the next session, which has not received the feedback. These two projects, completely different in field and in build, arrived separately at the very same shape of sorting feedback into three layers. Because they matched, I decided to write down the three-layer split, and the single criterion for deciding which layer to use, in a form I can hand to the next person who runs into the same trouble.
Three layers and one criterion
The three-layer split goes like this. General rules that another session needs in order to reproduce the same deliverable from the repository alone go into the instruction file the agent loads every session and into the documents that hold the operating rules. Work whose procedure is settled and can be repeated mechanically goes into a skill that gathers up that procedure. And only the primary record of the decision history, when I received what feedback and where I moved it, goes into the memory that the agent carries outside the session.
There is only one criterion for the sorting, whether another session, looking only at the repository, can reproduce the same operation and the same quality. Put what is needed for reproduction inside the repository, and put into memory only the history that is personal and is not used for reproduction. That alone does the sorting.
The failure of writing general rules into memory
The reason I arrived at this criterion is that I once stumbled by doing the opposite. In one of the projects, a practice of writing feedback straight into memory the moment I received it had, before I noticed, become an unspoken habit, and there was a period when I kept writing even general writing discipline, the kind that should work for anyone who reads it, into memory one after another. The operating rules themselves were written in the repository, as documents. But those documents sat in a position where they only took effect if a new session actively went to read them. Because of that, even when another session or an agent running in parallel looked at the repository alone, the discipline was not reproduced at all.
So I moved a summary of the operating-rules documents over to the top-level instruction file, the one that is always read every session. I rewrote the handful of general rules I had put in memory into brief records that keep only the background of the feedback I received, so the rules themselves live in the instruction file and the skills, with memory pointing to them.
Shifting toward a layer that works even when it is not read
Still, even after moving things into the instruction file, the weakness remains that the operation depends on the file being read at all. So I shifted one more step, toward a layer that works even when it is not read. In one project, I put in a hook that runs automatically just before editing design, Japanese wording, or documents, so that it surfaces a prompt to read the relevant skill. This hook does not issue a permission decision; it only passes along extra information, and it never stops the operation. In the other project, I converted some of the feedback into automated tests and split them into two layers: things a machine can settle as black or white, such as character counts, punctuation, and formatting, go to a program check, while judgments that depend on context, such as meaning and logic, use the AI itself as the judge.
That said, tests are not finished once they are in; weaknesses show up on the test side too. For example, the test that judges meaning would demand that I add hedging words and weaken the claim even in the body text of a piece written on a day when the number of cases I could observe for the research was small. So I placed a file that declares in advance how many cases and how many days I could observe on that day, and I loosened the judgment so it only drops claims that go beyond the declared range. I have also been fixing two things in the mechanism itself: it returned only a failure without showing how to fix it, and it failed the text by mistake because the context of the other sections was not passed in.
There is not much firm evidence I could use to judge that this reorganization into three layers was complete. As I touched on earlier, the hook does not change the permissions of an operation; it only prompts a reminder. Whether that hook is broken is guarded by a test that takes the script's location out of the registered hooks and checks that the file actually exists. What I could confirm goes about this far.
The dependence on being read that still remains
Even after all this, the dependence on being read has not disappeared. The mechanism that automatically runs checks on every code change sees only part of the program checks, and judging meaning stays a matter of running it by hand. There also remains a trap: if I am not signed in to the service that runs the judge, the judgment passes straight through and the test looks like it passed. The instruction file side has its own structural cost, because as the instruction file grows, the amount the agent loads and processes each session grows with it. I have not yet obtained firsthand data measuring how much slower that makes things.
Whether recurrences of the same kind of feedback have really decreased, I also do not yet know. All I can pull from the commit history is the trend in how many times I wrote rules and in the number of tests, because there is no data comparing the recurrence rate under matched conditions before and after putting this mechanism in.
The human work left after shifting toward the machine
To the extent I shifted the checking toward the machine, the boundary that humans judge stood out more clearly instead. It is the line between which checks to leave to the machine and which not to. Every time a test fails, it returns a suggested fix and a distinction, whether that piece of feedback is fixed by rewriting the body or whether it should go back to the measurement that backs it up. How I act on that is still in my hands. From my side, I have also given instructions: cut the tests that only match form mechanically, because they miss things, and do not turn rules I never actually raised into tests. And I think that deciding what to record as a rule and what not to record is itself work that still remains on the human side.
What I arrived at in the end was a practice of choosing, every time, which layer to put feedback in, rather than fixing it on the spot. If there is someone repeating the same feedback to an AI agent over and over, I would be glad if this article reached them as one way to choose where to put that feedback so it takes effect in the next session too.
Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.
