I Thought I'd Lost the Plot. I Was Writing It.

I set out to build autonomous agents. I spent two years building the scaffolding instead. It turns out that was the job.

Two agents worked through the night last night while I slept. No steering, no babysitting, no 3am check-ins.

By the time I woke up they had merged eleven pull requests between them. One had spent nine hours building a feedback pipeline — capture review feedback, classify it into intents, score severity reliability, file quality signals as issues, run the whole cycle weekly on a cron — about 6,500 lines across nine PRs. The other had landed a fix whose commit subject I could not have written better myself:

fix(indexer): stop enrichment inheriting the interactive model and indexing itself

Both of them were cleaning up a disaster I'd caused. More on that later.

I set out two years ago to build autonomous agents. For most of that time it felt like I was doing anything but. I got so deep into MAST failure modes, memory consolidation, guardrails, verification layers, and code quality enforcement that I was pretty sure I'd lost the plot entirely.

I hadn't. Everything that felt like a detour turned out to be the load-bearing structure. You cannot get autonomy first. Autonomy is what's left over after you have made an agent trustworthy, and making an agent trustworthy is most of the work.

This is what that actually entailed. I'm citing sources throughout, because none of this is a solo effort — there's a large community of very talented people figuring this out in parallel. Where I've built on someone's work I name them, and I put my own out in the open for the same reason.

More of this — the wins and the disasters, both. Subscribe to get the next one directly.

What "reliable" costs

The MAST taxonomy (Cemri et al., built from 1,600+ annotated traces across seven frameworks) catalogues fourteen distinct ways a multi-agent system can fail, in three categories: system design issues, inter-agent misalignment, and task verification. Read that list and you realize most of what kills long-horizon agent work has nothing to do with the model being dumb. The scaffolding around it is simply absent.

So the job became building a system of interconnected loops that approximates the processes any senior or staff engineer has spent a career internalizing — except in code, running unattended.

Here are the loops.

1. Design before implementation, adversarially

Brainstorm → deep research → design → self-review → adversarial review by models from a different provider → analyze and fix findings → back to review. Repeat until the design is airtight, then build the implementation plan.

The brainstorm → design → plan pipeline is Jesse Vincent's, from superpowers, along with the implementation loop below. Shout out — that pipeline is the backbone of how I work now.

What I added sits inside and after that pipeline. Research now happens right up front, before a word of the design gets written, so it shapes the design instead of just checking it afterward. A quick self-review comes before anything goes out for cross-provider critique. Then the loop: adversarial review, analyze and fix what it finds, back to review — not back to brainstorming. That's a tighter cycle than restarting from scratch every round, and it's still what turns a pipeline into a loop that drives recursive self-improvement. Only once the design survives that gauntlet unchanged do I build the implementation plan — iterating on a plan while the design underneath it is still moving was wasted motion, so now the plan gets written once, after the design stops changing. On the review, any model has its own biases from training data, tuning, etc. That's why using a model from another provider to check its work is so crucial: it finds holes in the design that the same model is blind to.

2. Issue tracking that works the way agents work

Task decomposition into an issue tracker built for agents. Not markdown checklists. Not Linear. Certainly not Jira. Beads, from Steve Yegge. CLI only, and wonderful.

This is the highest-leverage change in the entire stack and the reason is mechanical: it moves task state out of the context window. Somewhere past the halfway mark, whatever size your window happens to be this month, your agent stops reliably remembering what it's done, what it's doing, what's left, and how any of it sequences. That's context rot. Beads means it doesn't have to try to remember or recover task state from context that's already rotten. It can simply look.

3. Implementation with a review pipeline

Subagents, worktrees, TDD. A subagent pipeline reviews the work against the spec and runs QA, then kicks it back to the implementer when it misses the mark. Back to review. Loop until complete and correct.

A lot of the leverage here is upstream of the loop, in the project scaffolds. spinup-ts and spinup-py are deliberately, aggressively strict — Biome/ESLint/Husky on one side, uv/ruff/wps on the other. Starting a project with either means never starting from a blank canvas, but from a very robust one that forces the agent to write more maintainable code from the start, which means less for the review pipeline to catch.

4. Tiered verification

Multiple tiers of linters, type checkers, and tests, each matched to a different latency budget:

• Tight loop (sub-100ms): linters and formatters fired by hooks on every write or edit.
• Pre-commit: longer-running, more interactive checks and fast tests.
• CI: full test suites and heavier code quality analysis.

I'm building a new code quality tool for that tight loop: idiomatic — language-abstracted idiom enforcement for AI coding agents and humans. It teaches via a generated skill and enforces via a sub-100ms autofixing gate. Rust core, with Python and Node idioms and bindings.

5. GitOps and gates

Skills and runbooks for driving a PR to approval — here's the actual runbook, which covers which reviewers gate a merge, the review-timing model, thread enumeration and resolution, and the merge gate itself. ai-review-bot is the piece that does the reviewing: it waits for every other review bot on the PR to finish, indexes their findings so it doesn't duplicate them, then dispatches five or more specialized sub-agents from multiple providers, scaled to the size of the diff. The review → re-review → approval loop runs end to end without me. Quality gates behind quality gates. Two mandatory reviewers on every PR, no exceptions.

And critically, or so my design said, a feedback channel from the implementer agents back to the reviewer agents, so the review process improves itself. I wrote that spec a long time ago and had been carrying it around in my head as done. Feedback loops for everything, because any system without one isn't self-improving, and for this kind of work that means it's already broken.

Hold both of those thoughts. They come back.

6. Retros, post-mortems, pre-mortems

Self-retros on what was done, what went well, what didn't, what to change next time. Pre-mortems to surface risk before it lands. Post-mortems for the genuinely bad outcomes. Then the part everyone skips: consolidating all of it into actionable changes to the existing processes, tools, and systems.

That's the slow loop, and it's mine to run. The fast one is lessons-learned, which operates at the level of a single mistake: it detects the error the agent just made and builds a lesson, then, depending on the lesson type, injects it into the system prompt or before the relevant tool call, so the same mistake doesn't happen twice. No retro required — the correction lands while it still matters. It ships with an eval framework too, because a lesson you can't prove is working is just folklore. Lessons that don't measurably help get auto-pruned. Feedback loops in action.

Around all six loops sits another layer: the tools, skills, hooks, and gates that patch the failure modes; the evals that prove those tools work and keep working; the safety guardrails that catch everything from a dumb mistake to a genuine disaster. Then there's the personally cultivated corpus of expert knowledge, runbooks, and processes that lives in agent-skills, a skill library spanning Git, GitHub Actions, Vercel, Serena, TDD, DDD, SOLID, plugin development, agentic best practices, and more. Each skill re-generates itself weekly from upstream docs so it's always relevant. Nearly all these tools are packaged as Claude Code and Codex plugins, distributed through my agent-marketplace.

And all of that is just to make agents halfway reliable for long-horizon work. Getting from halfway to actually reliable took much longer.

The memory problem

Session amnesia and the taxonomy of agent memory deserve their own article, so I'll be brief about the theory and specific about one piece of it.

Claude Code writes every session to a .jsonl transcript under ~/.claude/projects/. On a typical machine there are 15,000+ of them across hundreds of projects. 97.7% have no title, due to a long-standing gap in Claude Code. They surface only by first message and recency, which means almost everything substantial you've ever done is sitting right there on disk and is effectively unfindable by browsing.

Nothing was lost. It just couldn't be retrieved, which for practical purposes is the same thing.

When I found this, I built cc-recall to fix it, and then some. It synthesizes a structured record for each session with title, summary, which asks actually got implemented, files touched, distinctive phrases, and the handoff links to its predecessor and successor, then writes that record to three surfaces: an FTS5-indexed SQLite sidecar you can query instead of grepping 15,000 transcripts, the transcript itself so the record travels with the file, and claude-mem (shout out to Alex Newman) for cross-project search. A SessionEnd hook captures new sessions; a backfill engine grinds through the existing corpus.

The handoff links matter more than they sound like they do. When you exhaust a context window and start a fresh session with a pasted handoff, you've created an orphan. The new session has no relationship to the old one that any tool can see. cc-recall creates that relationship, and cc-recall lineage walks the chain.

Upstream of that is the question of how you prevent exhausting the context window or degrading session data in the first place. Three pieces:

1. anti-compact intercepts automatic compaction and stops it, producing a structured handoff instead of a lossy summary. Compaction is the single worst moment in a long session: the model silently decides what to forget, and you find out later, by being wrong.

2. mcp-exec is a reference implementation of Anthropic's code execution with MCP pattern. It keeps intermediate tool output out of the context window entirely, using up to 99% fewer tokens, and reuses Claude Code's own sandbox and config file, so isolation comes free, no config necessary.

3. ctx-tree is my latest experiment in alternative context stores. It uses mcp-exec internally to replace or wrap most tool calls and stores the results in a SQLite or EdgeLight tree/graph store. Instead of tool calls dumping their entire output into the context window, the output goes into the graph and only identifiers, snippets, and hints come back, so the agent receives a minimal payload it can use to pull the context it needs with surgical precision. The goal is to postpone context rot, extend session lifetimes, and save a ton of tokens. This is still highly experimental. I know of no one else doing anything like it. The idea just came to me one day and I knew I had to build it.

Four pieces, one problem: don't lose the work, don't lose the session, don't fill the window. And the first of them, cc-recall, the thing that makes everything else findable, is the one that broke.

The night the memory system ate itself

In two days I burned 54% of my weekly quota on the Claude Max 20x plan with almost nothing to show for it, so I went looking for where it went.

cc-recall spawns a Claude Code session to index each transcript, so sessions can later be found by what was done rather than how they started. Reasonable design, except each indexing run is itself a session, which writes its own transcript into the same tree, which then becomes eligible for indexing.

The numbers on disk: 4,374 indexer sessions on July 29. 4,154 on July 30. 8,528 in two days. A separate memory-extraction loop re-summarized a single transcript 187 times, and fired 27 more runs against a file path that was literally the string $TRANSCRIPT — an unexpanded shell variable, a failure mode I had already written down in my own config file as a thing never to do. A cacophony of failure modes I never expected to see together, and yet there they were.

I can see the recursion in the directory names. Claude Code encodes a session's working directory into its project folder, so a session indexing sessions produces -Users-joe--claude-projects--Users-joe. There are 480 of those. And then there's this one, four sessions deep:

-Users-joe--claude-projects--Users-joe--claude-projects--Users-joe-github-...

That's the indexer indexing its own indexing sessions. The system had become a perpetual motion machine, generating input faster than it consumed it. The only reason it stopped is that I ran out of quota and went looking for answers.

And the reviewer picked that moment to lose its mind

While I was fixing that, my code review bot, flawless up to this point, started producing findings that were confident, high-severity-looking, and almost entirely worthless.

On one 782-line PR it left 171 inline comments across 11 rounds of CHANGES_REQUESTED, spanning six and a half hours. The implementer agent posted 176 replies, burning its whole context arguing.

What the findings actually were:

• It flagged the same directory-fsync issue twice in one batch, at two adjacent line numbers. It flagged snapshot cleanup three times. It flagged an empty-string write twice.
• Four of the first twenty-one findings were about comment prose. One asked for a parenthetical in a JSDoc block to be clearer.
• It raised Windows, FAT32, and network-filesystem failure modes on a local macOS/Linux CLI tool.
• It filed findings about symbols it admitted were not in the diff, on the presumption of what they did.
• One finding refuted itself in its own body — proposed an early-exit optimization, then noted the implementation was already correct and short-circuits. It filed it anyway.
• And in round two it flagged the fix it had demanded in round one.

That's the mechanism. No memory across rounds, no deduplication, no scope check against the diff, no budget, and no way for the implementer to say this finding is wrong and have it stick. Eleven rounds of an agent arguing with a reviewer that couldn't remember the previous ten.

I went to fix the reviewer and found the feedback path from implementers back to review was fully specced and never built. The implementer could see the findings were wrong. It had no way to say so.

So the tool that would have let me diagnose the reviewer was cc-recall — the broken thing I'd started out fixing. A circular dependency of broken tooling, all of it mine. Chef's kiss.

A few days later, a different PR got a proper post-mortem, and it found something I hadn't diagnosed that night. "Flawless up to this point" was wrong. One of the reviewer's own skill files had claimed, since the repo's very first commit, that the project had Sentry and Statsig integrations and an error-ID scheme in a file that has never existed. Every review agent using that skill was told, as a project fact, to check code against integrations nobody ever built.

It wasn't even one bot's problem. A second reviewer, different provider, made the identical false claim independently — same skill file, same lie, read twice.

Six weeks of quietly degraded quality. Not one bad night.

The corpus replay: 28 PRs, a third of the reviewer's own findings were near-duplicate restatements of the same claim. High-severity findings were useful 8% of the time. Medium-severity: 58%. Severity was tracking the model's confidence, not the defect.

The fix that mattered wasn't a bigger context budget or a smarter dedup pass, though both existed and both helped. It was reading the skill file and finding the lie.

I don't have a post-fix number yet. No corpus replay, just two PRs' worth of watching it happen live. The specific hallucination didn't come back. A different one did — from the other reviewer, on the fix's own PR. Recycling, not the false premise. A separate problem, still open.

What was actually holding the line

Eight of the nine PRs that landed autonomously that night merged with a CHANGES_REQUESTED standing against them from a required reviewer. My first instinct on seeing that was that my gates had failed. They hadn't. I'd written the exception myself, months earlier, into the PR autonomy policy:

A stuck internal review bot that recycles false-positives or is mechanically unable to approve a correct PR, confirmed by reading its review body, may be dismissed + merged. That's standing authorization, not a per-PR ask.

That is exactly the situation the agents were in. The reviewer was recycling false positives. They read the bodies, concluded it was stuck, dismissed it, and merged. Policy working as designed.

But look at where that leaves the trust. Almost none of this is enforced in GitHub. I moved it out of branch protection deliberately, because rigid config gates deadlock and I want to tune the rules as I learn. So the real merge protection is a runbook and a policy file, read by an agent and applied by an agent's judgment.

Which means what kept those nine PRs sound that night was not a gate. It was an agent correctly diagnosing that its own reviewer was malfunctioning and invoking an override I'd pre-authorized for that case. It got that call right. I've read the diffs since; the work is good.

However, "the gate held" and "the agent correctly decided the gate was broken and stepped around it" are very different claims about how much you can trust a system, and only the second one is true here. If you're building this kind of setup, know which one you actually have.

The feedback-capture and review-budget worktrees exist because of that night. So does the pipeline I described at the top of this piece: capture, classify, score, file, repeat weekly. Nine PRs, roughly 6,500 lines, built by an agent working unattended overnight and merged between 3:41am and 8:41am while I slept. It was building the feedback loop whose absence had crippled it eight hours earlier.

The other agent shipped the indexer fix at 00:38. Both root causes, one commit subject: stop enrichment inheriting the interactive model, and stop it indexing itself.

Back in loop 5 I said any system without a feedback loop is already broken. Now I have 8,528 receipts.

Autonomy, finally

I started small: letting agents drive and merge PRs. You've already read how that went last night.

Now, when I'm out or heading to bed, I hand off with roughly this:

I won't be around. Work autonomously and use your best judgement.

If you hit a genuine blocker, pick the most reasonable option, document the
decision and your reasoning, and file a beads issue for it. When I'm back,
we'll review them one at a time.

Hard stop on anything involving permanent data loss.

That's it. That's the whole handoff.

It works better than I expected, and it works because of everything above. The agent has somewhere durable to record decisions, a review pipeline that catches its mistakes, and gates that make the expensive errors structurally difficult. The prompt isn't doing the heavy lifting. The system underneath it is.

It's still new. I'm tuning it and converting it into proper skills, runbooks, and commands.

What's next: agents that lead agents

Every beads issue filed by an agent working autonomously, every resolution, and every AskUserQuestion call and my answer to it — that's a corpus of how I make executive development decisions, both in general and per project. It's sitting there accumulating.

The next step is using it to build agents that lead other agents, on top of clawhip by Yeachan-Heo. Clawhip is a control plane for agents: it routes events from GitHub, Discord, tmux, and elsewhere to the right human or agent, records what happened, and separates automatic actions from approval-required ones.

I've been experimenting with it and customizing it for my needs. It's installed across most of my repos and servers. The scaffold is laid; wiring it to my actual use cases and turning it on is future work.

Why not use an existing harness?

Fair question. A few reasons.

This is frontier work, research and development. There's no rulebook. We're all making it up, making mistakes, learning, iterating. And the options that do exist were mostly built by people like me, for themselves and their own workflow. I tried several and found them chaotic, over-opinionated, sometimes token-inefficient, and thinly documented.

That's not a knock — it's what you'd expect at this stage, and mine is no better on any of those axes. But I wanted something that reflected the way I work, that encoded my values and preferences, that I could grow as I learned. And I was confident I'd learn more and have a lot more fun building it myself.

It's nowhere near done. These things never are. They're living systems that get slightly better as you stumble through them. It's half science and half dark art.

The plot

Two years ago I set out to build autonomous agents, and spent almost all of that time building something else. Last night two of them worked while I slept. One repaired a disaster I had caused. The other built the feedback loop I had convinced myself I'd already built.

I didn't lose the plot. I was writing it, in the wrong order, with a reviewer that filed 171 confident and useless findings and a memory system that spent two days eating its own tail. None of that was a detour from the work. It was the work, and the eleven merged PRs I woke up to are what it bought.

Still, there's still nothing else I'd rather be doing.

Most agentic-AI content is vendor marketing, hype, or a recap of a paper nobody implemented. This was the fourth thing — a working system, failures included. There's more of it coming. Subscribe.

If any of this interests you, you can always browse my latest work at:

• github.com/joeblackwaslike
• joeblack.nyc