The first obstacle on the road to training an StS-playing agent is creating the training environment. Unlike humans, who can learn from instruction and very few samples, agents need to play tens of thousands of games in order to learn from the outcomes.
That meant the first order of business was to get to a point where you could reasonably run the game fast enough to accomplish that. Before me were two options:
- Use a synthetic environment, such as sts_lightspeed or decapitate-the-spire, and add the missing functionality (in the former case, The Silent character I wanted to focus on).
- Use ForgottenArbiter's work (spirecomm, CommunicationMod) to directly interface with the game engine, and speed up the game/gateways to allow playing fast enough.
An aside: coding agents can't estimate effort, or tell time
I asked Claude to estimate effort for the two options, finding out my first major lesson about coding agents: they have absolutely no clue how to estimate effort. In hindsight it's obvious: the training corpus contains many instances of breaking down a task into component parts, estimating said parts and aggregating. However, the estimates are for humans doing the work - there aren't enough documented instances of how long coding agents take to perform certain tasks. To make things worse, I found out Claude has no sense of time, e.g. it would work for ~20 minutes then conclude what it did took it five hours.
After several attempts to correct Claude ("no, that took twenty minutes"), I realized I'm barking up the wrong tree and simply added the following into SessionStart and UserPromptSubmit:
date '+[%Y-%m-%d %H:%M] Always start every text response with [YYYY-MM-DD HH:MM] using the time shown here. This includes after code blocks.'
This hook is somewhat wasteful (adds about 30 tokens to every turn in the conversation) but it meant Claude stopped gaslighting me about time, and it allowed me to try and extrapolate from how long things actually took.
Back to the decision
With estimates I could pretend to trust, I settled on the second option. My reasoning was that had it been simple to do, the authors of those projects would have fixed the bugs / added more characters already, since they're motivated to do so. Conversely, nobody really cared about the speed of the existing headless mode (lifted from SeedSearch) since it was designed to only play a few turns to discover seeds with some interesting properties. What I failed to consider is that this argument cuts both ways. Nobody cared about the fidelity of a seed-finding mod, either, beyond the first couple of floors. I ignored a core principle of software development: untested code is always wrong.
Not being a game developer, I envisioned a sort of model/view/controller design pattern, where the visual effects are a "view"-like layer decoupled from the business logic of the game engine and the model of the game state. And so, I set Claude on a happy pursuit of spawning the game in a headless mode and patching every time a visual effect tried to access a graphical entity and got a null pointer exception since we didn't instantiate those. After a day or two of that, the game stopped crashing and hanging, which could only mean it was running correctly.
The last order of business was speeding up the connection itself, since the agent ran in Python and the game ran in Java. Claude settled on using Py4J to pass the calls, and I pointed out where he should cache the state and batch calls to update it since there was a fixed overhead per-call of serializing and de-serializing objects.
And so, we were off to the races and could get started on actual data science work. At the time, I hadn't realized my wrong architectural decision would cost me several weeks (and not Claude-weeks; week-weeks), nor had I guessed I would end, as the Hebrew parable goes, "having to eat the stinking fish, get my lashes AND be exiled from the city". I would end up not only spending a lot longer than planned on headless mode, but also having to extend a synthetic environment anyway AND pay the cost of keeping the two aligned.
However, all that unpleasantness was in the future. Ahead of me lay my first big data science challenge: designing and training the agent. That is the topic of the next post, in which our hero discovers the problem to be slightly more complicated than giving an agent time and a reward signal.
