What Six Codex Runs Taught Me About Tool Design

The score was not the most interesting result

The String Calculator was never the object of this experiment. It was a small Java kata often used to practice TDD, giving a coding agent a concrete reason to use Seed4J CLI.

The object was the interaction among three parts: the agent interpreting the goal and choosing capabilities, a skill installed in the repository guiding how it should discover, plan and verify, and Seed4J CLI validating and materializing the selected transformations.

In From an Empty Repository to a Java Kata, One Module at a Time, I followed that interaction through one complete Codex run, from an empty repository to a Java and Maven foundation and then to the kata implementation.

I had also repeated the same task five more times, changing which model powered Codex or how much reasoning effort it used. All six runs started from the same commit and kata specification, received the same prompt and used Seed4J CLI v0.0.4 with Seed4J 2.2.0. All six produced a working String Calculator implementation, passed the build included in the project and passed the same independent tests covering the kata requirements.

I initially evaluated them with a shared rubric. It awarded up to 100 points across Seed4J effectiveness, required behavior, tests and design. The rubric was useful for comparing the runs against the same criteria, but it reduced each execution to a score.

Comparing the commands and messages recorded during the six runs did not change the subject of the experiment. It made the different paths through the same interaction visible.

One run showed the workflow in action. Six runs showed how differently agents navigated that workflow.

What I mean by a decision trajectory

I am not trying to infer the model's private reasoning. The recorded commands and messages do not expose it, and I do not need it for this comparison.

There is already a lot I can observe:

• which commands a run executes before changing the repository;
• which modules it inspects as candidates;
• which inspected modules it leaves out;
• whether it creates one complete plan or several smaller plans;
• how tool or environment feedback changes the next action;
• which capabilities it delegates to Seed4J;
• what it verifies after generation.

I think of that visible sequence as a decision trajectory.

The six runs used three models: Sol, Terra and Luna. Four used Sol at low, medium, high and xhigh reasoning effort; the other two used Terra and Luna at xhigh.

Three paths through the same CLI

The three xhigh runs provide the cleanest narrative comparison because the reasoning effort stays fixed while the selected model changes.

In the detailed evaluation, Sol xhigh scored 98/100, Luna xhigh 95/100 and Terra xhigh 91/100. All three earned the full 30 points for the required String Calculator behavior. Their remaining differences came from Seed4J usage, tests and design.

The behavior scores confirm a common functional baseline. They do not describe the different paths each run took through the interaction among agent, skill and CLI.

Sol xhigh anticipated the foundation

The Sol xhigh run inspected init, maven-java, maven-wrapper and jacoco-with-min-coverage-check, then placed all four modules into one apply-set plan.

Its visible trajectory looked roughly like this:

goal → candidate capabilities → one composition → plan → execution

This run anticipated that the project should have its own Maven launcher and an enforced coverage gate before attempting the kata implementation.

It also exposed an important division of responsibility. The run requested the modules in this order:

init
maven-java
maven-wrapper
jacoco-with-min-coverage-check

Seed4J executed them in this order:

init
maven-java
jacoco-with-min-coverage-check
maven-wrapper

The run chose the capabilities. Seed4J validated the composition and resolved how to materialize it.

Terra xhigh adapted to feedback

The Terra xhigh run began with a smaller composition: init and maven-java. It planned those modules, applied them and inspected the generated project.

Then it tried to use the global Maven command. The environment returned mvn: command not found.

The next visible message said:

The environment has Java 25, but does not have mvn.

That sentence is my English translation of the original Brazilian Portuguese message in the Terra run record.

The run then inspected maven-wrapper, generated an individual plan, applied the module and continued through ./mvnw.

Its trajectory was different:

initial composition → environment feedback → new capability → plan → execution

Calling this merely a mistake would hide the useful part. In this run, a missing environmental capability produced feedback, and that feedback changed the composition. The final project still recorded the wrapper through the same Seed4J history and commit mechanism.

Luna xhigh explored and kept the scaffold small

The Luna xhigh run inspected a broader group of candidates. In addition to init, maven-java and apply-set, it inspected java-base and spring-boot.

After that exploration, its visible message stated that the specification required incremental TDD in Java but did not require a framework or application structure. The run chose a minimal Maven project with JUnit 5 and left java-base and spring-boot out of the composition.

It later inspected maven-wrapper and planned the final set:

candidate exploration → minimal boundary → composition → plan → execution

This is not evidence that Luna generally explores more or that it is inherently more disciplined about scope. It is evidence that this particular run inspected two plausible capabilities and did not apply them.

The other runs left signals too

The remaining Sol runs reinforce the point that a score or final file tree does not describe the complete interaction.

This table does not rank the trajectories. It gives me a vocabulary for comparing them: discovery breadth, candidate rejection, planning granularity, proactive composition, reaction to feedback and delegation to the tool.

The skill constrained the protocol, not every decision

All six runs had the same Seed4J CLI skill installed in the repository.

The skill defines an operating protocol:

1. discover the CLI, runtime and catalog;
2. infer candidate modules from the task;
3. inspect the active help for those modules;
4. plan before mutation;
5. execute the validated composition;
6. verify the generated result and Git state.

That protocol explains part of the consistency across the runs. Every one of them discovered the runtime and catalog. Every one produced a plan before applying modules. Every successful module left history and a commit.

But the skill did not prescribe the exact module set for the kata. It did not say that JaCoCo was mandatory. It did not tell the run whether to inspect Spring Boot, when to add the Maven Wrapper or whether to construct one large plan.

That left a meaningful decision surface for each run:

• the run selected the capabilities it wanted;
• the skill guided how to interact with the tool safely;
• Seed4J validated dependencies and parameters, calculated execution order, changed files, updated module history and created commits.

This separation is more interesting to me than trying to put every correct decision into the skill.

The tool reduced the space of decisions that had to be right

Without a project generator, a coding run building the same foundation might need to choose Maven plugin versions, write the POM, create the wrapper, configure JaCoCo, place files correctly, preserve project history and decide how to divide infrastructure changes into commits.

With Seed4J, the request can be closer to a set of capabilities:

Java with Maven
Maven Wrapper
coverage gate

The tool then turns that explicit intent into deterministic transformations.

This does not make module selection irrelevant. A run can still omit a useful capability, add one the task does not need or choose parameters poorly. Seed4J also cannot guarantee that the kata implementation itself will be well designed.

What it can do is narrow the area in which those choices operate. Once a valid capability is selected, the run does not also need to reproduce all of the configuration details behind it from memory.

The models can take different routes through discovery and planning without every difference becoming a different manually written build configuration.

This connects to how I think about skills and loops

In When Skill Evolution Means Removing Instructions, I argued that deterministic knowledge should move into deterministic mechanisms whenever possible. A skill can become smaller when a test, hook or tool can enforce what used to depend on an instruction.

In I Had Already Built Three Agentic Loops Without Naming Them, I described autonomy becoming safer when feedback and exit conditions live in the environment instead of depending on the model remembering them.

Seed4J gives me another concrete example of both ideas.

The skill does not contain a static catalog or teach the model how to write every generated file. It sends the run to the active CLI. The plan provides feedback before mutation. The runtime validates the composition. The module history and Git commits make the result inspectable afterward.

The model still decides. But it decides inside a workflow with explicit feedback and deterministic boundaries.

What this experiment does not show

This was one kata, one prompt and one run for each model and effort configuration, all executed on the same host. Each run took place in a separate chat, and execution time was not part of the comparison. The records also differ in format and in what they omit, which limits direct comparisons of presentation and completeness.

There was no control group implementing the kata without Seed4J. I therefore cannot claim that Seed4J made the runs faster, cheaper or more correct than another approach.

Most importantly, these trajectories do not establish stable model personalities. I can say that the Terra xhigh run added the wrapper reactively. I cannot conclude from one run that Terra is a reactive model. I can say that the Sol xhigh run anticipated the coverage gate. I cannot conclude that Sol always plans infrastructure better.

The detailed experiment report keeps the full protocol, commands, scores and limitations.

A principle I want to test again

These six runs do not prove a general rule about agent tools. They make one design principle concrete enough for me to keep testing.

Different runs can explore different candidates, compose at different moments and react differently to feedback. A tool does not necessarily need to normalize all of that behavior. It can preserve room for judgment while making dependencies, ordering, parameter resolution, history and mutation more predictable.

A good agent tool may not need to eliminate model variability. It may need to constrain where that variability can cause damage.

If this experiment made you curious about the approach, consider giving Seed4J and Seed4J CLI a star 🌟 on GitHub. It helps more people discover the projects and follow their evolution.

References

• Seed4J
• Seed4J CLI
• Seed4J CLI String Calculator Kata
• Detailed experiment report
• Sol xhigh run record
• Terra xhigh run record
• Luna xhigh run record
• OpenAI model catalog