goose swarm: pytest | head -80 exits 0 when nothing ran, and pipefail only trades the lie

Key takeaways

• pytest exits 4 for a missing file, 5 for zero collected and 1 for a real failure. Piped through head -80, all three exit 0 — measured in bash 3.2.57 and zsh 5.9, both of which ship with pipefail off.
• goose's shell tool only tells the model 'Command exited with code N' when the code is non-zero. A pipe therefore deletes the harness's only failure signal, and the model is left reading a calm 'collected 0 items' on stdout while the honest ERROR line sits on stderr.
• set -o pipefail restores 4/5/1 — and then reports exit 1 on a run where all 200 tests passed, because head -20 broke the pipe. A grep -q over a 2,000,000-line file returns 141 under pipefail while succeeding.
• Neither setting is honest, because the pipeline is the wrong instrument. Redirect to a file, read the code, then bound the text: the exit code survives and the output is still short.
• The worker's | head -80 bought nothing. The output was six lines, and goose already caps tool output at 2,000 lines / 50 KB and shows both ends when it does cut. The pipe cost the exit code for no saving at all.
• The fix that shipped is eleven lines of prompt text, not a guardrail. I am saying that plainly because a probabilistic fix to a deterministic bug is worth exactly what it sounds like.

Yesterday a worker in my swarm was told to implement one module and then run pytest to check its work. It pointed pytest at a path that did not exist. It read the output, decided the check had passed, ran it a second time to be sure, decided the same thing again, and finished the task reporting success.

Nothing ran, either time.

Nothing in that sequence was a model failure in the interesting sense. The worker did the reasonable thing with the evidence it had. The evidence was wrong, and it was wrong because of a single character in the command it ran: a pipe.

This class has now caught this repo at least five times in five different costumes, and this is the first time I have seen it reach a dispatched worker rather than my own shell. So this week I stopped treating it as a series of embarrassing one-offs and measured it properly — on pytest and on cargo, in bash and in zsh, with and without pipefail. The result is worse than "remember to use pipefail," which is the advice everybody including me has been repeating.

What the worker actually ran

The swarm is goose-local-edition, our fork of block/goose that runs a fleet of local 27B-class models as a plan/execute/judge loop rather than one model in a chat window. If you want the architecture, I wrote it up in Inside goose-swarm — the short version is that a planner decomposes a spec into owned files, workers build them on separate nodes, and a judge grades the result. A "worker" here is one local model with a shell, an editor, and a spec for exactly one file.

The command was:

python3 -m pytest <path> | head -80

The path is the part that was wrong, and it is not the interesting part. Everything else about that line is perfectly sensible-looking. The model was managing its own context, pytest can be verbose, head -80 bounds it. Every human I know types this.

Here is what came back on stdout:

============================= test session starts ==============================
platform darwin -- Python 3.13.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/pipe-rig
collected 0 items

============================ no tests ran in 0.00s =============================

Read that as a language model with a spec that says "prove your file collects and runs." There is no red. There is no FAILED. There is no traceback. There is a session header, a count, and a line saying the run finished in four hundredths of a second. collected 0 items is the only clue, and it is phrased as a statistic, not as an error.

And the exit code — the one unambiguous, machine-readable signal that would have settled it — never reached the model at all.

The measurement

Everything below was run on the Mac Studio this fleet lives on: Apple M3 Ultra, 96 GB, macOS 26.6.1, GNU bash 3.2.57(1)-release, zsh 5.9, Python 3.13.13 with pytest 9.1.1, cargo and rustc 1.92.0.

The rig is three files. A test module with one passing and one failing test, a module with no test functions in it at all, and a path that does not exist.

First, pytest on its own, with the pipeline removed:

Those match pytest's documented exit codes, read 18 August 2026: 4 is a command-line usage error, 5 is "no tests were collected", 1 is "tests ran and some failed". Three distinct, correct, unambiguous failure signals.

Now the same four commands with | head -80 appended:

Every one. tail behaves identically to head, so the reflex of "use tail instead, the summary is at the bottom" fixes the readability and not the status.

The reason is the POSIX rule that a pipeline's exit status is the exit status of its last command. head succeeded. The shell reports what head did. What pytest did is still available for exactly one statement, in bash's PIPESTATUS array:

$ python3 -m pytest tests/test_nonexistent.py 2>/dev/null | head -80 > /dev/null
$ echo "pipeline=$?  PIPESTATUS=(${PIPESTATUS[0]} ${PIPESTATUS[1]})"
pipeline=0  PIPESTATUS=(4 0)

test_nonexistent.py   pipeline=0   PIPESTATUS=(4 0)
test_empty.py         pipeline=0   PIPESTATUS=(5 0)
test_real.py          pipeline=0   PIPESTATUS=(1 0)

The 4, the 5 and the 1 all exist. They are produced, they are recorded, and they are discarded one statement later because nobody asked. And no agent I have ever seen asks — the models write cmd | head, read the text, and move on.

This is not a bash quirk you can escape by picking a different shell. zsh, the default on this machine, does exactly the same thing on all three cases, and ships with pipefail off just like bash does:

Why the harness could not save it

The part that turns this from a shell trivia question into an agent bug is what the harness does with the number.

goose's shell tool runs your command as bash -c "<command>" (it picks bash if bash is on PATH, sh otherwise, and GOOSE_SHELL overrides both). When the command comes back, it renders the result for the model. The relevant branch, in crates/goose/src/agents/platform_extensions/developer/shell.rs, is this:

if is_error {
    if let Some(code) = execution.exit_code.filter(|c| *c != 0) {
        rendered.push_str(&format!("\n\nCommand exited with code {code}"));
    }
    // … returns CallToolResult::error(…)
}

A non-zero exit gets two things: a line of prose telling the model the code, and a tool result flagged as an error, which is a different-shaped object in the conversation than a success. That is a strong signal and models respond to it.

A zero exit gets neither. No exit-code line is emitted at all, because there is nothing interesting to say about a 0. The tool result is a plain success carrying the command's stdout.

So the pipe does not merely obscure the failure. It converts the harness's loudest available signal into its quietest one. The worker was not ignoring a warning; there was no warning to ignore.

And then there is the stream split, which I only noticed while building the rig. That ERROR: file or directory not found line pytest prints — I had assumed it was part of the output the model saw as one blob. It is not. It goes to stderr:

--- stdout only ---
============================= test session starts ==============================
platform darwin -- Python 3.13.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/pipe-rig
collected 0 items

============================ no tests ran in 0.00s =============================

--- stderr only ---
ERROR: file or directory not found: tests/test_nonexistent.py

cmd | head -80 pipes stdout only. stderr goes straight through untouched, so that line does survive into the tool result — but it lands in a separate stderr section, one line long, adjacent to six lines of calm, structured, green-shaped stdout that says the session started and finished cleanly. Given a spec that says "prove it collects", stdout is the artefact and stderr is noise. The model weighed them the way the formatting invited it to.

The worst part is that the pipe bought nothing. goose already bounds tool output on its own: OUTPUT_LIMIT_LINES = 2000, OUTPUT_LIMIT_BYTES = 50_000, and when a command exceeds either, it saves the full output to a file and shows the model both ends — 25 lines of head, 25 of tail, with an elision marker between them. The output it was guarding against was six lines long, so nothing would have been truncated anyway; and on a genuinely long run the harness would have shown the tail, which is where pytest puts its verdict and exactly where head cuts. The worker paid the exit code to protect a context budget that was never under threat.

"Just turn on pipefail" — measured, and it does not hold

The standard answer is set -o pipefail, which makes a pipeline return the rightmost non-zero status instead of the last one. It does exactly what it says:

set -o pipefail; pytest tests/test_nonexistent.py | head -80   → exit 4
set -o pipefail; pytest tests/test_empty.py       | head -80   → exit 5
set -o pipefail; pytest tests/test_real.py        | head -80   → exit 1

Three for three. If the story ended here I would have shipped set -o pipefail into the worker preamble and written a much shorter post.

Here is the run that stopped me. Two hundred tests, every one of them passing, -v so pytest prints a line each — 208 lines of stdout. Piped through head -20, with pipefail on:

set -o pipefail; pytest tests/test_many.py -v | head -20   → exit 1
                                                  PIPESTATUS=(1 0)

Exit 1. On a run where nothing failed. head took its twenty lines and closed the pipe; pytest was still writing, and exited 1. stderr was empty — no traceback, no explanation, nothing for the model to reason about. Just a status that says the tests failed, on a suite that is completely green.

It is not flaky and it is not a race. I walked the boundary:

Cut the output anywhere before the end and you get a 1. Let it through and you get a 0. Five consecutive runs of the head -20 case returned 1 every time. The rule is simply "truncated means failed", which is not a rule anyone wants.

The classic form of this is even blunter. A search over a large file, under pipefail, that succeeds:

$ python3 -c "import sys
for i in range(2_000_000): sys.stdout.write('line %d needle\\n' % i)" > /tmp/big.txt
$ set -o pipefail
$ cat /tmp/big.txt | grep -q needle    # 2,000,000 lines, every one of them matching
$ echo $?
141

141 is 128 + 13, SIGPIPE. grep -q found the needle immediately and exited, as designed. cat was still pushing two million lines into a closed pipe and got killed for it. Without pipefail the same command returns 0. The successful search reports failure precisely because it was efficient.

That one is not hypothetical for us either. Our fork's boundary check used grep -q under pipefail against a strings dump of the engine binary. It found its marker, strings was killed mid-dump for writing into the closed pipe, and the repo's own note records that the gate "would have refused every boundary forever" — permanently stuck shut because the thing it was checking was too big to read to the end.

The table that matters

Put both settings against both outcomes and the shape of the problem is obvious. This is the cargo pipeline out of our own findings file, minus the package flag, run against a two-line Rust program that either compiles or does not:

Neither column is honest. pipefail does not fix the pipeline — it changes which half of reality it lies about.

cargo build 2>&1 | grep -E '^error' -A 6 | head -20

With pipefail off, PIPESTATUS on the failing build reads (101 0 0): cargo's 101 is right there and thrown away. With pipefail on, the successful build returns 1 because grep found no errors to match — grep's "no match" status is 1, and pipefail has no way to know that "no match" was the good news.

Neither column is honest. pipefail does not fix this pipeline; it moves which half of reality it lies about. And a false red is not obviously the safer failure for an agent — a false green finishes a task that was not done, but a false red sends a worker to repair code that was never broken, which in a swarm means a repair wave, a judge round, and a few hundred node-seconds of fleet time spent fixing nothing.

The pipeline is the wrong instrument. That is the finding. A shell pipeline is a text transformer, and the moment you route a build or a test through one you have chosen to receive the transformer's opinion instead of the compiler's.

Five costumes, one bug

What convinced me to write this up rather than patch and move on is that I have now been caught by this in five different disguises inside one repo, and I did not recognise any of the later ones as the same thing as the first.

It reported a broken build as clean, twice. My own build check was cargo build 2>&1 | grep -E '^error' -A 6 | head -20; echo "BUILD_DONE". The build had failed with two real errors — an E0433 for a crate that was not a dependency and an E0308 for a function whose signature I had assumed. BUILD_DONE printed anyway, because echo runs unconditionally and reports on itself. I told myself the build was clean and then told myself again on the next tick.

It reported a successful start as a failure. ./loop.sh start | tail -6 returned 1 and I read it as "the launcher failed". It had not failed — the process was up with a pid. I ran start a second time, which parked the working tree twice; harmless only because that command copies rather than moves.

It flipped a release gate. A green-gate check read RED through a pipe and GREEN when run directly, in the same minute, on the same tree. The direct exit was the authority, and only because someone thought to try it.

It jammed a boundary check shut, permanently. The grep -q over a strings dump, from the section above — under pipefail, a check that had already found what it was looking for was killed for being efficient about it.

It let a worker finish a task it had not done. The one this post opened with.

Different shells, different tools, different directions of error. One mechanism. The line I wrote in the findings file after the second one is the one I would put on a wall: a status marker you print yourself is not a status. echo OK after a pipeline, a boolean computed from a needle you chose, a commit that "succeeded" — each is a claim by the harness about the harness, not evidence about the work. Every single time in that repo that a self-authored flag has disagreed with the underlying tool's own output, the flag was wrong.

This is the same discipline that made the benchmark work at all. The reason our grader runs the software instead of reading it is that a model's report of its own success is worth nothing, and a regex over source code is worth only slightly more. It took me embarrassingly long to notice that a shell pipeline is the same category of lie, one level down.

What shipped, and what it is worth

The fix is commit 59f39a6e8 — one file, eleven lines added, four removed, in the two places where the swarm dispatcher builds a worker's preamble. Test authors get this:

Then run `python3 -m pytest` ONCE to prove your file COLLECTS and RUNS.
READ that output: `collected 0 items`, `no tests ran` or `file or directory
not found` means NOTHING ran — that is a FAILURE even if the command exits 0,
fix the path/collection before finishing. NEVER pipe the test command through
`head`/`tail` — the pipe replaces the real exit code with the pipe's and a
broken run reads as a pass.

and implementers get the shorter version of the same two sentences.

I want to be straight about what that is. It is a prompt, not a guardrail. It tells a 27B model not to do something, which means it will mostly not do it and sometimes will, and the failure mode when it does is silent and identical to the one I just described. A probabilistic fix to a deterministic bug is worth what that sounds like.

The real fix is harness-side and I have not built it yet. A shell tool that knows the exit code of every stage — bash hands it over in PIPESTATUS for free — should say so when the last stage passes and an earlier one did not. Something as small as appending Pipeline stage 1 exited 4 (pipeline reported 0) to the tool result would have made this incident impossible, without changing a single command any model writes, and without the false-red that pipefail introduces. That is the version worth building, because it makes the harness honest rather than asking the model to be careful. It is now the top item on the shell-tool list.

I will also say what I have not measured, because it matters for how far you should carry this. I do not have a rate. I know this happened to build-meridian-client twice in one run because I was reading that run's transcript; I do not know how often across the fleet, and the honest reason is that the failures are invisible by construction — they exit 0, so they cannot appear in any count of failing shell calls. A waste audit across 41 of our runs found that 8.1% of all shell calls fail, roughly 14,000 node-seconds of the fleet's time. If that audit counts a failure the way the harness does — a non-zero exit — then every piped failure is excluded from that 8.1% by construction, and the number I actually want is the one the instrument cannot produce.

The pattern I use now

Not pipefail. Not PIPESTATUS — it works, but it is one statement away from being lost, and no model is going to write it unprompted.

Redirect, read the code, then bound the text:

python3 -m pytest tests/test_nonexistent.py > /tmp/pytest.log 2>&1; code=$?
echo "exit=$code"
tail -5 /tmp/pytest.log

Three things happen. The exit code is the command's own, because there is no pipeline. Nothing is ever writing into a closed pipe, so there is no broken-pipe status to misread. And the output is still bounded — you read the tail afterwards, from a file, which is exactly as short as | tail -5 and costs nothing.

On the missing-file case:

exit=4
platform darwin -- Python 3.13.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/pipe-rig
collected 0 items

============================ no tests ran in 0.00s =============================

Same reassuring text — and a 4 sitting above it that no amount of reassuring text can talk you out of. On the 200-test green run the same form captures all 208 lines to the file and reports the verdict, which head -20 had cut off:

exit=0
============================= 200 passed in 0.04s ==============================

If you are writing prompts for agents that run tests, that three-line shape is the thing to teach, and the reason to teach it is not style. It is that | head and | tail are the two most natural things in the world for a model to append to a verbose command, they look like good context hygiene, and they quietly delete the single most reliable signal your harness has.

Why this is the change worth writing about

Two hundred and seventy-six commits landed on this fork since the last time I wrote about it, and 187 of them are benchmark work — the scoring, the controls, the rolling curve, the product-tier grading that stopped a build with a blank front page from scoring well. That is where most of the hours went, and almost none of it transfers off this repo.

This one does. It is eleven lines of prompt text and it has nothing to do with local models, quantisation, node counts or any of the levers I spent last month tuning. It applies to any harness that hands a model a shell and then decides, from what comes back, whether the command worked — which is all of them. I have only traced the plumbing in our own, so I will not tell you how your framework renders an exit code; I will tell you that the shell hands it the same 0 it handed us. If your agent has ever confidently told you the tests pass, check whether it piped them.