Last time the system said nothing. This time it said yes.
A couple of weeks ago I wrote about three outages that reported nothing at all: a corrupt video every tool called valid, a backup that stopped running behind a healthy-looking timer, an API account that hit zero without a word. The lesson there was to check the artefact instead of the process.
Then I spent a week finding the sequel, which is worse. These four did not stay quiet. They actively reported success. Everything downstream believed them, including me.
The report that said yesterday cost nothing
A daily briefing lands in my dashboard at 08:00 with a section headed "Yesterday". It had been printing zero revenue and zero cost for weeks. I read it as a quiet week.
The section asked for period="today", which resolves to midnight-until-now. The briefing runs at 08:00. The expensive scheduled work starts at 09:00. So every morning it summarised a window in which the system had not yet done anything, and truthfully reported nothing in it.
The numbers were never wrong. The label was. One day it claimed $0.00 for a day that had really cost $0.82, and I only caught it because I was reading the row underneath for another reason.
What made it invisible: zero is a plausible answer. A report that said "error" would have been fixed in a day. A report that says a small true-looking number can be wrong for months.
The guard that watched the wrong thing
The storefront is a static site rebuilt nightly. I added a health check for it, and I was pleased with the design: instead of trusting the exit code, it re-reads the published HTML and compares it against what the API says right now. Assert the artefact, not the process. Exactly the lesson from last time.
It compared one thing: whether a sale banner on the page matched whether a sale was actually running. That was the bug I had just fixed, so that was what I checked for.
Meanwhile the product catalogue on that same site had gone empty. Every product page was rendering "Catalogue coming soon" over a shop with 88 items. The guard ran, inspected the banner, found it consistent, and wrote status: ok.
What made it invisible: I built the guard the day after an incident, so it was shaped like that incident. A check written in the shadow of one failure tends to only see that failure.
The line that waved through the outage it was written for
So I extended the guard to count products on the published page. Reasonable line:
PRODUCTS=$(grep -c 'cover' page.html || echo 0)
if [ "$PRODUCTS" -lt 10 ]; then alarm; fi
Then I tested it against a deliberately emptied copy of the page, and it printed a shell error instead of an alarm.
On zero matches grep -c prints 0 and exits 1. The || echo 0 then appends a second zero. The variable holds "0\n0", the comparison dies with "integer expression expected", and because the script has no set -e it carries on and writes "ok".
Read that again: the guard worked for every healthy page and broke precisely when the catalogue was empty, which is the one case it existed for. Had I shipped it without the negative test, it would have sat there looking responsible forever. The fix is grep -o ... | wc -l, which always yields one integer.
What made it invisible: the failure mode and the trigger condition were the same condition. Testing the happy path could never have found it.
The fallback that kept the build green and the shop empty
Why was the catalogue empty in the first place? The site fetches it at build time, and that fetch had this shape:
try {
const r = await fetch(ENDPOINT);
if (!r.ok) return []; // keep the build green
} catch { return []; }
The static site generator evaluates each page separately, so an un-cached fetch fired once per page: about fifty parallel requests for the same URL. That tripped a rate limiter on the API. Every page got HTTP 429. Every 429 became an empty list. The build went green and published an empty shop over a working one.
The comment is the confession. "Keep the build green" was written as resilience, and it is the opposite: it converts a loud, fixable network error into a quiet, correct-looking deployment. Two changes fixed it. One shared fetch per build instead of fifty, and the catalogue now throws rather than returning empty, because the deploy only publishes after a successful build. A stale shop beats an empty shop.
What made it invisible: a single manual curl from the same machine always worked. The limiter only triggers on the burst, and the burst only happens during a build.
Watch it go red before you trust the green
There is one habit behind all four fixes, and it is cheap: after writing a check, deliberately break the thing it watches and confirm it fails. Not a thought experiment. Feed it a doctored copy of the file. Point it at an empty response. Force the error case.
It takes about two minutes and it is the only way to distinguish a working check from a check-shaped object. Every one of these four passed a green run happily. Three of them only revealed themselves when I made the underlying thing fail on purpose. The fourth was found by a human noticing a page looked wrong, which is the monitoring strategy we are all trying to replace.
Two corollaries worth stealing. First, be suspicious of a check written the day after an incident, because it will be shaped like that incident and blind to the next one. Second, treat every "keep it green" fallback as a decision to hide a class of failure, and ask out loud whether that class is one you can afford to stop seeing.
Originally published at neuragrowth.co. I run a one-person digital-products studio and write up what breaks in production.
