Twenty-six commits across six repos, and the day had one accidental theme running through it: things that were green and shouldn't have been. A test suite that exercised a code path incapable of reaching a certificate authority. Fixtures written to match a parser's bug. A deploy step that failed on every release and got filed as cosmetic. Forty-six confirmation messages nobody could see.
Plus the substantial piece — teaching a provisioning platform to ask a machine what it is before changing it, which has its own write-up.
1. A row in your database is not a machine
The big one. A control plane that provisions VMs over SSH could only ever answer "is there a row with this address?" — which is equally true of a machine you bootstrapped, a machine somebody rebuilt on the same IP, and a machine another control plane administers.
Three fixes stacked: a world-readable manifest the machine writes about itself, a read-only probe that runs before any pipeline and returns an outcome enum where three of six cases are refusals, and a version ledger so an upgrade becomes a diff instead of re-running fourteen steps on a host that's serving traffic.
The rule underneath all of it: unknown never resolves to proceed. Full write-up in the linked post.
2. Four fakes that certified nothing
An ACME client that could never have issued a certificate
Accepting an HTTP-01 challenge is a POST carrying an empty JSON object. The method passed [] — and its own comment said "an empty JSON object" — but json_encode([]) is [], a JSON array. Every issuance died with urn:ietf:params:acme:error:malformed.
The intent was right and the encoder silently disagreed with it. Nothing caught it because the fake client never encodes anything, so the entire SSL suite exercised a path that cannot reach a CA.
The fix isn't JSON_FORCE_OBJECT — that would turn the order request's identifiers list into an object and break ordering instead. There are three distinct payload shapes and they need three distinct encodings:
private function encodePayload(?array $payload): string
{
return match (true) {
// POST-as-GET: empty string on the wire.
$payload === null => '',
// An empty JSON *object* — json_encode([]) would give "[]".
$payload === [] => '{}',
default => json_encode($payload, JSON_THROW_ON_ERROR),
};
}
And the tests now assert all three at the byte level, by decoding the JWS payload the request actually carried — rather than trusting the argument that went in. That distinction is the whole lesson. Asserting on your input asserts on your intent.
A health check that could only ever say "down"
The stats parser matched a proxy name against the bare listener name. HAProxy reports servers under the backend section, which the config renderer names {name}_back — so the match never succeeded, the "up" set stayed empty, and every backend on every driver read as unhealthy regardless of what HAProxy actually thought. Three drivers shared the trait; three drivers shared the defect.
Found by pointing a new driver at two real machines: HAProxy was happily balancing 5/5 across both, its own stats CSV said both servers were UP, and the health check reported both DOWN.
The fixtures are why it survived. They carried proxy names HAProxy never emits. They'd been shaped to the parser's behaviour rather than to the machine, so they actively certified a health check that couldn't work.
If a fixture was written by reading your code instead of by capturing real output, it isn't a test. It's a mirror.
Four caches that were never built
Every deploy ended with a red views ... FAIL line, and it was treated as cosmetic. It wasn't. artisan optimize runs its stages in order and stops at the first failure — so views, icons, and two other caches were skipped on every single release. The health check passed. The site served. The only symptom was a line in output nobody read.
Two missing Blade components, from different causes. One was a component that simply never existed in the tree, at two call sites — replaced with the UI kit's own equivalent rather than shimming a component to match the wrong name. The other was assumed by a dependency's view: view:cache compiles every registered view path including a package's, so a component missing from a library you installed for one command fails the entire stage even though nothing in your app routes to it.
The guard against recurrence compiles every view the finder knows about, in memory via compileString() — deliberately not by running view:cache, which writes to one shared compiled directory and turns three parallel test processes into a fight. Same compiler, same error, no disk.
it('compiles every registered view', function () {
$failures = [];
foreach (allViewFiles() as $view) {
try {
Blade::compileString(file_get_contents($view));
} catch (Throwable $e) {
$failures[$view] = $e->getMessage();
}
}
expect($failures)->toBeEmpty();
});
And it was verified by removing a component and watching it go red — not trusted because it passed. A guard you've never seen fail is a guard you're guessing about.
Forty-six invisible confirmations
Seventeen components flashed a toast key and redirected. Nothing read the key. You delete a project, the page changes, and nothing tells you it worked.
The obvious fix — dispatch a browser event instead — is wrong here: a dispatched event doesn't survive redirect(..., navigate: true). The flash is the right mechanism for "this worked, now go somewhere else"; the reader was what was missing. The toast component now drains flashed keys in x-init, so both mechanisms feed one renderer and all forty-six call sites stay exactly as they are.
There was a layout <script> failing the same family of way, too: it ran during parse, before Alpine bound its listener, so the event had no listener to reach.
3. Eligibility as drivers, not as if-statements
Different product. The requirement: an event organiser wants a ticket type only certain people can buy — members of an association, holders of a @company email, people in a particular group chat.
The wrong shape is a growing if in the purchase flow. The right shape is a contract plus a config map, so adding a source is a class and an entry, never a change to the gate:
interface EligibilityVerifier
{
public function identifier(): string;
/** Drivers assert claims, never roles. */
public function claimType(): ClaimType;
public function verify(VerificationContext $context): VerificationResult;
/** Cheap enough to re-run at the check-in gate? OTP and deep links aren't. */
public function supportsRecheck(): bool;
public function recheckTimeout(): int;
}
Four drivers landed: an email OTP with a suffix allowlist, a directory lookup against a membership system, a group-membership check, and a manual document review as the fallback for everyone the automated sources can't answer for.
The line in that interface I care most about is the docblock on verify(). An implementation must distinguish a definitive negative ("the provider says: not a member") from an indeterminate one ("the provider didn't answer"). The check-in gate fails open on the second one — you do not turn away a paying attendee at the door because an upstream API had a bad minute — and it must never be handed a rejection it would then admit on. Two different negatives, and collapsing them into false is how you either strand people at a gate or let everyone through.
Modes are an enum too: verify at purchase, at check-in, or both; match any attached source or all of them.
4. A membership check that hands back as little as possible
Related product, first machine-to-machine surface: one endpoint answering "is this person a member of this organisation, and in what standing?"
Design notes worth keeping:
- Look up by exactly one identifier. Email or member number or id — never two, because two that disagree have no honest answer.
- Found is 200, unknown is 404, and both carry the same body shape. Identity and standing only. No email, no phone, no national ID crosses the boundary — whichever value was used to look the member up.
- The key selects the tenant. A per-organisation API key both authenticates the caller and chooses whose members it can see. The request host is never consulted, so a key issued for one organisation cannot reach another's data by pointing at a different domain.
- Stored as a SHA-256 digest. Deterministic, so the lookup is one indexed query, and a database dump hands out no working keys. Plaintext shown once at issue; a lost key is rotated, not recovered.
- Rate limited per key, not per IP — an event desk behind one office address shouldn't be punished for sharing it, and a stolen key shouldn't be able to buy capacity by calling from more machines.
- Every lookup is audited, hit or miss, with the looked-up value fingerprinted rather than stored — so a run of misses is visible without the audit trail quietly becoming a second copy of the directory.
5. The public toolkit
Two public repos, so I can name these.
nasrulhazim/claude 2.3.0 — two new skills (project-status, which cross-checks a planning tree against live GitHub issues and the code, on the rule that a planning document's claim is never status until the code confirms it; and deploy-app, with a recovery path per failure mode), plus a batch of production lessons folded back into the existing ones.
One of those lessons is worth flagging on its own: the bundled pest-testing skill is now kickoff-pest-testing. Laravel Boost's boost:install --skills writes its own pest-testing into the same .claude/skills directory, and it had already silently overwritten the toolkit's copy in a live project. Found only by diffing the two trees. If you maintain skills, assume name collisions in a shared directory are silent and namespace accordingly.
cleaniquecoders/kickoff — three defects that ship green, all found in a production app built from it:
-
composer testno longer runs--tiaby default. Test impact analysis needs a coverage driver to record before it can skip anything; under Xdebug a cold record takes ~90 minutes and Composer's 300s process timeout kills it partway, leaving an unusable graph that re-records from scratch forever. It's opt-in now, withdisableProcessTimeoutand pcov. -
[x-cloak]{display:none!important}added to the stub CSS.x-cloakis only an attribute Alpine removes on init — without the rule, everyx-show="false"element renders visible on first paint. Nothing errors. -
seed:prepareforwards--force. It's the production install path, runs with no TTY, anddb:seedprinted its production prompt, cancelled, and exited 0 — leaving a migrated-but-unseeded database and a caller that believed it had succeeded.
Also: PASSKEYS_USER_HANDLE_SECRET in .env.example, because deriving the WebAuthn user handle from APP_KEY means rotating the key silently invalidates every registered passkey. That's a rough Monday.
6. A derived column that never re-derived
Last one, and the smallest diff of the day at two files.
A user attribute was derived once, at SSO login, from a directory attribute — and defaulted to a fallback value whenever that attribute was missing or didn't match an expected pattern. It was never re-derived afterwards. So anyone whose first login didn't resolve cleanly kept the wrong value permanently.
Downstream, that column routed a password reset to the wrong backend directory, where the account didn't exist, and the whole operation failed fast with nothing written anywhere.
The fix is one of my favourite shapes: the request had already resolved the correct value from the authoritative source a few lines earlier, and then discarded it. Realigning the stored column before the sync runs means affected rows self-heal on next use. No data migration.
Two things I'd generalise:
- A default that fires on missing input, in a value you never recompute, is permanent drift — not a fallback. If you can't re-derive it, don't default it; record that you couldn't determine it.
- If the correct value is already in scope at the point of failure, writing it back is usually cheaper and safer than a migration.
What's next
The upgrade runner — the thing that takes the version diff and executes only the steps a node is actually missing. The outcome enum has the case; nothing emits it yet, deliberately, because an outcome nothing can execute is a promise the code doesn't keep.
