The deploy script that always said it worked

TL;DR

• I stood a Laravel control plane up on a bare Ubuntu host today. The provisioning script was the easy part. The deploy script was the one that had been lying to me.
• git pull had been failing on every deploy since the first one — dubious ownership, because the script chowns the tree to www-data on the way out and runs as root on the way in. Nothing checked git's exit status, so the deploy went on to install, migrate and health-check code it had never updated. And it passed, because the old release was fine.
• php artisan operations --force had never once run. --force is a migrate flag; the deploy operations command takes --isolated. It exited non-zero every single deploy and nobody looked.
• The route cache was never cleared — only config and views. When / changed from rendering a view to redirecting, production started 500ing on every page, and the health check stayed green, because /up was in that same stale cache and still answered 200.
• Under set -o pipefail, cmd | grep -q exits 141, not 0. grep -q leaves on the first match and the producer takes SIGPIPE. Read into a variable, then match with a here-string.

Today was the day the control plane had to run somewhere real: a bare Ubuntu host, provisioned from nothing, deployed to over SSH. I expected to spend it on the provisioning script — nginx, PHP-FPM, Redis, supervisor, the usual apt-flavoured tedium.

Instead I spent most of it on a deploy script that had been reporting success for weeks without doing its job. Three separate faults, all with the same shape: something failed, nobody read the exit status, and the next step happily proceeded on stale state that looked healthy.

That's the whole post. Everything below is the four places it showed up.

1. A pull that failed silently, for weeks

The deploy script ends by chowning the release tree to the web user. It starts by running as root. Git, quite reasonably, refuses to touch a repository owned by somebody else:

fatal: detected dubious ownership in repository at '/var/www/app'

Which would be fine — obvious, even — except the script looked like this:

php artisan down --retry=60

echo "Pulling codes"
git checkout "$BRANCH"
git pull "$REMOTE" "$BRANCH"

echo "Install dependencies"
composer install --no-dev --optimize-autoloader

No set -e on that stretch, no status check. Both git commands printed their refusal to stdout, exited non-zero, and the script moved on. Composer installed. Migrations ran. The health check hit the site and got a 200 — of course it did, the previous release was serving and it was perfectly healthy.

Every deploy after the first one deployed nothing, and every one of them said it worked.

The fix is two things, and the second matters more than the first:

# Deploys run as root against a tree owned by www-data. Declare it safe once,
# so a deploy cannot silently skip its own checkout.
git config --global --get-all safe.directory 2>/dev/null | grep -qx "$PROJECT_PATH" \
    || git config --global --add safe.directory "$PROJECT_PATH"

if ! git checkout "$BRANCH"; then
    echo "ERROR: could not check out $BRANCH -- nothing was deployed."
    php artisan up
    exit 1
fi

if ! git pull "$REMOTE" "$BRANCH"; then
    echo "ERROR: could not pull $BRANCH from $REMOTE -- nothing was deployed."
    php artisan up
    exit 1
fi

echo "Now at: $(git rev-parse --short HEAD)"

Note the php artisan up before each exit 1. If you're going to bail mid-deploy, decide deliberately whether the site comes back. Here nothing has changed yet, so it should. Later in the script, after migrations have run, the answer flips — more on that in a second.

And Now at: <sha>. One line, and the failure mode becomes visible from the deploy log alone: if the SHA doesn't move, the deploy didn't.

The general rule: any command whose failure would be survivable is a command whose failure you will not notice. Those are precisely the ones that need an explicit check, because the ones that blow up loudly take care of themselves.

2. A command that had never run, because of one wrong flag

Same script, further down:

php artisan migrate --force
php artisan operations --force

--force is a migrate flag. It means yes, in production, I'm sure. The deploy-operations command (I use dragon-code/laravel-deploy-operations) doesn't have it — its equivalent guard is --isolated, plus --no-interaction for the prompt. So every deploy, that line printed:

The "--force" option does not exist.

…exited 1, and the script carried on to restart Horizon. Post-deploy operations had never run. Not once.

if ! php artisan operations --isolated --no-interaction; then
    echo "ERROR: deploy operations failed."
    echo "The application is still in maintenance mode at $PROJECT_PATH."
    echo "Fix the operation, then re-run this script or 'php artisan up'."
    exit 1
fi

This is the flipped case from earlier: here I don't call php artisan up. Migrations have already run, so the release is half-applied, and a half-applied release must not go live. Maintenance mode stays on and the message says so explicitly, including the way out. A 503 that tells you why is a much better outcome than a live site running new schema against code that never finished deploying.

Which sets up the nastiest one.

3. An operation that succeeded and reported failure

The migration to give every legacy account its own tenant ran as a deploy operation. I put a progress line in it, the way you would in any command:

public function __invoke(): void
{
    $action = app(CreatePersonalOrganizationAction::class);

    User::query()
        ->whereDoesntHave('ownedOrganizations')
        ->cursor()
        ->each(function (User $user) use ($action): void {
            $action->execute($user);

            $this->line("  organisation created for {$user->email}"); // 💥
        });
}

An Operation is not a Command. It extends nothing console-shaped. No line(), no info(), no $this->command. So that call threw — after $action->execute($user) had already committed.

Look at the ordering, because it's the worst one available:

1. The work lands. The organisation is created, in the database, for real.
2. The operation throws on the reporting line.
3. The command exits non-zero.
4. bin/deploy does exactly what I just taught it to do and stops, leaving maintenance mode on.
5. Production serves 503 over a change that had, in fact, succeeded.

And it hides on the re-run. By then every affected account already has an organisation, so the query returns nothing, the closure never executes, and the throwing line is never reached. The operation "passes". You'd have to reproduce it against a fresh copy of production data to see it again.

Two things I'd take from that:

Side effects before reporting is a footgun. If the observable outcome differs depending on whether the logging worked, the failure is uninterpretable from outside. Do the work, or don't — but don't let a cosmetic line decide whether the caller thinks it happened.

Idempotent-by-query means self-concealing. Anything that loops over "rows not yet fixed" erases its own reproduction case as it goes. That's a feature for reliability and a trap for debugging, and it's worth knowing which one you're relying on at any moment.

The fix was one deleted line and a comment explaining why it must stay deleted.

4. A health check that shared the stale state it was meant to catch

This one is my favourite, because every individual piece is correct.

The deploy cleared caches:

php artisan config:clear
php artisan view:clear

Routes: not cleared. And the host had had route:cache run on it once, at some point, by some hand. So it was serving the route table exactly as it was at that moment, forever.

Then / changed from rendering a view to redirecting. Deploy, and production 500s on every request:

View [welcome] not found

— against a release whose routes/ file says nothing of the sort. That alone will cost you twenty minutes, because you're staring at code that is plainly right.

The part that made it silent, though, is the health check:

HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "${APP_URL}/up")

if [ "$HTTP_STATUS" = "200" ]; then
    echo "Health check passed"
fi

/up is a framework route. It lives in the same stale route cache. It answered 200 throughout, so the deploy reported healthy while every real page in the application was throwing.

A health check that reads from the same cache as the thing it's checking isn't a health check. It's a second copy of the bug agreeing with the first.

The repaired shape:

echo "Clear config, route and view caches"
php artisan config:clear
php artisan route:clear
php artisan view:clear

# ... migrate, operations ...

# Rebuild after migrations, before restarting processes, so the caches always
# describe the release that is about to serve.
php artisan optimize

# ...

# /up is not evidence the release works — probe a real page too, following
# redirects, since / is one.
HTTP_STATUS=$(curl -s  -o /dev/null -w "%{http_code}" "${APP_URL}/up")
PAGE_STATUS=$(curl -sL -o /dev/null -w "%{http_code}" "${APP_URL}/")

echo "Health check: /up=$HTTP_STATUS /=$PAGE_STATUS"

if [ "$HTTP_STATUS" = "200" ] && [ "$PAGE_STATUS" = "200" ]; then
    echo "Health check passed"
else
    echo "WARNING: Health check failed (/up=$HTTP_STATUS /=$PAGE_STATUS)"
    # ... roll back to $PREVIOUS_COMMIT
fi

curl -L on the second probe, because / redirects — without it you assert on a 302 and learn nothing about whether the destination renders.

The generalisation: probe at the layer you care about. /up proves PHP is alive and the framework booted. It says nothing about whether your application's pages render. Those are different claims and they fail independently, so check both — and print both numbers, so the log tells you which one broke.

Bonus: grep -q and pipefail

While hardening the provisioning script I hit a genuinely confusing one. This aborts under set -euo pipefail:

if apt-cache policy "php${PHP_VERSION}-opcache" | grep -q 'Candidate:'; then

grep -q exits the moment it finds a match. apt-cache is still writing, gets SIGPIPE, dies with 141 — and pipefail makes the pipeline's status the last non-zero one. So a successful match reports failure. Assign first, match against a here-string:

OPCACHE_POLICY="$(apt-cache policy "php${PHP_VERSION}-opcache" 2>/dev/null || true)"

if grep -q 'Candidate:' <<<"$OPCACHE_POLICY"; then
    PHP_PKGS="$PHP_PKGS php${PHP_VERSION}-opcache"
fi

Why the conditional at all: opcache stopped being a separate package in PHP 8.4. It's compiled into the interpreter now. Asking apt for php8.4-opcache doesn't warn, it aborts the whole install — so on a modern PHP you have to not request it, while on 8.3 and below you must. Hence: ask apt whether the package exists, then assert on the outcome instead of the package:

PHP_MODULES="$(php"${PHP_VERSION}" -m 2>/dev/null || true)"

if ! grep -qi 'zend opcache' <<<"$PHP_MODULES"; then
    echo "ERROR: opcache is not loaded for PHP ${PHP_VERSION}"
    exit 1
fi

A missing accelerator is a silent slowdown, never an error. So make it one.

Same category, different tool: install the PostgreSQL client from PGDG, not from the distro. Ubuntu noble ships client 16, and pg_dump is only backward compatible — point it at a newer server and it refuses with aborting because of server version mismatch. Managed clusters upgrade on their own schedule; your backup script finds out at 3am. Pin a client at least as new as the server and print pg_dump --version during provisioning so the log carries the evidence.

Testing the untestable bit

None of the above is unit-testable in any satisfying way — it's a shell script talking to a real host. What is testable is the thing the shell script deploys, and after today I test one specific claim: that the health-check target actually renders.

it('serves the root page the health check probes', function () {
    $this->get('/')
        ->assertRedirect()          // / redirects; the check follows it
        ->assertSessionHasNoErrors();
});

it('answers the framework health route', function () {
    $this->get('/up')->assertOk();
});

Trivial tests. They would have caught the stale-route incident in CI instead of in production — not because they're clever, but because they assert the same two things the deploy asserts, from a place where nothing is cached.

That's the pattern worth keeping: whatever your deploy uses as its definition of "healthy", assert the same thing in your test suite. If the two ever disagree, one of them is reading a stale copy — and it's the one running in production.

Takeaway

Every fault today was an unchecked exit status wearing a different costume. The lesson isn't "add set -e" — I had set -e where it mattered and it still slipped through the pipelines and the ifs.

It's narrower than that:

When a step fails and the next step can still succeed on yesterday's state, you have built a machine that reports success. Check the status, print the SHA, probe a real page — and make the deploy prove it changed something.

Next up: getting the provisioning script to the point where a fresh host is one command away, and having the deploy write its resulting SHA somewhere the application itself can display. If the release is going to claim it deployed, it should be able to show me what.