Here's the claim that will get me disagreed with in the comments: your CI pipeline is not a staging environment, and treating a green checkmark as a deployment gate is a worse practice than having no automated gate at all. Teams with no CI at least know they're flying blind. Teams with CI often believe they've verified something they haven't.
The disagreement isn't about whether CI is useful. It obviously is. The disagreement is about what a passing build actually certifies. Most engineers I've argued with insist that if the build passes, the artifact is "known good," and any failure after that point is an infrastructure problem, not a testing problem. I think that's backwards. The build passing certifies that your code compiles and your unit tests run inside the specific, narrow, often-privileged execution context that CI constructs for itself — a context that frequently has almost nothing in common with the container or VM that will actually serve traffic.
The worked example
A team I worked with ran a Python service on ECS Fargate. Their CI ran on GitHub Actions using ubuntu-latest runners with a Docker-in-Docker setup that built the image, ran pytest, and pushed to ECR on green. Deployment had been stable for fourteen months — 340 successful deploys, zero rollbacks attributable to the app itself.
Deploy 341 took down checkout for 22 minutes. Here's what happened, with numbers.
The base image was python:3.11-slim, pinned. Nothing changed there. What changed was a transitive dependency, cryptography, which shipped a new wheel that required a newer version of libssl than what was present in the runtime base layer used in production — but not in the CI build layer, because CI's Docker-in-Docker host had a newer kernel and a cached apt layer with a more recent libssl-dev already present from an unrelated job that ran three hours earlier on the same runner pool.
The build succeeded in 41 seconds. Tests passed in 3.2 seconds — 118 tests, all green, because the test suite never imported the code path that triggered the native extension load for the affected TLS routine. In production, that code path fired on 100% of requests, because it was in the auth middleware. Time to first error after the new tasks registered healthy in the ALB: 90 seconds. Time to full outage as old tasks drained per the deployment's rolling policy: 4 minutes. Time to identify root cause: 19 minutes, most of it spent staring at a healthy CI run insisting nothing was wrong.
The fix, once found, was three lines — pin libssl explicitly in the production base image and add a smoke test that actually exercises the auth path in a container built from the exact production base, not the CI convenience image. The postmortem cost was not three lines. It was 22 minutes of downtime, a customer-facing status page incident, and about six hours of two engineers' time to trace a problem that had nothing to do with the code that was "changed" in the diff under review.
The failure mode, generalized
This is not a story about libssl. It's a story about a category of failure that has a name once you've been burned by it enough times: runtime parity drift. CI environments and production environments start as siblings and, without active maintenance, become strangers. The drift accumulates silently along a few predictable axes:
- Base image lineage — CI builds and runs the app; production runs a different derived image with different layers cached, different security patches applied, different init systems.
- Privilege and capability sets — CI runners are often more permissive (network egress, filesystem writes, DNS resolution) than production containers running under restrictive security contexts.
-
Dependency resolution timing — a
requirements.txtwithout hashes, or a lockfile that's "advisory" rather than enforced, means the packages installed at 2pm during CI are not guaranteed to be the packages installed at 2:03pm during the actual deploy, especially with floating minor versions. - Untested code paths — the classic one. Test coverage numbers measure lines executed, not conditions under which they're executed. Auth, TLS negotiation, feature flags, and anything gated by environment variables are chronically under-exercised in CI because CI doesn't have real secrets, real flags, or real load.
None of these show up as a "bug" in code review. They show up as an outage that looks, to the on-call engineer, like the deployment tooling itself is broken — because from where they're standing, something that was proven to work five minutes ago has just died for no visible reason.
Why "add more tests" isn't the answer
The instinctive response is to write more integration tests, which helps, but doesn't solve the structural issue: CI is optimized for speed and isolation, and production is optimized for realism and scale. Those goals are in tension. You cannot make CI a perfect mirror of production without making it as slow and expensive as production, at which point you've just built a second production environment that also needs monitoring.
What actually works is treating runtime parity as an explicit, versioned contract between the build stage and the deploy stage — not an assumption. That means the artifact that gets tested is byte-for-byte the artifact that gets deployed, dependency resolution is pinned and hash-verified rather than "mostly reproducible," and there's a mandatory smoke-test stage that runs after the image is built, in an environment provisioned identically to production, before traffic is shifted. It's slower. It has caught, in every team I've implemented it with, at least one class of failure that unit tests structurally cannot catch.
I wrote up the specific checklist, dependency-pinning patterns, and the smoke-test stage architecture that came out of incidents like this one — the actual contract, not a slogan — here: https://dasdorf.gumroad.com/l/cpzgn
If your last outage postmortem contained the sentence "but it passed CI," this is written for the team that had to write that sentence and never wanted to write it again.
