I rebuilt the same commit and got a different image. That is still the part I can't explain.
The first build was tagged from current source and its compiled openapi.so was a day old. The builder stage had done its job, the artifact it produced matched what the current source should compile to, byte for byte. The runtime stage's COPY --from=builder came out of the layer cache and carried the previous build's output. New source, new tag, old image, and not one error anywhere along the way.
Rebuilding fixed it. That is the worst available outcome. A reproducible failure is a thing you can chase. A build that comes out right the second time tells you the next occurrence will be just as invisible as this one was.
Docker's cache invalidation docs say this shouldn't happen the way I've described it. For most instructions the cache is keyed on the text of the instruction, and they're blunt about what that means: "when processing a RUN apt-get -y update command the files updated in the container aren't examined to determine if a cache hit exists. In that case just the command string itself is used to find a match."
Copying is supposed to be the exception. For ADD and COPY, and for RUN with a bind mount, Docker calculates the cache checksum from file metadata rather than from the instruction string, and a file's modification time is deliberately left out of that checksum. So a COPY line that hasn't changed is not on its own enough to earn a cache hit. The source had changed. The metadata should have moved with it.
That's as far as I get. Mtime being excluded is the only thread I have and I can't pull it into an explanation I'd defend. If you've hit this and can account for it properly, I'd rather read that than keep guessing.
What I did instead of understanding it was stop trusting the tag. The build now asks the image what it actually contains: for each module, take the compiled artifact out of the image, take the hash of the source file it should have come from, and compare against the compilation cache entry that source must have produced. Disagreement means the build stops before anything gets pushed.
Then, weeks later, a release went out to change exactly one module. The check printed this:
OK: <image> carries the current source (3 modules checked).
True, and useless. The three modules it checked were three other ones. The module the release existed for wasn't among them, and the output says so plainly if you read it as a count rather than as the word OK. Three. Nobody read it that way, including me.
The list is three names in a tuple, hand-maintained, against a package of 65 modules. And here is the part I keep turning over, because the reasoning next to that tuple is not stupid. It says the list is kept small on purpose, that these are the modules whose staleness would be both invisible and expensive, and that widening it to every module would cost time for little gain since one stale layer stales the whole COPY.
Read that last clause again, because the whole check rests on it. If staleness always arrives a whole layer at a time, then three witnesses are as good as sixty-five, sampling is sound, and that OK was informative after all. If staleness can ever land on one module while its neighbours stay current, three witnesses are three lottery tickets and the OK means nothing.
I don't know which is true. Neither did the check. It has never tested its own central assumption, and it prints the same word either way.
The cheap move here is the one I keep having to relearn: point the checker at something you already know is broken, and do it in both directions. Not once. Twice, because one run tells you the checker can fire and the second tells you what it's blind to. Reduced to something you can paste and run:
import hashlib
source = {'billing.py': b'rate = 0.01', 'openapi.py': b'version = 3', 'jobs.py': b'retries = 2'}
artifact = dict(source)
WITNESSES = ('openapi.py', 'jobs.py') # hand-maintained, deliberately small
def check(artifact):
bad = [m for m in WITNESSES
if hashlib.md5(artifact[m]).hexdigest() != hashlib.md5(source[m]).hexdigest()]
return f'FAIL: {bad}' if bad else f'OK ({len(WITNESSES)} modules checked)'
print('unchanged ', check(artifact))
a = dict(artifact); a['openapi.py'] = b'version = 2' # stale, on the list
print('stale, on the list ', check(a))
b = dict(artifact); b['billing.py'] = b'rate = 0.10' # stale, off the list
print('stale, off the list ', check(b))
unchanged OK (2 modules checked)
stale, on the list FAIL: ['openapi.py']
stale, off the list OK (2 modules checked)
The middle line is the one that feels like testing and the last line is the one that's worth anything. An artifact shipping a billing rate ten times too high, and the check calls it OK and tells you how thorough it was being.
The real version of that second control is harder and I haven't run it: deliberately stale one module inside a real image while leaving its neighbours current, and see whether the check still goes red. If it turns out you cannot construct that image at all, that isn't a failed experiment. That's the proof the sampling argument was right, and the check has earned the OK it's been printing.
What actually happened is smaller than that. Somebody added a fourth name to the tuple, with a comment saying it went in "after a build whose only change lived here passed this check without it being looked at". Which fixes that release and not the shape of the problem, since the next module to matter is by definition the one nobody has thought about yet. A hand-maintained list of what to verify decays exactly like a hand-maintained list of anything else, quietly and in the direction of the last thing that went wrong.
The version I'd actually want compares the set of modules that changed in this release against the set the check looked at, and refuses to print OK when the intersection is empty. That's maybe fifteen lines. I haven't written it, and writing it up like this is mostly me admitting that out loud.
If you run a build check like this, I'd like to know how you keep its list honest, or whether you gave up on lists and hash everything.
The tool I work on is HumanPen; it rewrites prose inside .docx files, which is why there's a module in that tuple deciding what a job costs.
