`gh attestation verify` said yes to an image we never released

The four questions, and the one nobody asks

Pulling a container image somebody else built asks four separate questions. Most
people answer three of them and skip the fourth:

The fourth is the one that has no substitute, and it is the one that is almost
always skipped. What follows is all four, run for real against our own v1.0.0.

1. An image wears more than one sha256

This is where the confusion starts. Get an anonymous pull token — a public package
needs no account:

TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:soit-ai/soit/server:pull&service=ghcr.io" \
  | sed -n 's/.*"token":"\([^"]*\)".*/\1/p')

Ask what the v1.0.0 tag resolves to:

curl -sI -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  https://ghcr.io/v2/soit-ai/soit/server/manifests/v1.0.0

docker-content-digest: sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929
Content-Type: application/vnd.oci.image.index.v1+json
Content-Length: 856

856 bytes: this is an image index, not the image. Inside it, two entries:

manifests[0]: sha256:84e7f539...d32f  2589 bytes  linux/amd64
manifests[1]: sha256:a6bd430d...e1be   565 bytes  unknown/unknown
              vnd.docker.reference.type:   attestation-manifest
              vnd.docker.reference.digest: sha256:84e7f539...d32f

That unknown/unknown entry is not corruption — it is BuildKit's own attestation
for the amd64 manifest, and it is a different mechanism from the Sigstore
signatures we are about to read. First thing people conflate.

One level down, the amd64 manifest says:

config: sha256:021814982fcc214a02a2b322cbc3729b3ce43bd6e191aa0fa39da540a3e3cd1f  8769 bytes
layers: 12, 684216450 bytes compressed (about 652 MiB)

Three hashes so far, three different meanings:

The signature covers the first one. A fourth hash shows up in section 8, and it
is none of these.

2. Pulling the signatures without Docker and without logging in

The OCI spec defines a referrers API for "things attached to this thing":

curl -s -H "Authorization: Bearer $TOKEN" \
  https://ghcr.io/v2/soit-ai/soit/server/referrers/sha256:96b80ae1...5929

{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}

Same error for the platform manifest digest. Do not conclude "unsigned" from
this. The spec has a fallback: replace the colon with a dash and treat the digest
as a tag. List the tags and there it is:

{"name":"soit-ai/soit/server","tags":[
  "v1.0.0",
  "sha256-3a5b3b1a2d14e0646298826602124ba22e63f8c078f653e506a0a042bfd18246",
  "sha256-236201a5a3ee861ffdb5fdd7ec134454619eb1ab0e777439c4a22a65002f874f",
  "sha256-96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"]}

The last one is ours. Remember the first two — section 9 is entirely about them.
Fetching sha256-96b80ae1… returns a small index holding two Sigstore bundles:

sha256:658b3428...80b9   812 bytes
  artifactType : application/vnd.dev.sigstore.bundle.v0.3+json
  predicateType: https://slsa.dev/provenance/v1
  created      : 2026-08-05T16:14:12.946Z

sha256:59bac8c5...d9cf   814 bytes
  predicateType: https://spdx.dev/Document/v2.3
  created      : 2026-08-05T16:14:22.004Z

Build provenance and an SBOM attestation, signed nine seconds apart. Both manifests
carry subject pointing back at sha256:96b80ae1…5929 — the single place where
"this signature belongs to that image" is actually written down.

Pull the bundle blob and hash it:

curl -sL -H "Authorization: Bearer $TOKEN" \
  https://ghcr.io/v2/soit-ai/soit/server/blobs/sha256:eae5d902...dd86 -o bundle.json
sha256sum bundle.json
# eae5d902186b490570aba6f702f7ce03c46f935e403bf881ec27d99480d7dd86 *bundle.json

The hash equals the digest I asked for. A content-addressed registry means this step
verifies itself; no trust required yet.

3. What the signature actually covers

The bundle has three top-level fields — mediaType, verificationMaterial,
dsseEnvelope. The envelope's payloadType is application/vnd.in-toto+json, and
the base64 payload is the only thing the signature covers:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{
    "name": "ghcr.io/soit-ai/soit/server",
    "digest": {"sha256": "96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"}
  }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://actions.github.io/buildtypes/workflow/v1",
      "externalParameters": {
        "workflow": {
          "ref": "refs/tags/v1.0.0",
          "repository": "https://github.com/soit-ai/soit",
          "path": ".github/workflows/release.yml"
        }
      },
      "internalParameters": {
        "github": {
          "event_name": "push",
          "repository_id": "910429753",
          "repository_owner_id": "193298865",
          "runner_environment": "github-hosted"
        }
      },
      "resolvedDependencies": [{
        "uri": "git+https://github.com/soit-ai/soit@refs/tags/v1.0.0",
        "digest": {"gitCommit": "8105cae074f1f27d7916acfe02f9d4eabb63169f"}
      }]
    },
    "runDetails": {
      "builder": {"id": "https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0"},
      "metadata": {"invocationId": "https://github.com/soit-ai/soit/actions/runs/31023642816/attempts/1"}
    }
  }
}

One complete sentence: commit 8105cae… on refs/tags/v1.0.0, built by
.github/workflows/release.yml on a GitHub-hosted runner in run 31023642816,
produced the image with digest 96b80ae1….

I had the repository on disk, so I closed the loop immediately:

git rev-parse v1.0.0^{commit}
# 8105cae074f1f27d7916acfe02f9d4eabb63169f

Match. Note repository_id and repository_owner_id: numeric IDs are worth more
than the repository name, because names can be renamed and transferred, IDs cannot.

4. The certificate that lived for ten minutes

verificationMaterial.certificate.rawBytes is a 1735-byte DER certificate:

issuer=O = sigstore.dev, CN = sigstore-intermediate
subject=
notBefore=Aug  5 16:14:11 2026 GMT
notAfter =Aug  5 16:24:11 2026 GMT
X509v3 Subject Alternative Name: critical
    URI:https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0

Two things worth stopping on.

The subject is empty. The field that traditionally names the holder is blank; the
identity moved entirely into that SAN URI, and it is not a person or an organisation —
it is a workflow at a ref.

The validity window is exactly ten minutes. That is keyless signing: the pipeline
trades an OIDC token for a short-lived certificate, signs, and throws the key away.
Nothing to store, nothing to rotate, nothing to steal later. Ten minutes on, the
certificate is expired — and the signature still verifies, for the reason in section 5.

The certificate also carries a block of Sigstore extensions describing the build. The
values below are what I read; I am deliberately not putting names on the OIDs —
those live in Fulcio's OID documentation, and the values speak for themselves:

Extended Key Usage is Code Signing, Key Usage is Digital Signature only, and
there is a CT precertificate SCT timestamped Aug 5 16:14:11.958 2026 GMT — the act
of issuing this certificate was itself logged publicly.

5. Why an expired certificate still counts

verificationMaterial.tlogEntries holds exactly one record:

logIndex        : 2346649359
integratedTime  : 1785946452   ->  2026-08-05T16:14:12Z
kindVersion     : {kind: dsse, version: 0.0.1}

That record lives in Rekor, the public append-only transparency log, and its job is
to witness when. The question a verifier asks is not "is this certificate valid
now" — it expired long ago — but "was it valid at the moment of signing". The log
pins that moment at 16:14:12Z, inside the 16:14:11–16:24:11 window.

A detail from the actual bundle: timestampVerificationData is an empty object.
No RFC 3161 timestamp; the Rekor entry carries the time on its own. Not wrong, just
worth knowing what your trust actually rests on.

6. Where the SBOM lives and what is in it

The second bundle's predicate is a complete SPDX document. The bundle blob is
3437645 bytes:

spdxVersion   : SPDX-2.3
dataLicense   : CC0-1.0
name          : ghcr.io/soit-ai/soit/server
creationInfo  : creators ["Organization: Anchore, Inc", "Tool: syft-1.42.3"],
                created "2026-08-05T16:14:08Z"
packages      : 710
relationships : 2840

Those 710 packages by purl type: 469 pkg:deb, 217 pkg:pypi, one
pkg:generic (python 3.11.15) and one pkg:oci (the image itself).

The 469 Debian packages come from the base image — server/Dockerfile starts at
python:3.11 — and only 217 are our own Python dependencies. That ratio is itself a
finding: two thirds of what you just signed, you did not write.

All three images side by side show the split clearly:

The 64 extra pypi packages in knowledge-worker include torch, transformers and
nvidia-cublas-cu12. The two images are two targets of one Dockerfile; the entire
difference is uv sync --extra knowledge-worker (server/Dockerfile:19 vs :27).
web is node:24-alpine, so the Debian packages become 18 apk ones.

7. The 16 MiB ceiling, and what it cost us

An SPDX document normally carries a files section recording which files each
package was identified from. All three of ours have none — and it is deliberate,
right there in the workflow (.github/workflows/release.yml:132–149):

jq 'del(.files)
    | .packages |= map(del(.hasFiles))
    | .relationships |= map(select(
        ((.spdxElementId // "") | startswith("SPDXRef-File") | not)
        and ((.relatedSpdxElement // "") | startswith("SPDXRef-File") | not)))' \
  "$f" > "$f.tmp"
mv "$f.tmp" "$f"
size="$(stat -c%s "$f")"
test "$size" -le 16000000

The comment above it gives the reason: per-file SPDX entries push an ML-heavy image's
SBOM past the 16 MiB subject limit of actions/attest. In plain terms, the SBOM had
to be shrunk until it could be signed.

The cost is real. You keep the package inventory and the package-to-package
relationships; you lose "which file did this package come from", which is exactly what
you want when a CVE lands and you need to locate it. The test -le 16000000 line is a
hard gate: if a future image blows past the ceiling the release fails there rather
than shipping an SBOM that cannot be signed. That part I think is right — better a
broken build than a half-signed artifact.

8. An SBOM is a scan result, not the truth

Two things I found in ours.

Five Windows launchers in a Linux image. Among the 710 packages,
Simple Launcher 1.1.0.14 appears five times, each with a CPE and no purl:

SPDXRef-Package-binary-Simple-Launcher-d60858f6579e7bb1
  versionInfo : 1.1.0.14
  externalRefs: cpe:2.3:a:Simple_Launcher:Simple_Launcher:1.1.0.14:*:*:*:*:*:*:*

Syft's binary classifier found them: Python packaging wheels ship small Windows
launcher executables, which will never execute inside this Linux image but are
inventoried all the same. And because the files section was stripped (section 7),
you cannot even find out where they are.

A fourth hash that matches nothing. The pkg:oci entry representing the image
itself reads:

pkg:oci/ghcr.io%2Fsoit-ai%2Fsoit%2Fserver@sha256%3Ae7c993b9ac5d7322058e0169c678b4ce21fe42fac72c5d3374404bf4642939ea?arch=amd64

e7c993b9… is not the index digest, not the platform manifest digest and
not the config digest. Asked for directly, the registry returns 404. I could not
determine what it is — the reasonable guess is an identifier the scanner computes
locally — so I am reporting the observation and not a conclusion. The practical rule
is clear enough: do not reconcile an SBOM's internal digest against the signature's
subject. The binding lives in exactly one place, subject.digest, and that value is
96b80ae1…, matching the registry exactly.

9. Three v1.0.0 images, all of them verifiable

Back to those two extra tags. Fetched the same way:

All three provenance statements name refs/tags/v1.0.0 and the same signing identity.
The commits differ:

All three commits are on main locally, 23:33 / 23:44 / 00:05 in my timezone. The
story reads itself off the commit subjects: the tag was re-pointed twice that
night while the SBOM step was being fixed — and each failed attempt had already
pushed and signed its images before failing.

So this happens:

gh attestation verify \
  oci://ghcr.io/soit-ai/soit/server@sha256:3a5b3b1a...8246 \
  --repo soit-ai/soit --format json

exit code : 0
subject   : 3a5b3b1a...8246
predicate : https://slsa.dev/provenance/v1
ref       : refs/tags/v1.0.0
commit    : 0dacfc5219c4ae57d024f9cb4208e1efad5f0d17
san       : https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0
tlog      : rekor.sigstore.dev @ 2026-08-05T23:41:11+08:00

Exit 0. Nothing is wrong with the signature. It states honestly what it is: an
image built by our release workflow, on refs/tags/v1.0.0, from commit 0dacfc52….
The one thing it does not say is that it is not the image we shipped.

No attack here — we built all three. But swap in a nastier scenario and it holds
immediately: a pipeline compromised once, fixed, re-tagged, with the poisoned
intermediate still sitting in the registry. Attestation will not catch that.
gh attestation verify answers "did this come out of that pipeline". It never
promised to answer "is this the release".

10. The missing link: a manifest that pins tag, commit and digest

That answer is not in the signature. It is an asset on the GitHub Release, which
carries seven files:

soit-1.0.0.tar.gz          3148831
SHA256SUMS                     433
release-artifacts.json        2877
server.spdx.json           3313874
knowledge-worker.spdx.json 4004516
web.spdx.json              3862614
source.spdx.json           3694033

release-artifacts.json is the manifest:

{
  "featureKey": "release.artifacts",
  "schemaVersion": 1,
  "version": "1.0.0",
  "release_tag": "v1.0.0",
  "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f",
  "clean_worktree_at_tag": true,
  "images": [{
    "component": "server",
    "name": "ghcr.io/soit-ai/soit/server",
    "release_tag": "v1.0.0",
    "digest": "sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929",
    "reference": "ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929",
    "sbom": {"format": "spdx-json", "path": "server.spdx.json", "sha256": "b83c3bc7...ef68", "attestation": "..."},
    "provenance_attestation": "..."
  }]
}

It says the thing the signature would not: v1.0.0 is 96b80ae1…, not the other
two. It is assembled by jq inside the publish-release job
(release.yml:300–347) and validated on the spot. I fed the copy downloaded from the
public internet straight into the same script that lives in the repository:

{
  "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f",
  "images": ["knowledge-worker", "server", "web"],
  "passed": true,
  "release_tag": "v1.0.0"
}
exit=0

178 lines, standard library only, no network
(server/scripts/verify_release_artifacts.py). It insists that:

• release_tag equals v + version (:42);
• commit is 40 lowercase hex characters and not all zeroes (:45);
• clean_worktree_at_tag is true (:49);
• every image reference equals name@digest (:91) — a name:tag reference is rejected outright;
• digest matches ^sha256:[0-9a-f]{64}$ (:88);
• exactly the three expected components are present (:15, :114);
• no attestation URL is reused across entries (:107–112).

A test guards that last-but-two rule: it rewrites the example's reference into tag
form and asserts the script fails with digest-pinned
(server/tests/unit/test_release_operations_contract.py:85–88).

11. I reproduced the source tarball, byte for byte, on Windows

The release also ships a git archive tarball (release.yml:253–263) described as
deterministic. Worth testing rather than believing. First, the checksum:

grep "soit-1.0.0.tar.gz" SHA256SUMS | sha256sum -c -
# ./soit-1.0.0.tar.gz: OK

Then rebuild it locally:

git archive --format=tar.gz --prefix="soit-1.0.0/" \
  --output=local.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f

c11179c379ba7390c215133f342c92a7517a548f1536959cb43e187b16a7bab3 *local.tar.gz
5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *soit-1.0.0.tar.gz

Different — and not just at the gzip layer; the inner tars differ too. Comparing
members explains it:

members   : 1551 local / 1551 released   -- same
only-one-side: 0 / 0                     -- same
differing : 1280 members
mtime     : 1785945959 everywhere        -- same
mode      : 0664 everywhere              -- same
example: soit-1.0.0/.github/workflows/quality.yml   local 15270  released 14803

Sizes only, and each delta equals that file's line count. CRLF. My machine has
core.autocrlf=true, so git archive helpfully rewrote the line endings. Turn it
off:

git -c core.autocrlf=false archive --format=tar.gz --prefix="soit-1.0.0/" \
  --output=local2.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f
# 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *local2.tar.gz

Identical. A Windows box, git 2.55.0, a month later, reproducing the tarball
GitHub Actions produced.

Both halves matter. Determinism is real — git archive writes no build timestamp and
takes mtimes from the commit — but it is sensitive to local git configuration, and
the first suspect when reproduction fails is core.autocrlf, not the publisher.

12. The checklist

Everything above, compressed into something you can paste:

# 1. Resolve the digest (never keep books against a tag)
docker buildx imagetools inspect ghcr.io/soit-ai/soit/server:v1.0.0 | head -3

# 2. Verify build provenance
gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit

# 3. Verify the SBOM attestation (NOT checked by default)
gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit \
  --predicate-type https://spdx.dev/Document/v2.3

# 4. Check the release manifest: is this digest the one that shipped
curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/release-artifacts.json
curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/SHA256SUMS
sha256sum -c SHA256SUMS

# 5. Run it by digest, not by tag
docker pull ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929

Step 3 deserves emphasis: gh attestation verify checks provenance only by
default. Without --predicate-type it returned exactly one result here, the SLSA
provenance. With it, the SPDX attestation comes back — 6022876 bytes of JSON holding
the same 710 packages counted in section 6. A lot of people assume "verified" includes
the SBOM. It does not.

Step 4 is the point of this whole piece. Without it, the green checkmark from step 2
shines just as brightly on the 3a5b3b1a… image from section 9.

13. Eight things that still do not line up

House rule: the last section is about our own problems.

1. Our own compose pins tags, not digests. All six services in
docker/docker-compose.images.yml use :${SOIT_IMAGE_TAG:-v1.0.0} (:17–:32), and
the README says the same (README.md:146). We demand digest-pinned references inside
release-artifacts.json and then hand users a tag. Impact: a tag can be re-pointed.
Workaround: step 5 of the checklist. I plan to open an issue; none filed at the
time of writing, hence no link.

2. SHA256SUMS does not cover release-artifacts.json. The 433-byte file lists
five entries — four SPDX documents and the tarball. Neither the manifest nor
SHA256SUMS itself is in there (the latter cannot be, by construction). The manifest
does get its own attestation (release.yml:350–353), so it is not unprotected, but
the instinctive "download everything and run sha256sum -c" does not reach it.
Also going to open an issue.

3. docs/release-process.md lags the pipeline. It tells you to copy the real
tag, commit, digests, SBOM checksums and attestation URLs into an evidence document
and then run the verifier (:24–:32). The pipeline has been doing that
automatically for a while, and uploads the result as a release asset. Impact: a reader
concludes the manifest is hand-written after the fact, which makes it look weaker than
it is. Also going to open an issue.

4. Superseded builds are never cleaned up. The two images from section 9 are still
pullable, still verifiable, still runnable. We have no "delete same-tag intermediates
after a successful release" step. Impact: as described. Workaround: pull the digest
from release-artifacts.json.

5. The verifier checks shape, not truth. provenance_attestation only has to be a
non-empty string (:58, :64, :106, all via _require_text). The script never
fetches the URL and never compares the attested subject digest to the one in the
manifest. Impact: a well-formed manifest full of fabricated URLs passes. Workaround:
it is a structural check, not a trust check — the trust comes from
gh attestation verify, and you need both. Defensible division of labour, but the
docs never say so.

6. REQUIRED_IMAGES is hard-coded. verify_release_artifacts.py:15 pins
{"server", "knowledge-worker", "web"} and line 114 demands exact equality. Add a
fourth image and the release breaks at the final step. Impact: a bad surprise on
release day. Workaround: know it is there — and the case it blocks (silently shipping
with an image missing) is worth more than the nuisance.

7. No process for SBOM false positives. Sections 7 and 8. Nobody owns annotating
Simple Launcher as noise, so the next reader has to work it out again. Impact: the
signal-to-noise ratio degrades as images grow.

8. I could not identify e7c993b9…. That is where section 8 stops. This entry
exists so I do not pretend otherwise: I can confirm it is absent from the registry
and differs from the other three digests; I cannot tell you how it is derived.

Disclosures

• This is a verification walkthrough, not a runtime one. I did not pull the images and stand up a stack; every conclusion stops at the bytes-and-signatures layer.
• Section 9 is not a security incident. We built all three images ourselves on the same night. It illustrates the semantics of attestation, not a breach.
• Section 8's mismatched digest is an observation, not a conclusion. The guess is labelled as a guess.
• All numbers belong to the v1.0.0 release (tag commit 8105cae…). Later releases will have different values; the method does not change.
• I only exercised the success path of gh attestation verify (exit 0). I did not construct a tampered image to see how it fails.
• Disclosure: I maintain SOIT.

One-line takeaway

A passing attestation says "these bytes came out of that pipeline"; it does not say
"this is the release." What joins the two is a manifest binding tag, commit and
digest — ours is release-artifacts.json, watched by a 178-line script whose hardest
rule is a single line: the reference must be name@digest, and a tag reference fails.

Try it, and please poke holes

The repository is github.com/soit-ai/soit, and every step above is reproducible:

1. the curl calls in section 2 need no account — an anonymous token is enough;
2. the 3a5b3b1a… image from section 9 is still there; verify it yourself and watch it exit 0;
3. for the byte-identical rebuild in section 11, remember -c core.autocrlf=false.

If any claim here is wrong — especially the digest in section 8 that I failed to
identify — open an issue and say so. I would rather learn where this is wrong than be
told the pipeline looks thorough.