Your Integrity Checks Are Watching the Wrong Layer

Ciao Amici 👋

Quick story before we get into the weeds. Last week I took a patient health record sitting in Amazon S3, a file clearly stamped as regulated under HIPAA, and I changed its security label to "public." Anything reading that label would now happily treat protected health information as freely shareable.

Here is the part that should make you put your coffee down. I never touched the file. Not one byte. The object's checksum was identical before and after. Every integrity monitor you could aim at that bucket would have looked, seen nothing, and gone back to sleep.

Because, technically, nothing did change. The file was fine. It was the context wrapped around the file that I quietly poisoned.

This is a story about Amazon S3 annotations, a feature that is only a few weeks old, and a gap in it I do not think anyone has written up yet. We are going to build a real classification pipeline on top of annotations, break it, catch the break, and then close the hole. It is a long one. Settle in.

First, what even are S3 annotations?

If you blinked and missed the announcement in June 2026, here is the short version. AWS gave S3 a new way to attach metadata to your objects, and it is far more generous than anything that came before.

You have always been able to bolt small things onto S3 objects. Tags. User metadata. But those older options are cramped little cupboards. Annotations are a warehouse. Look at the difference:

That bottom row is a different species. A full gigabyte of structured context per object, changeable at any time without rewriting the object, flowing automatically into an Apache Iceberg table you can query with Athena. AWS aimed it squarely at AI agents and analytics: give your data enough context that an agent can find and understand it without a human in the loop.

And honestly? It is a great feature. I am not here to dunk on it. But read that table one more time, specifically the "change without rewriting the object" column, and just sit with it for a second. The context can move while the object stays frozen solid. Hold that thought. It is the whole article.

The idea, and the idea that was already taken

My first instinct was the obvious one. "Annotations mean you can finally delete your metadata database." Attach the context to the object, query it with Athena, retire the DynamoDB table you have been syncing with a Lambda and praying never drifts. Clean. Satisfying. Blog-worthy.

Then I did my homework, like you should before you write anything, and found somebody had already published exactly that take. Plus a sharper follow-up about how moving context onto the object changes who is allowed to edit it. Good pieces. They beat me to it.

So I sat with it longer. And the thing that kept poking me was that "mutable" column. If the context is now a first-class, independently writable thing living on the object, then the permission to write that context is also a first-class, independent thing. Which means somebody could be holding it without you ever realizing you handed it over.

That is not a metadata-database story. That is a security story. And that one, nobody had written.

Building something worth attacking

You cannot demo a heist against an empty vault. I needed a legitimate, believable system first, something that uses annotations the way AWS intends, so that breaking it actually means something.

So I built a document classifier. The job is simple: look at a file, decide how sensitive it is, write that verdict onto the object as an annotation. Four levels, least to most touchy: public, internal, confidential, regulated.

I kept the classifier deliberately dumb. No Bedrock, no model, just rules and regex. Partly to keep it free and reproducible, partly because the classifier was never the interesting part. The annotations are. Here is the core of it:

_SSN = re.compile(r"\b\d{3}-\d{2}-\d{4}\b")
_CARD = re.compile(r"\b(?:\d[ -]?){13,16}\b")

_REGULATED_TERMS = re.compile(r"\b(HIPAA|GLBA|protected health information|PHI|patient|diagnosis|MRN)\b", re.I)
_CONFIDENTIAL_TERMS = re.compile(r"\b(confidential|hr restricted|do not share|salary|card on file)\b", re.I)
_INTERNAL_TERMS = re.compile(r"\b(internal[- ]only|do not distribute|\.internal\.|runbook)\b", re.I)
_PUBLIC_TERMS = re.compile(r"\b(for immediate release|approved for public|press release)\b", re.I)

Now, one design choice here matters more than it looks, and it paid off in a way I did not plan. The rules run most-restrictive first. Regulated beats confidential beats internal beats public. If a file trips two rules, the scarier one wins. And when nothing matches at all, the default is internal, not public. Fail safe, not fail open. That last bit is a small philosophy with big consequences: when you are unsure how sensitive something is, you treat it as more sensitive, not less.

Here is the whole decision, in code:

def classify(text):
    if _REGULATED_TERMS.search(text) or _ROUTING.search(text):
        return "regulated", ["regulated-term-or-financial-identifier"]
    if _SSN.search(text):
        return "regulated", ["ssn-pattern"]
    if _CARD.search(text) or _CONFIDENTIAL_TERMS.search(text):
        return "confidential", ["card-or-confidential-marker"]
    if _INTERNAL_TERMS.search(text):
        return "internal", ["internal-marker"]
    if _PUBLIC_TERMS.search(text):
        return "public", ["public-marker"]
    return "internal", ["no-signal-default"]

Read top to bottom, it is basically a ladder of "how bad is this if it leaks," and the first rung that matches wins. Simple. Predictable. Which, spoiler for later, is exactly what makes the defense possible.

The test data, and a small happy accident

I generated six synthetic documents. Fake, but realistic, with genuine signal planted inside: a public press release, an internal architecture note, a customer invoice with a card number, a payroll CSV with real-looking SSNs, a HIPAA patient record, and a mortgage application stuffed with financial identifiers. No real PII anywhere. All invented. That matters, because this repo is public and the last thing I want is to ship someone's actual Social Security number to GitHub.

Ran the classifier. Here is the spread:

See the payroll row? I expected that one to land on confidential, because the file literally has the word "CONFIDENTIAL" stamped at the bottom. But the classifier found actual Social Security numbers inside and bumped it up to regulated. Which is correct. A human skimming for the "CONFIDENTIAL" label would have stopped there and under-classified the file. The most-restrictive-wins ordering caught something a careless reader would miss.

Tiny thing. But that tiny thing is how you know the logic is sound before you build anything on top of it.

Writing the verdict onto the object

Now the annotations. And here is our first war story, because AWS made me earn this one.

The plan was innocent: use the AWS CLI to write a classification annotation. The command exists, aws s3api put-object-annotation, with an --annotation-payload flag. Should be a one-liner.

Except that payload flag is a "streaming blob," and on Windows the CLI's blob parser refused every single path I fed it. fileb://, file://, absolute path, relative path, forward slashes, backslashes. Did not matter. Same error, over and over:

Error parsing parameter '--annotation-payload': Blob values must be a path to a file.

I lost a genuinely annoying chunk of time to that. Then I did the smart thing, which is stop fighting a tool that is losing you time, and switched to boto3, the Python SDK. It takes the payload as plain bytes and does not care about any of this path drama:

def write_classification(s3, bucket, key, level, reasons):
    payload = json.dumps({
        "level": level,
        "reasons": reasons,
        "source": "context-drift",
    }).encode("utf-8")
    return s3.put_object_annotation(
        Bucket=bucket,
        Key=key,
        AnnotationName="classification",
        AnnotationPayload=payload,
    )

Worked first try. So if you are on Windows and you hit that blob wall, there is your escape hatch: use the SDK, skip the CLI. That is not a workaround so much as the correct tool for the job, but it is worth writing down because you will not find it in the docs.

I wired that into a scan that walks the bucket, classifies each object, writes the annotation, then reads it straight back to confirm the round trip actually landed. Six files, six clean writes, every annotation verified through S3. The legitimate system was alive and behaving.

Now, the fun part. Breaking it.

Here is the scenario. Picture an access gate, or an AI agent, that reads the classification annotation to decide who gets to see a file. Confidential and regulated stuff gets locked down. Public flows freely. Totally reasonable design, and it is literally the use case AWS put in the announcement.

The question I wanted answered: who is actually allowed to change that annotation?

Your gut says it lines up with who can change the object. If you can rewrite the file, fine, maybe you can rewrite its label too. But if you can only, say, upload files, you should not be able to touch the security classification. That is the intuition. And the intuition is wrong, because of a single wildcard character almost everyone types without thinking.

The wildcard nobody thinks about

Be honest with yourself. You do not write s3:PutObject in your IAM policies. You write s3:Put*, because it is convenient and you do not feel like enumerating every put action by hand. I do it too. Everybody does it.

The problem is that s3:Put* does not just mean "upload objects." It quietly swallows s3:PutObjectAnnotation. So the second you grant s3:Put* to your friendly little uploader role, you also handed it the power to rewrite the context that decides how sensitive every object in that bucket is.

I did not want to just assert this and move on. I wanted receipts. IAM has an API, simulate_custom_policy, that evaluates a policy without you attaching it to a live identity. So I fed it the naive uploader policy and asked two questions: can it upload, and can it annotate?

Naive 's3:Put*' uploader policy:
  s3:PutObject           -> allowed
  s3:PutObjectAnnotation -> allowed   <-- the gap

Hardened policy:
  s3:PutObject           -> allowed        (uploads still work)
  s3:PutObjectAnnotation -> explicitDeny   (tampering blocked)

There it is in black and white. Top block: an "uploader" role can rewrite classifications, no questions asked. The person who wrote that policy believes they granted file uploads. They actually granted context rewriting. Same costume, very different powers underneath.

The money shot: same bytes, different label

Simulation proves the gap in theory. But I wanted to watch it happen to a real object. So I pointed a tamper script at the patient record, the HIPAA one, currently and correctly labeled regulated.

The script does three things. Read the object's ETag and its current classification. Overwrite only the annotation, flipping it to public. Read the ETag and classification again. The object bytes are never touched.

Target: patient-record-882.txt
  object ETag before : "f1f48ca4ab47402c49afa1433a50ae02"
  classification before: regulated
  --- annotation overwritten, object bytes untouched ---
  object ETag after  : "f1f48ca4ab47402c49afa1433a50ae02"
  classification after : public

Look at those two ETags. f1f48ca4... and f1f48ca4.... Identical. The file did not move. Its checksum did not change. If you had a Lambda watching for object modifications, or a bucket-level integrity monitor, or versioning quietly tracking every write, not one of them would fire. From the point of view of every tool that watches the bytes, absolutely nothing happened here.

But a HIPAA patient record is now labeled public, and any agent trusting that label will hand it to whoever knocks.

That gap between the last two rows, integrity says fine while safety says compromised, is a thing worth naming. I have been calling it context drift. The data stays exactly where it is while the meaning attached to it wanders off, and every defense you have is standing guard over the data.

Catching it: you cannot trust the label alone

Okay, so an attacker can poison the context invisibly. Great, scary. But an article that stops at the attack is just fear with a repo attached. The useful question is: how do you catch it?

The answer falls straight out of how we built the classifier. The classification is derived from the object's content. So you can re-derive it any time, from the bytes, and compare against what the annotation claims. If the two disagree, something is wrong. That is your drift detector, and it is almost embarrassingly simple:

for key in keys:
    body = s3.get_object(Bucket=BUCKET, Key=key)["Body"].read().decode("utf-8", "replace")
    content_level, _ = classify(body)
    annotated_level = read_classification(s3, BUCKET, key)["level"]
    if content_level != annotated_level:
        print(f"  {key}  content={content_level}  annotation={annotated_level}  <-- DRIFT")

Ran it. It lit up the tampered file instantly. And it caught a second one I had completely forgotten about, a leftover test object from when I was first poking at the API with junk data. Two hits:

  patient-record-882.txt   content=regulated   annotation=public        <-- DRIFT
  probe.txt                content=internal    annotation=confidential  <-- DRIFT

That accidental second catch is, honestly, the best argument for the whole approach. Drift detection does not only catch malicious tampering. It catches stale labels, honest mistakes, half-finished migrations, anything where the story on the object disagrees with the story inside the object. It is a consistency check, and consistency checks earn their money in boring ways just as much as dramatic ones.

The fix, and it is refreshingly small

Prevention is two lines of policy. Stop using the wildcard for roles that should only upload. Grant s3:PutObject explicitly, then explicitly deny the annotation writes:

{
  "Sid": "UploadsOnly",
  "Effect": "Allow",
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::YOUR-DEMO-BUCKET/*"
},
{
  "Sid": "NoContextTampering",
  "Effect": "Deny",
  "Action": ["s3:PutObjectAnnotation", "s3:DeleteObjectAnnotation"],
  "Resource": "arn:aws:s3:::YOUR-DEMO-BUCKET/*"
}

An explicit Deny in IAM always wins over any Allow, so even if some other policy hands this principal s3:Put* down the road, the Deny holds the line on annotations. Uploads keep working. Tampering is dead on arrival. That is the entire fix.

But, and I want to be straight with you, prevention and detection do different jobs and you want both. The Deny stops new tampering going forward. It tells you absolutely nothing about what already got changed before you woke up to any of this. So: the Deny at the front door, the drift audit sweeping the floors. One protects the future, the other investigates the past.

Where this honestly falls apart

I am not going to oversell the detector, because it has a real limitation and pretending otherwise would be exactly the kind of dishonest that annoys me in other people's write-ups.

The drift audit works here for one specific reason: the classification is fully derivable from the object's bytes. Deterministic classifier, ground truth living in the content. Re-run, compare, done.

The moment your annotation carries information that is not in the object, this breaks. Human-added context. A reviewer's notes. An external enrichment score. A model's summary that you cannot exactly reproduce. Anything a machine cannot recompute from the file alone. For those, there is no ground truth to check against, and this style of drift detection simply cannot save you. You would need something heavier: signed annotations, or an audit log of who wrote which annotation and when, which is a whole other design and a whole other article.

So take the detector for what it is. Strong when truth lives in the bytes. Useless when it does not. The important skill is knowing which situation you are in before you rely on it.

What I deliberately left on the table

One thing I skipped on purpose, and it is where this goes next. Annotations really shine when you enable S3 Metadata annotation tables, which index everything into an Apache Iceberg table you query with Athena. "Show me every regulated object across this petabyte bucket" becomes a single SQL query. That is genuinely powerful, and it is where the feature earns its keep at real scale.

I left it out to keep this focused on the security story. But the obvious sequel writes itself: run the drift audit not object-by-object in a Python loop, but as one Athena query across the entire annotation table, catching drift at scale in seconds instead of iterating. Same idea, industrial size. Consider that teased.

The takeaways, quickly

A few things I am walking away with, and you might too:

• New features quietly move the walls of your house, and your alarms do not move with them. Annotations made object context mutable and independently writable, and every integrity tool we own is still staring at the bytes. Whenever AWS ships something new, the question to ask is: what just became writable that was not before?
• Audit your wildcards. s3:Put* felt harmless for years. One new action slipped underneath it and turned it into a context-rewriting grant nobody signed off on. Go look at your Put* and * statements with fresh eyes now that annotations exist.
• Detection and prevention are not the same tool, and you want both. Deny at the door. Sweep for drift. One stops the next attack, the other tells you about the last one you missed.
• And when something you rely on tells you "nothing changed," ask it what exactly it was watching. Because in this case everything it watched really was fine. The whole problem lived in the one place it never thought to look.

Also, the full build is on GitHub. MIT licensed, synthetic data, runs against your own bucket in about five minutes. Clone it, poison your own patient record, then catch yourself doing it.

Repo: https://github.com/mursalfk/context-drift

Happy Coding! 👋