Your storage format is the one thing you cannot refactor

Ten days ago I published a post about encrypting personal data at rest in Symfony. A reader, Mads Hansen, left a comment that did not argue with the cryptography at all. It argued with seven characters that were not there.

The type I had shipped told encrypted rows from legacy plaintext rows by looking at the first byte: a { or a [ meant "this row predates encryption, hand it back as JSON". His objection, compressed: that makes the storage format ambiguous forever, and the ambiguity is a security property, not a cosmetic one, because a row that merely looks like JSON opts out of decryption no matter who wrote it.

He was right, and fixing it cost about eighty lines including tests. But the reason I am writing a second post rather than editing the first one is that the bug was not really about encryption. It was about a rule I had followed everywhere else in that codebase and broken here, and it is the one rule of software design I would keep if I had to throw away all the others:

everything you write can be refactored later, except the format of the bytes you have already stored.

Code is soft, storage is not

We say "software" and mean it. A class name is a rename away from being better. A bad abstraction survives until someone has an afternoon. Even a database schema, which feels heavy, is genuinely malleable: a column is added, backfilled, and dropped, and the tooling for that has been boring and reliable for twenty years.

Formats are different in kind, not in degree, and it took me an embarrassing number of years to feel the difference in my hands rather than know it as a maxim. A format is not code you own. It is an agreement between the process that wrote a value and every process that will ever read it, including the ones running a version of your code that no longer exists, on rows written by a colleague who left, in a database restored from a backup taken before the decision you are about to make.

You cannot change one side of an agreement unilaterally. What you can do is read the old shape and write the new one, which is exactly what a migration is, and which is exactly what requires you to be able to tell the two apart. That last part is the whole game.

Sniffing is the tax you pay for skipping the version

When a stored value carries no marker, the only way to know what it is, is to look at it and guess. Every codebase has these, and once you learn to see them they are everywhere:

• if ($v[0] === '{') then it is JSON, otherwise it is PHP serialize() output.
• if unserialize() throws, fall back to json_decode().
• if the first two bytes are 0x1f 0x8b, it is gzipped, otherwise it is raw.
• if the string is 32 hexadecimal characters, it is the old token format.
• if the value is base64-decodable, it is the encrypted one.

Each of these works. That is precisely the problem: they work well enough that nobody replaces them, and each one quietly promotes an accident of the payload into part of the contract. The last example is the one I shipped, and it is the worst of the family, because "looks like plaintext" and "is plaintext" were being treated as the same statement on a column that exists to keep buyers' names and addresses from being readable.

There is also a subtler cost, the one that bites long after. A heuristic is not just a guess, it is a constraint on all future values. Once "starts with {" means legacy, no future version of the format may ever start with {. You have spent a byte of design space without noticing, and you will notice the day you want to store something that starts with {.

The window closes when the first row is written

Here is the asymmetry that makes this worth a post.

Adding a version marker to an empty column costs seven bytes and about four minutes. Adding one to a column with a million rows costs a migration that must read values written without a marker, which means the migration needs a heuristic, which is the very thing you were trying to get rid of. You do not escape the guess, you only get to bury it in a one-off script instead of the read path, and you get to keep it in the read path anyway until the backfill is provably complete.

So the marker is free exactly once, and the free moment is the moment the column is empty. Which is also the moment when you are most certain you will never need it, because there is only one format and it is obviously the right one. Every ambiguous storage format I have ever met was created by a competent person on the day it was cheapest to prevent.

My table had five rows. That is the only reason this post describes a fix rather than a permanent scar. If it had held a million, the honest advice would have been the opposite of what follows: keep the heuristic, document it as load-bearing, and write the version prefix only into new values, carrying both paths for years.

What the fix looks like

The value stored is now an envelope, and the envelope is three fields that each earn their place:

enc:v1:k1:<base64 nonce + ciphertext>

enc:v1: is the format version. k1 is the key identifier. The payload is the payload. Written out as PHP, the read path stops asking the value what it is:

public function convertToPHPValue($value, AbstractPlatform $platform): ?array
{
    if (!\is_string($value) || '' === $value) {
        return null;
    }

    if (!CipherEnvelope::isEnveloped($value)) {
        // Fail closed: a readable row is a bug or an intrusion, never a fallback.
        throw new \RuntimeException('This value is not in the "'.self::PREFIX.'" envelope.');
    }

    [$keyId, $ciphertext] = CipherEnvelope::split($value);

    return $this->decode(self::keyring()->get($keyId)->decrypt($ciphertext));
}

Three consequences follow, and only the first one is obvious.

Reading fails closed

There is no longer a branch that returns plaintext. A row that does not carry the envelope stops the application instead of quietly serving personal data in clear.

That is a real cost, paid at the worst moment: restore a pre-migration backup and the app breaks rather than degrades. I want that trade on this column, because the failure mode it replaces is silent. As a general rule for anything with a security dimension: when a read path has two branches and one of them hands back the unprotected version, the branch is the vulnerability, no matter how carefully the condition is written.

The key identifier makes rotation real

The previous post admitted, in as many words, that key rotation was not implemented and probably never would be, because rotating means holding two keys at once and the ciphertext carried no way to say which one produced it.

Once the identifier is in the envelope, rotation stops being a flag day and becomes three ordinary steps: promote a new active key while the old one stays in the keyring as read-only, backfill in the background, then drop the old key. Nothing has to happen atomically, and the day a key is retired too early, the failure is a message naming the missing key identifier instead of an authentication error you have to reverse-engineer.

/** @throws \RuntimeException when a stored value names a key we no longer hold */
public function get(string $id): TokenCipher
{
    return $this->ciphers[$id]
        ?? throw new \RuntimeException(\sprintf('This value was encrypted with key "%s", which is not in the keyring.', $id));
}

Note what happened there: a format decision bought back a capability. Seven bytes plus an identifier turned "we cannot rotate keys" into "we can rotate keys during business hours". That is the usual shape of this trade, and it is why the version prefix is not speculative generality. It does not anticipate a feature, it preserves the right to have one.

One detail worth stealing: the identifier that comes out of the database is validated against /^[A-Za-z0-9_-]{1,32}$/ before it is used or logged. It is attacker-influenced data on the day the threat model says the database is the thing you lost, and it ends up in exception messages.

The backfill does not guess either

The conversion command reads pre-envelope values by attempting decryption first, with every key in the keyring, and only treating the value as plain JSON if all of them fail:

foreach (EncryptedJsonType::keyring()->all() as $cipher) {
    try {
        return $this->decode($cipher->decrypt($stored));
    } catch (\Throwable) {
        // Not this key, or not a ciphertext at all: try the next one.
    }
}

// Nothing decrypted it: it can only be a value from before encryption.
try {
    return $this->decode($stored);
} catch (\Throwable) {
    return null;
}

This ordering is not stylistic. Base64-decoding {"name":"Ada"} fails outright, so plaintext can never be mistaken for ciphertext, while the reverse test (does it look like JSON?) has no such guarantee. When you must disambiguate, prefer the test that fails for structural reasons over the test that inspects appearances.

And because the type now refuses anything unenveloped, the same command doubles as the answer to "is the database fully converted?":

bin/console app:encrypt-buyers --dry-run

exits non-zero as long as one row is outside the envelope. That is a cron job and an alert, rather than a paragraph in a runbook that nobody reads after the deploy.

Where else the seven bytes apply

Encryption made this vivid, but the column was not special. Anything that outlives the process that wrote it is a format, and deserves a version:

• Serialized columns. A json column holding a value object is a format the moment the object gains or loses a field. Version it, or you will be reading array_key_exists archaeology in two years.
• Cache payloads. On disk, in Redis, anywhere. The classic failure is a deploy that changes a cached structure and reads yesterday's shape for the ten minutes before the cache turns over. A version in the key is the cheapest variant: old shape, old key, no reader ever sees it.
• Queue messages. During a rolling deploy, two versions of the consumer run at once, by design. The message on the wire is a format shared between two codebases that happen to be the same repository at different commits.
• Public identifiers. Anything you print on an invoice, put in a URL, or let a customer paste into a support ticket has escaped your control permanently. A one-character prefix (inv_, cus_, the whole Stripe convention) is the same seven bytes wearing a different hat, and it also makes a leaked identifier greppable.
• Exported files. A CSV whose column order is the contract, because someone built a spreadsheet on top of it.

The test I now apply, and it takes one second: will this value be read by code I have not written yet? If yes, it needs to say what it is. Not because I expect to change it, but because "I will never need to change this" is a prediction about the future, and the version prefix is the cheapest insurance ever written against being wrong about one.

Is this not YAGNI?

The objection deserves a straight answer, because "you are not going to need it" is usually right and I use it constantly.

YAGNI is a rule about code, and it works because code is deletable. Write the abstraction you do not need, discover it, delete it: the cost of being wrong is bounded and it is paid by one person in one afternoon.

Formats break the assumption the rule stands on. You cannot delete a format that is already in the database, because you do not own all of its copies: some are in backups, some in a partner's system, some in a CSV a customer downloaded last March. The cost of being wrong is unbounded and it is paid by whoever is on call. When the cost of being wrong is asymmetric by three orders of magnitude, the cheap option is not the smaller diff.

What it actually cost

Eighty lines, tests included, and two hours. A prefix constant, a small class that wraps and splits the envelope, a keyring with one active key and any number of retired ones, a read path that throws instead of falling back, and a backfill command that tries decryption before it tries trust.

The part I keep thinking about is that none of it came out of a threat model, a checklist, or a review. It came from someone reading a public post carefully enough to notice that a condition was doing more work than it looked like. If you write technical posts and treat the comments as the part that comes after the work, that is backwards: the objection is the work, and it is the only free code review you will ever be offered.

The code above lives in InvoiceAnvil, a Shopify app that issues compliant invoices, built on ShipAnvil, the Symfony kit that ships the auth, billing, admin and deploy pipeline these apps stand on. The post that started this one is encrypting personal data at rest in Symfony, and the foundations are in deploying Symfony to a production VPS.

https://shipanvil.com/blog/storage-format-cannot-refactor