Every write-heavy system I've worked on eventually hits the same wall, and it's rarely the wall anyone budgeted time for.
Not a traffic spike. Not a bad query someone forgot to index. Something quieter: the database itself, slowly getting heavier under its own weight, until one day a query that used to take milliseconds takes seconds, and nobody can immediately say why.
That's bloat. And in 18 years of building payment and transaction systems, it's cost me more debugging hours than almost any other single category of problem — precisely because it doesn't announce itself.
Why it happens, in plain terms
Most relational databases don't delete or update a row in place. Postgres marks the old row version as dead and writes a new one; MySQL's InnoDB does something conceptually similar with its own row versioning under MVCC. This is by design — it's how databases give you consistent reads without locking half the table.
The catch: those dead rows don't vanish. They sit there, taking up space in the table and every index on it, until something goes and reclaims them. In Postgres, that's VACUUM. In InnoDB, it's a mix of purge threads and, eventually, a table rebuild if things get bad enough.
If your write volume is high — lots of updates, lots of deletes, high-churn queues or state machines — you can generate dead rows faster than routine cleanup reclaims them. The table keeps growing on disk even though the logical data isn't. Indexes get fatter and less cache-friendly. Every query touching that table pays a small, invisible tax. Multiply that tax by every request, every day, for months, and you get exactly what I saw once: a single-character logic bug in how we handled a status flag turned into 53 seconds of unexplained latency, because it was quietly generating far more row churn than anyone had accounted for.
Why it's so easy to miss until it's expensive
A few reasons this class of problem slips past normal engineering discipline:
- It degrades gradually, not suddenly. There's no alert that fires at "20% bloated." By the time it's visible in your latency graphs, it's usually been building for weeks.
- It doesn't show up in code review. The bug isn't in your query. It's in the pattern of writes your query generates, which is much harder to reason about statically.
- Standard monitoring doesn't surface it well. You'll see slow queries. You often won't see why they're slow, unless you're specifically looking at table and index bloat metrics — which most teams aren't, until they've been burned once.
- Autovacuum/purge settings that were fine at launch stop being fine at scale. The defaults are tuned for a generic workload, not your specific write pattern. Nobody revisits them until something breaks.
What's actually worked for me
Treat bloat as a metric, not an incident. Track table and index bloat percentage the same way you track p99 latency or error rate — as an ongoing dashboard, not something you check when things are already slow.
Match cleanup aggressiveness to write pattern, not table size. A small table with extremely high churn (think: status flags, counters, queue-like tables) often needs more aggressive vacuum/purge tuning than a much larger, mostly-append table. Tune per table, not globally.
Be suspicious of "just add a flag column." Status flags, soft-delete columns, and toggle-heavy schemas are the most common source of the update churn that drives bloat. They're also the easiest thing for an engineer to add without thinking about the write amplification they cause. I've learned to ask "how often does this column change, and what does that do to the row?" before approving that kind of schema change now.
Build the runbook before you need it. Know in advance what a manual bloat-reclaim operation costs you — how long a table rebuild or full vacuum takes, whether it locks writes, what your rollback plan is. Figuring this out during an incident, on a production payments table, is a bad place to learn it.
The real lesson
Bloat isn't really a database problem. It's a systems-thinking problem — it's what happens when the shape of your writes over time isn't something anyone is explicitly tracking, only the shape of your reads. The fix isn't a smarter query. It's treating write patterns as a first-class design concern, with the same rigor you'd apply to an API contract.
The systems that don't get bitten by this aren't the ones with cleverer engineers. They're the ones who made "what does this write pattern do to the table over six months" a normal question to ask in design review — before it became a normal question to ask during an incident review.
What's the sneakiest "invisible tax" problem you've hit in a production system — the kind that doesn't show up until months in? Curious what others have run into.
