"Our uptime was pretty shaky." That is how Alex Chan, an engineer at Tailscale, opens the postmortem of the strangest database incident his team has ever chased (the writeup, 1,141 points and 217 comments on Hacker News at the time of writing). Over six months, a single bug corrupted shard databases across their control plane 19 separate times. No pattern. No obvious cause. No way to reproduce it on demand. Just "impossible" corruption, recurring, until months of forensics revealed that the fault had been sitting inside SQLite itself for at least 16 years.
I am a Senior Software Engineer II at BS23 in Dhaka, and I have spent the last six years building Spring Boot services and, more recently, running my own AI agent infrastructure. Database corruption is the bug class I dread most, because it does not announce itself with a stack trace. It shows up as a checksum error in a backup pipeline, or a transaction that quietly did not persist, or an index that disagrees with the table it points at. Full disclosure: everything in this article comes from Tailscale's published postmortem and the Hacker News thread; I have not touched their codebase. What I can stand behind is the debugging methodology, because it is the same discipline I use when a Spring Boot service starts misbehaving in ways the docs say cannot happen. This postmortem is one of the best debugging case studies I have read this year, and almost all of it transfers directly to Java teams running Postgres, MySQL, or even SQLite inside a Spring Boot app. Here is what happened, how they cracked it, and the checklist I now use when a database says "impossible."
The incident: 19 corruptions in six months
Tailscale's control plane, the service that coordinates every tailnet, is split into shards. Each shard has an SQLite database holding the configuration data for the tailnets on it, and a single Go process exclusively accesses that database. That single-writer design is exactly how SQLite is meant to be used, which makes what happened next so unsettling.
Their backup pipeline takes a complete snapshot of the database every few minutes and uploads the whole file to an S3 bucket. It ran without incident from early 2023 until August last year, when a data pipeline reading those S3 backups reported an error. The team ran SQLite's PRAGMA integrity_check against the backup and confirmed it: the database was corrupted.
- The first instinct was to treat it as a one-off. SQLite corruption is possible but rare, and not something you expect in normal operation. They repaired the affected database and investigated the cause. They found nothing.
- Then it happened again. And again. In total, 19 separate corruption incidents over six months before the underlying bug was resolved.
- Each incident meant real downtime. When corruption was detected, the control plane process on that shard had to stop while the database was repaired or restored. Early incidents took over an hour of downtime per shard. Tailnets on that shard lost the admin console and the API during recovery, and devices coming online could not learn about the network until the control plane was back.
The data itself was never catastrophic: these databases hold tailnet metadata, not private keys or traffic. But a company whose entire product is reliable connectivity cannot afford repeated status-page events, and every incident eroded trust even when most shards were unaffected.
Why it resisted every standard technique
Here is the part every developer who has ever debugged a heisenbug will recognize. The bug did not want to be found.
- There were no recent changes to blame. Nobody had touched the low-level SQLite interaction code for years. A fine-toothed review found nothing.
- There were no common factors. Corruption was not tied to a single shard, customer, tailnet feature, time of day, or load level.
- It could not be reproduced synthetically. With no reliable trigger, the only option was to deploy passive forensic telemetry into the live environment and wait for the next corruption to happen.
To make it worse, the corruption had no schedule. Incidents could be hours apart or weeks apart. There was a six-week period between October and December with zero incidents, and then they returned "as an unwelcome Christmas present."
The team made two moves that I think are the real story of this postmortem. First, they bought a professional support contract from the SQLite developers (their support page), which gave them direct access to the people who wrote the database. Second, they systematically mapped out theories, ruled them out with data, and refused to guess. The candidate theories included broken POSIX locks on close(), mismanaging memory owned by SQLite, and using SQLite from multiple threads while thread safety was disabled. Every incident produced more data, and every theory was eliminated one by one.
The clue: a transaction that vanished
While the root cause was still unknown, the team had a live platform to keep running. They automated hard-stops on corruption, deployed a backup monitor that continuously ran PRAGMA integrity_check over backups, and improved runbooks. That cut recovery time to under an hour. Then they built something clever: a transaction logging pipeline.
The idea was simple. Stream every SQL statement that modifies the database to a separate log file. Because SQLite is a single-writer database with serializable transactions, the transaction history is completely linear and deterministic. Replaying those transactions against the last known-good backup should restore the database to its most recent state, safely bypassing the corrupted pages. This is a technique I want to call out for Java teams, because it only works in single-writer systems. It would not hold in a multi-writer Postgres or MySQL deployment.
The pipeline worked. And then it produced the clue that cracked the case. In two incidents, the transaction logs failed to replay cleanly. Data written and committed by one transaction was inexplicably invisible to later transactions. A write had vanished into thin air without raising any error.
That should be impossible. In a serializable, single-writer database, a committed write cannot disappear. The fact that it did pointed directly at the checkpointing layer, the only component with enough concurrency to hide a lost write.
The bug: a 16-year-old race in the WAL
To understand the fix, you need the two-file model SQLite uses with Write-Ahead Logging. The database file is a series of pages. When you update data, new pages are not written directly to the database file. They go into the WAL file first, for performance and concurrency. At some point, a checkpoint copies those pages back into the main database file. In most deployments SQLite decides when to checkpoint on its own. Tailscale, however, takes manual control of the checkpoint process so it can run fast, consistent backups.
That non-standard choice matters, because it is what exposed them to the bug. During corruption incidents, their metrics showed SQLite reporting that it copied more pages from the WAL file than actually existed there. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.
The SQLite developers had been building a new debugging tool for exactly this layer: a wrapper around the virtual filesystem that writes additional tracing logs about changes to the database. It is called the tmstmpvfs shim, and the source lives in the SQLite public repository. Tailscale deployed it into production and waited. They did not have to wait long.
The logs exposed a rare data race between a checkpoint and a write transaction. If a write occurs at a specific moment during a checkpoint, the checkpoint gets confused: it thinks some pages have been copied from the WAL into the main database file, but they have not. Those pages are never written, and the data is permanently lost. Worse, other pages that reference the missing ones, such as an index, do get written. The database file becomes structurally corrupt, which is exactly what PRAGMA integrity_check was detecting all along.
The SQLite developers named it the WAL-Reset bug, and they estimate it had been in SQLite for at least 16 years. It survived that long because it was rare enough that they had to add code to deliberately trigger it in their test environment. The fix adds a check to the checkpointing function that detects when the WAL has been reset by another thread.
The false alarm that nearly derailed the fix
The fix shipped as SQLite 3.52.0, and Tailscale rolled it out carefully: a few canary shards first, then the rest of the control plane. Then their backup monitor promptly turned red, reporting corruption in 13 different databases.
This is my favorite part of the whole story, because it is a lesson in how even a correct fix can be misread. The 13 databases had not suffered real corruption. SQLite 3.52.0 also contained an optimization that subtly changed rounding behavior for text-to-floating-point conversions, and Tailscale stored high-precision timestamps as text, converting them to floating point in a VIRTUAL generated column. Stale expression indexes, where an index on a computed value no longer matches after the computation changes, get reported as corruption by PRAGMA integrity_check. The canary shards simply did not have any timestamps that triggered the changed rounding, so the phased rollout missed it.
- The SQLite developers withdrew 3.52.0 entirely and published 3.51.3 containing only the WAL-Reset fix.
- Tailscale fixed its side by storing timestamps as integer seconds instead of text, since text-to-integer conversions are unambiguous.
- SQLite added an automated self-healing index feature in 3.53.0 to prevent the stale expression index problem permanently.
Then came the most disciplined part of the investigation. An absence of corruption is not proof of a fix, because the team had already lived through one six-week period of deceptive calm. So they patched their SQLite driver to log a warning whenever a write transaction and a WAL-reset overlap. If the warning fired while the database stayed healthy, they would know the fix had saved them from a potential incident.
They waited two months. The alert finally fired, proving the precise conditions for the WAL-Reset bug do occur in their production environment. Since that alert, they have run another four months without a single database incident.
What this means for Java and Spring Boot teams
You are probably not running SQLite as your primary database, and you are certainly not checkpointing it manually. But this postmortem is not about SQLite. It is about what to do when your database insists something impossible happened. Here is the playbook I now keep in mind, mapped to the Java stack.
"Impossible" is a hypothesis, not a conclusion. The team's single-writer, serializable setup made a vanished committed write theoretically impossible, and that is exactly when they found it. For a Spring Boot team, the equivalent is the Hibernate query that returns stale data despite a committed transaction, or the Postgres row that reappears after deletion. Instrument first, assume second.
Verify that your backups actually restore. Tailscale's entire recovery pipeline depended on snapshots they could replay. They built the transaction log pipeline precisely because restoring from the last known-good backup would lose too much data. For Java services, this means restore drills, not just backup jobs. A backup that has never been restored is a hypothesis.
Put integrity checks in the pipeline, not just in incident response. They ran PRAGMA integrity_check continuously over backups, which is what caught the first corruption and every one after. The Postgres equivalents are pg_checksums, amcheck, and pg_stat views; MySQL has CHECK TABLE. A nightly job that scans for corruption turns a silent data problem into an alert.
Canary your database engine upgrades, not just your application. The 13-database false alarm happened because the canary shards lacked the data shape that triggered the rounding change. When you bump a JDBC driver, an embedded database, or a migration tool, make sure your canary environment exercises the same data shapes as production. Otherwise the canary validates nothing.
Boring technology in a non-standard way is a risk. The postmortem says it plainly: everything Tailscale did was documented and supported, but taking manual control of the checkpoint process and running at an aggressive pace took them off the well-trodden operational path. In Spring Boot terms, this is the case for not hand-rolling transaction management, connection pools, or migration frameworks when the framework already provides them. The standard path is the tested path.
Single-writer designs give you a superpower. The transaction replay pipeline only worked because SQLite's single-writer model made the history linear and deterministic. If you run a single-writer service, keep its write path strict, because it makes this kind of forensic replay possible. If you run multi-writer Postgres, plan for logical decoding or point-in-time recovery instead.
The 6-step checklist when a database says "impossible"
Here is the save-worthy part, condensed from how Tailscale actually cracked this:
- Collect the evidence before changing anything. Checksums, integrity checks, and error logs from the moment the first incident appears.
- Look for common factors across incidents, including the absence of them. No pattern is a finding, not a dead end.
- Rule out theories with data, one at a time. Write down every candidate, and find the experiment that eliminates one.
- Add passive telemetry into production. You cannot debug what you cannot observe, and reproduction may not be possible.
- Buy expertise when the bug outlasts your team's assumptions. A support contract with the core maintainers was the fastest path to the shim that found the race.
- Prove the fix with a positive signal, not just the absence of errors. They waited for a warning that proved the race conditions still occur, safely.
The full postmortem is worth a read on its own, especially the Hacker News thread, where Simon Willison picks up on an often-missed detail: the whole investigation funded the SQLite VFS shim, an open source debugging tool that will help track down similar bugs in any future database that uses it. Six months of pain, a 16-year-old bug, and the fix is now baked into every SQLite user's upgrade path.
I write about Java, Spring Boot, and AI every week. Subscribe, it's free.
Have you ever debugged a database issue that looked impossible at first? What was the clue that actually cracked it? I would love to hear how you found it.
