A project team asked us in September to restore a folder they had deleted in July. It held about six weeks of design material for work that had since gone out to tender, and the reason nobody raised it sooner is that the people who used it every day were the same people who removed it.
We could not get it back. The retention our supplier offers on that service is a short window measured from the moment of deletion, and it had closed in early August. There is no tape, no snapshot and no export, because the platform is theirs and we had never arranged one.
The interesting part was not the loss, which cost some rework and no money. It was that three people in my own department believed that data was backed up, and each had a sensible reason for believing it. The infrastructure team backs up everything in our data centre, rigorously, and this was not in our data centre. The service contract talks about resilience, availability and replication between sites, every word of it accurate, and none of it any defence against a deletion that is copied everywhere faithfully in under a second. And the supplier documents a shared responsibility model in which platform availability belongs to them and tenant contents belong to us, stated plainly, on a page nobody involved in the purchase had ever opened.
We audited the rest of the estate. Eleven of our seventeen significant cloud services had no recoverable copy outside the supplier, including the one holding every signed customer contract. Two offered an export that had never once been tested. One had no bulk extraction mechanism at all, which is a fact you want well before a renewal rather than during a disagreement.
Each service now carries a line recording who holds a recoverable copy, how far back it reaches, how the restore is tested and who last tested it. Where the honest answer was that no one did, we either bought third-party backup or a director signed a note accepting that we would carry the loss.
Replication defends you against a disk failing. It reproduces a mistake with exactly the same diligence, and considerably faster.
– Serguey Shinder
