We bought two internet circuits from two different carriers, at some expense, specifically so that one failure could not take the site off the network. An excavator working on a drainage scheme four hundred metres away cut both of them at eleven in the morning, and we were dark for five hours.
The contracts both said diverse routing. They were not lying, in the sense that the diversity was real somewhere further up. The last mile was not. Both fibres entered our building through the same duct, crossed the same field, and terminated in the same exchange three miles away. Carrier B, it turned out under questioning, was reselling capacity on carrier A's physical infrastructure for the segment that mattered, which is an entirely normal commercial arrangement that nobody had thought to ask about because the logos on the two invoices were different.
Then we looked at the rest of it, which was worse. The link to our cloud environment and the remote access concentrator both sat behind the same pair of circuits. The disaster recovery site's connectivity had been ordered by the same account manager at the same time, from the same carrier, and followed a route nobody had ever requested a diagram for. And the building has one point of entry for communications because that is how it was built in 1998.
What I would tell anyone buying resilience at this layer is that diversity is a property of the ground, not of the contract. The questions that matter are physical and slightly tedious. Which duct, which entry point, which exchange, which power feed, and who actually owns the fibre rather than who sends the invoice. Carriers will answer these, in writing, if you ask before you sign.
We now hold a register of circuits with their routes and their real underlying providers, refreshed annually. A second entry point into the building was expensive and is done. Card payments and telephony fail over to a wireless service, which is slower and uses a different medium entirely, which is the whole point. And the recovery test includes physically unplugging a circuit, because everything else we were told about it turned out to be a description rather than a fact.
– Serguey Shinder
