At nine on a Tuesday the platform team began a planned upgrade of our shared development cluster. It was announced twice, it touches nothing in production, and by twenty past nine nobody in the company could merge a pull request. That lasted until half one in the afternoon.
The cause was a helper in our platform test library called validToken. Four years ago somebody needed a JWT that our services would actually accept in a test, and the quickest way to get one was to ask the development identity provider for it. A real client credentials call, over the network, in a method named as though it were a constant. Twenty three test classes used it directly and another eighteen inherited it from a base class, and all of them ran in the job named unit, which is the required status check on every repository that depends on the library.
It had never been noticed because it was fast and almost always worked. The call took about sixty milliseconds and the token was cached for the run. It also explained something we had been living with for a year: a scatter of failures around two in the morning, which is when that cluster redeploys itself, and which we had all learned to describe as the dev cluster being odd and to start again by hand.
I put an outbound connection guard into the test harness and ran the whole suite. Forty one tests reached the network, to five distinct hosts: the identity provider, an internal feature flag service, an artefact store, a partner endpoint and, in one case, a public service used to check a timezone database was current.
Tokens are now signed locally with a keypair the services trust under the test profile. The guard is on by default, and a test that opens a socket fails with the host and the calling frame printed. The handful of checks that genuinely need a live dependency moved to a nightly job that reports and blocks nothing.
A required check is a dependency for every engineer in the building. We had an inventory of what our services need in order to run and nothing anywhere describing what our build needs in order to pass.
– Sergey Shinder
