In August we started building a fourth region, for disaster recovery, out of the same modules that build the three we run. Forty one minutes into the first apply it stopped: the maximum number of addresses has been reached. A second attempt failed on a limit for VPCs, a third on vCPUs for the instance family our nodes use. Nine days later the region existed. None of those nine days involved writing any code.
Every account has quotas, per region, and almost all of ours had been raised at some point by a human being submitting a form. I went back through four years of support cases and counted fourteen increases against our main production account: addresses, VPCs, interfaces per instance, rules per security group, two separate vCPU families, a quota on the number of parameters you may store. Each had been requested during a growth push or in the middle of an incident by whoever hit the wall, approved somewhere between four minutes and six days later, and then forgotten, because once a limit is high enough it never speaks to you again.
So the environment we described as fully reproducible was reproducible from our repository plus fourteen support cases nobody had written down. The code was right. The account was new.
There is a preflight job now, and it runs before plan. It reads the quota API for every limit we know we depend on, counts what the configuration is about to need, and prints a table of limit, current value, required value and the request to submit, failing the run before anything at all is created. There is also a small module that records, per account and region, every quota we rely on with the value we need, so the list is reviewed like code rather than remembered. A weekly job reports usage against each of those limits and opens a ticket at eighty percent, which caught an interface quota in our busiest region three weeks later. And the recovery rehearsal is quarterly now, in a fresh account, because a clean account is the only thing that reliably finds the next one.
Infrastructure as code describes resources. It does not describe permission to create them, and the part of your environment that only another organisation can grant you is nowhere in the repository you believe is complete.
– Sergey Shinder
