In June our security team asked for the bucket holding signed contracts to be reachable only from the office and from our own network. I wrote the bucket policy in Terraform: deny every S3 action unless the request's aws:SourceIp is one of the office ranges or one of the two public addresses of our NAT gateways. I tested it from my laptop in the office and from an instance in a private subnet, and both worked. The plan was reviewed and the apply was green on a Wednesday.
On Friday morning the contracts service failed every upload and every download with access denied, from an explicit deny.
On Thursday evening the network team had added a gateway endpoint for S3 to the VPC, to stop paying NAT charges for traffic that never needed to leave AWS. It is a good change. It also means requests to S3 from inside the VPC no longer leave through the NAT. They travel through the endpoint, and aws:SourceIp only ever holds a public address, so none of our servers' requests carried an address on my list. The condition stopped matching for every one of them, and the deny applied to all of them.
Undoing it was harder than it should have been. The deny covered s3:*, which includes changing the bucket policy, and our CI runners sit in the same VPC. Terraform could not remove the policy it had written. We used the account's root user, which AWS allows to delete a bucket policy that has locked everyone else out, and it took forty minutes to find the person who held its second factor.
The policy now admits the VPC through aws:SourceVpce, naming the endpoint, and the office through aws:SourceIp, and the deny applies only when a request matches neither. A break glass role is excluded by aws:PrincipalArn, so we never need root for this again. Any policy change containing a network condition is run in the pipeline through the IAM policy simulator, with the endpoint and the office as request contexts. And the endpoint, the route tables and the policies that depend on them now live in one module, so a network change shows up in the same plan as everything it affects.
A condition on where a request comes from is a condition on the network as it is drawn today. Ours was redrawn the next day.
– Sergey Shinder
