By 08:47 the terraform plan output was up on the shared screen and nobody wanted to be the one to type apply. The summary line read 'Plan: 5 to add, 0 to change, 5 to destroy.' Every one of the 5 destroys was a resource actively serving production. The Aurora cluster. The RDS proxy in front of it. Three security groups. The failover had happened at 02:14 that morning. SRE had brought the us-west-2 replacements up by hand while the primary region recovered. Six hours later, terraform did not know they existed.
Problem signals:
- terraform plan shows -/+ replace on resources actively serving production traffic
- The plan wants to destroy resources that were created out-of-band during a recent incident
- aws describe commands confirm the resources exist and are healthy, but terraform state does not know about them
- The team has stopped running terraform plan against this workspace because the output is too scary to act on
Five destroy-and-replace actions against live production
The plan wanted to destroy the primary database
The workspace was the one that owned the data layer for the primary application. Its state held the pre-failover Aurora cluster, the RDS proxy, and three security groups, all with us-east-1 ARNs. During the incident someone had updated the module's provider region from a variable to a hardcoded 'us-west-2', trying to bring the manually-created resources under management. That change alone did not import anything. What it did was make terraform want to replace every resource in the module, because state said us-east-1 and config now said us-west-2. Five resources in the module, five replacements, five destroys and five creates in the summary.
$ terraform plan
Terraform will perform the following actions:
# module.data_layer.aws_rds_cluster.primary must be replaced
-/+ resource "aws_rds_cluster" "primary" {
~ arn = "arn:aws:rds:us-east-1:...:cluster:app-primary" -> (known after apply)
~ endpoint = "app-primary.cluster-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
~ id = "app-primary" -> (known after apply)
# (27 unchanged attributes hidden)
}
# module.data_layer.aws_db_proxy.primary must be replaced
-/+ resource "aws_db_proxy" "primary" {
~ arn = "arn:aws:rds:us-east-1:...:db-proxy:prx-0a1b2c3d4e5f60718" -> (known after apply)
~ endpoint = "app-primary-proxy.proxy-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
# (14 unchanged attributes hidden)
}
# module.data_layer.aws_security_group.db_ingress[0] must be replaced
-/+ resource "aws_security_group" "db_ingress" {
~ arn = "arn:aws:ec2:us-east-1:...:security-group/sg-0abcdef1234567890" -> (known after apply)
~ id = "sg-0abcdef1234567890" -> (known after apply)
}
# module.data_layer.aws_security_group.db_ingress[1] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }
# module.data_layer.aws_security_group.db_ingress[2] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }
Plan: 5 to add, 0 to change, 5 to destroy.
The plan that stopped the room at 08:47
The apply would have done exactly what the plan said. It would have called DeleteDBCluster on the us-east-1 cluster, which was still up and still holding connections from the pools that had not been cut over yet. It would have called CreateDBCluster in us-west-2 with the identifier 'app-primary', which was already the identifier on the live cluster that SRE had created at 02:14. That second call would have returned DBClusterAlreadyExistsFault, and the state would have been mid-transaction, with the us-east-1 cluster gone and the us-west-2 cluster still not tracked. The same shape of failure was queued up for the proxy and the security groups, each with its own AWS-side name-collision error.
Why terraform state rm looked right and was wrong
What we almost did that would have made it worse
The reflex in the room was to run terraform state rm module.data_layer.aws_rds_cluster.primary on all 5 entries, then re-plan. State would forget the us-east-1 resources, plan would show 5 creates in us-west-2, and terraform would be back in a consistent-looking place. We had the commands typed. We did not run them, because someone asked the question that saved the afternoon: what does the create actually do when the resource already exists?
The answer is: it fails, and it fails deterministically. CreateDBCluster refuses an identifier that already exists in the region and returns DBClusterAlreadyExistsFault whether or not the properties you are asking for match the live resource. There is no adopt-if-identical path in the AWS API. CreateDBProxy behaves the same way, and duplicate security group names inside a VPC come back as InvalidGroup.Duplicate. Worse, a replace is a destroy followed by a create, so the destroy half lands first: the us-east-1 cluster is gone, the create fails, and state holds five addresses whose real resources terraform just deleted and could not recreate. Sorting that out is a bad afternoon and a worse changelog entry.
The right question is not 'how do we make terraform stop wanting to destroy things'. It is 'how do we get the resources that ARE running into state under terraform management, with the state entries reflecting reality'. Different question, different answer. state rm alone does not answer it, because the plan step after state rm still wants to create. Import blocks answer it, because they tell terraform 'this thing already exists, adopt it, do not create it'.
The Terraform 1.5 import block
Import blocks over state manipulation
Terraform 1.5 shipped import {} blocks in June 2023. Before that, the only tool was terraform import at the CLI, one resource at a time, with no config generation. You had to write the resource block by hand, run the import, then plan until it was empty. For 5 resources that is a slow afternoon. For 30, which is the size of the recovery we have done more often after a full-region failover, it is a week and a half.
The block form lives in HCL and terraform reads it during plan. Our module already declared all 5 resources (the only thing that had changed was the provider region), so there was nothing to generate: the plan matched each import block to the resource block already in the module and staged the adoption for the next apply. -generate-config-out is the companion flag for the other case, when you are adopting something you have no configuration for at all, and it writes a starter resource block for each import block that has no matching config.
# imports.tf (temporary, delete after one apply cycle)
import {
to = module.data_layer.aws_rds_cluster.primary
id = "app-primary"
}
import {
to = module.data_layer.aws_db_proxy.primary
id = "app-primary-proxy"
}
import {
to = module.data_layer.aws_security_group.db_ingress[0]
id = "sg-0a1b2c3d4e5f60718"
}
import {
to = module.data_layer.aws_security_group.db_ingress[1]
id = "sg-1a2b3c4d5e6f70819"
}
import {
to = module.data_layer.aws_security_group.db_ingress[2]
id = "sg-2b3c4d5e6f708192a"
}
One import block per resource; the id is the AWS resource ID, the to is the terraform module address
There is a state cleanup step first. The us-east-1 entries at those module addresses have to go, otherwise the import collides with an existing state entry and errors with 'resource already managed by Terraform'. We ran terraform state rm on the 5 stale entries. This is safe here because those state entries no longer match anything we intend to manage from this workspace (the us-east-1 resources are being handled by a separate decommission workspace). state rm is only dangerous when the plan step that follows tries to create things. The plan step that follows here only reads: import blocks are resolved during plan, and nothing touches AWS until apply.
$ terraform plan
Terraform will perform the following actions:
# module.data_layer.aws_rds_cluster.primary will be imported
resource "aws_rds_cluster" "primary" {
cluster_identifier = "app-primary"
engine = "aurora-postgresql"
engine_version = "15.3"
# ... 24 more attributes read from AWS
}
# module.data_layer.aws_db_proxy.primary will be imported
# module.data_layer.aws_security_group.db_ingress[0] will be imported
# module.data_layer.aws_security_group.db_ingress[1] will be imported
# module.data_layer.aws_security_group.db_ingress[2] will be imported
Plan: 5 to import, 0 to add, 0 to change, 0 to destroy.
The successful plan after state rm + import blocks: 5 imports, 0 creates, 0 destroys
Read the summary line: 5 to import, 0 to add, 0 to change, 0 to destroy. That is the shape you want, and any add or destroy in it means an import block is pointing at the wrong address, or the module's attributes have drifted from what SRE actually built at 02:14. Fix that before you apply, not after. Commit imports.tf. Apply. The apply runs the 5 imports. After that, delete the import {} blocks. Once the address is in state they are no-ops on every subsequent plan, so leaving them in place is not an error, it is just noise that outlives the reason it was added.
How to align config to reality without overwriting the live resource
The generated HCL is 95% right and the last 5% matters
The generated config reflects what is running. That includes anything SRE added at 02:14 that was not in the pre-incident module. On this recovery, the next plan showed three drifts, all small, all load-bearing.
$ terraform plan
Terraform will perform the following actions:
# module.data_layer.aws_rds_cluster.primary will be updated in-place
~ resource "aws_rds_cluster" "primary" {
~ backup_retention_period = 1 -> 7
~ tags = {
- "incident-response" = "2024-11-14" -> null
- "incident-owner" = "sre-oncall" -> null
}
}
# module.data_layer.aws_db_proxy.primary will be updated in-place
~ resource "aws_db_proxy" "primary" {
~ debug_logging = true -> false
}
Plan: 0 to add, 2 to change, 0 to destroy.
The next plan after import: the live state differs from the pre-incident module in three specific places
Two tags SRE had added at create time so the resources would be enumerable later. backup_retention_period at 1 day (the AWS default, because the incident timer was ticking and SRE clicked through the console fast) instead of the module's 7. debug_logging = true on the proxy, because SRE wanted verbose logs during the incident. Every one of these was a live state difference from what the module said the resource should look like.
For each drift item the decision is: keep the live state (add to config) or accept the plan (module wins)? On this recovery we kept the two tags for a followup and added them to the module's default tag map. We bumped backup_retention_period in the module to 7 to match the module's intent, accepting the plan on that one. We kept debug_logging=true for one more week and let the plan sit non-empty on that single field until the incident postmortem was done, then flipped it. The tempting move is to apply the whole plan at once and let the module reset everything. Do not do that without walking through each item. The point of the import was to bring the live resource under management, not to overwrite it with pre-incident config that no longer applies.
The other lesson from this recovery went into the incident runbook. When SRE creates resources by hand at 02:14, they now write the resource IDs into incident-resources.txt at the same time and commit it before the incident closes. That file becomes the source of truth for the import blocks the next morning, and cuts the recovery from 'grep CloudTrail for what got created between 02:00 and 03:00' to 'read the file, generate the imports'.
The 30-minute triage for a plan you cannot safely apply
If your terraform plan looks like this right now
The mechanical part of this recovery, the state rm and the import blocks, is a few hours of work. The hard part is the days before that, when the plan sits red and every engineer is afraid to touch it. Meanwhile someone else adds another out-of-band resource, and the next plan gets 12 lines longer, and the fear compounds. We have seen workspaces sit in this state for six months. The team stops running plan against them at all, which is its own kind of terrifying, because now nobody knows what actually differs from IaC.
We run these engagements every week. The failover-import case we have done three times this quarter alone, plus the 'someone tagged everything in the console during audit prep' variant, plus a half-dozen other shapes of out-of-band drift. Every one follows the same pattern: get the running resources into terraform without touching production, then decide what to reconcile and in what order. The reason it works is that the mechanical steps and the judgment steps are kept separate, so nobody applies the plan at 08:47 to make the red output go away.
If your workspace has a plan you cannot apply and no one is sure how it got there, book an infrastructure review and we will be on a bridge with you the same day, starting with a 30-minute diagnostic call to figure out whether the safe recovery is import blocks, moved blocks, a state file surgery, or a fresh workspace. For the class of problem this fits into, see the terraform apply fear pattern and the terraform state recovery playbook.
Originally published at https://infraforge.agency/insights/terraform-plan-destroy-live-failover-resources/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
