Terraform State: Remove, Move and Migrate Resources, and Set Up a Remote Backend

Terraform state is the file that maps every resource in your configuration to a real object somewhere: an instance ID, a bucket name, a DNS record. Most days you never look at it. Then someone renames a resource, splits a repository, or needs Terraform to let go of a database without deleting it, and suddenly the state file is the only thing that matters.

This post covers the state operations that come up again and again: removing a resource from state, renaming or moving it, carrying it to another project, bootstrapping a remote backend, and the DynamoDB error people hit while building a lock table. Each one has an old way (a CLI command that edits state directly) and, in recent Terraform versions, a new way (a block in your configuration that goes through plan like any other change). The new way is almost always better, and the sections below show why.

All the terminal output in this post comes from real runs on Terraform 1.15.8, using the built-in terraform_data resource so the examples run anywhere without a cloud account.

TLDR

• To stop managing a resource without destroying it, use a removed block with destroy = false (Terraform 1.7+). terraform state rm does the same thing but skips plan, and it will recreate the resource if you forget to delete it from the config.
• To rename a resource, use a moved block (Terraform 1.1+). It shows up in plan and gets code review. terraform state mv still works for one-off fixes.
• To move a resource to another project, remove it from the old one with a removed block and adopt it in the new one with an import block. Editing state files by hand is the fallback.
• To bootstrap a remote backend, create the bucket with local state first, then add the backend block and run terraform init -migrate-state.
• The S3 backend can lock with a lock file in the bucket (use_lockfile = true, added in Terraform 1.10). DynamoDB locking has been deprecated since 1.11. Lock files also work on S3-compatible storage like DigitalOcean Spaces.
• Never commit .tfstate to Git. Do commit .terraform.lock.hcl.

Prerequisites

• Terraform 1.7 or later for removed blocks, and 1.11 or later for the S3 lock-file examples (the feature arrived in 1.10 as experimental)
• A configuration with existing state to practice on, or the Terraform terminal simulator if you want to try terraform state list in the browser first
• Access to an object storage bucket (AWS S3 or DigitalOcean Spaces) for the backend sections

What state actually tracks

1. Configuration what you want
2. State what Terraform thinks exists
3. Real infrastructure what actually exists

By default, every plan compares three things: your configuration, the state file, and the real objects the provider can see. State is the link in the middle. It is keyed by resource address (aws_instance.web, module.network.aws_vpc.main), so almost every problem in this post comes down to one of two questions: which address points at which real object, and which state file holds that address.

You can see the addresses in your state at any time:

terraform state list

$ terraform state list
terraform_data.db
terraform_data.web

Stop managing a resource without deleting it

The situation: a database, a DNS zone or a bucket was created by Terraform, and now it should live outside this configuration. Maybe another team owns it, maybe you are splitting a repository. You want Terraform to forget it, not destroy it.

The old way: terraform state rm

state rm, then plan

$ terraform state rm terraform_data.db
Removed terraform_data.db
Successfully removed 1 resource instance(s).
$ terraform plan
...
Plan: 1 to add, 0 to change, 0 to destroy.

That second command is the trap. state rm removed the resource from state, but the resource "terraform_data" "db" block is still in the configuration, so the next plan wants to create it again. On a real database that means a second database, or a name collision. state rm only works safely when you delete the block from the configuration in the same change, and nothing in the workflow reminds you to do that.

It also skips plan entirely. The change happens the moment you press enter, it never shows up in a pull request, and in CI there is nothing to review.

The new way: a removed block

Delete the resource block and put a removed block in its place:

removed {
  from = terraform_data.db

  lifecycle {
    destroy = false # forget it, do not delete it
  }
}

Now the removal is a normal change that goes through plan:

plan with a removed block

$ terraform plan
Terraform will perform the following actions:
  # terraform_data.db will no longer be managed by Terraform, but will not be destroyed
  # (destroy = false is set in the configuration)
  . resource "terraform_data" "db" {
        id = "a38fe64a-ecd1-489f-f733-8048076db5f0"
        # (2 unchanged attributes hidden)
    }
Plan: 0 to add, 0 to change, 0 to destroy.

Warning: Some objects will no longer be managed by Terraform
$ terraform apply -auto-approve
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
$ terraform state list
terraform_data.web

The plan says exactly what will happen, your reviewer sees it, and there is no window where the configuration and the state disagree. Once the change is applied everywhere, you can delete the removed block. If you leave destroy out (it defaults to true), the block becomes a way to destroy a resource on purpose, which is also useful, just not here.

Tip

Before any state change, take a copy: terraform state pull > backup.tfstate. It costs nothing and gives you a way back. Restoring it after another change is not a plain push: in our run terraform state push backup.tfstate refused with "cannot import state with serial 3 over newer state with serial 4". terraform state push -force backup.tfstate overwrites the current state and skips both the serial and the lineage checks, so treat it as an exception: take a fresh backup of the current state first, and remember it restores Terraform's records, not the infrastructure.

Rename or move a resource inside a project

Renaming aws_instance.web to aws_instance.frontend looks harmless in the code. Terraform sees it differently: one address disappeared and a new one appeared, so the plan destroys the old instance and creates a new one. For anything with data on it, that is an outage.

The new way first: a moved block

resource "terraform_data" "frontend" {
  input = "web-server"
}

moved {
  from = terraform_data.web
  to = terraform_data.frontend
}

plan with a moved block

$ terraform plan
Terraform will perform the following actions:
  # terraform_data.web has moved to terraform_data.frontend
    resource "terraform_data" "frontend" {
        id = "0f0915bf-cbea-ff5f-1129-a346d2268e84"
        # (2 unchanged attributes hidden)
    }
Plan: 0 to add, 0 to change, 0 to destroy.

No destroy, no create, just a move that anyone can read in the pull request. moved blocks also handle the moves that are painful by hand: pulling resources into a module (from = aws_s3_bucket.logs, to = module.logging.aws_s3_bucket.this), or switching from count to for_each (from = aws_instance.web[0], to = aws_instance.web["primary"]).

A moved block is cheap to keep. In a reusable module, keep it for good: consumers can skip versions, and a removed moved block turns their upgrade into a destroy and recreate. In a root configuration you can delete it once every state that uses the configuration has applied the move.

The old way: terraform state mv

terraform state mv

$ terraform state mv terraform_data.frontend terraform_data.web
Move "terraform_data.frontend" to "terraform_data.web"
Successfully moved 1 object(s).
$ terraform plan
No changes. Your infrastructure matches the configuration.

Here we undid the rename from the previous section: after applying the moved block, we renamed the block back to web, deleted the moved block, and moved the state to match. It works, and for a quick fix on your own sandbox it is fine. The problem is the same as state rm: it changes shared state immediately, outside review, and the code change that goes with it has to land separately. In a team, prefer moved. Both state rm and state mv accept -dry-run if you want to see what they would touch first.

Move resources to another project

Splitting a large configuration into smaller ones (network in one project, applications in another) means carrying resources from one state file to a different one. There are two ways to do it.

The reviewable way: removed plus import

In the source project, delete the resource and let go of it:

removed {
  from = aws_instance.web

  lifecycle {
    destroy = false
  }
}

In the target project, add the resource block and adopt the existing object with an import block (Terraform 1.5+):

import {
  to = aws_instance.web
  id = "i-0a1b2c3d4e5f67890" # the real instance ID
}

resource "aws_instance" "web" {
  ami = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.small"
}

Apply the source first, so the object is only ever managed by one project, then the target. Make sure nobody runs either project in between. The target's plan shows 1 to import, and if your resource block does not match the real object, the plan shows the differences before anything changes. If you would rather not write the resource block by hand, leave it out, keep only the import block, and run terraform plan -generate-config-out=generated.tf (to a file that does not exist yet): Terraform writes a starting block from the real object. Both changes go through plan and review, and at no point does anyone edit a state file.

The fallback: move between state files directly

Some resources cannot be imported, and sometimes you need to move dozens at once. terraform state mv can write to a different state file. It only moves state, so move the configuration in the same change: delete the resource block from the source, add it to the target, and fix any references. With a remote backend, pull both states to local files, move the resource, and push them back:

# in the source project
terraform state pull > source.tfstate

# in the target project
terraform state pull > target.tfstate
terraform state mv -state=../source/source.tfstate -state-out=target.tfstate \
  aws_instance.web aws_instance.web
terraform state push target.tfstate

# back in the source project
terraform state push source.tfstate

Here is the core of it on two local projects:

move a resource to another project

$ terraform state mv -state=../source.tfstate -state-out=../network/terraform.tfstate terraform_data.web terraform_data.web
Move "terraform_data.web" to "terraform_data.web"
Successfully moved 1 object(s).
# in the target project, which now holds the resource
$ terraform state list
terraform_data.web
$ terraform plan
No changes. Your infrastructure matches the configuration.

In this run the target configuration already contained the terraform_data.web block, which is why its plan shows no changes. Nobody else should run Terraform against either project while you do this, and you should keep the backups from the tip above. Like state rm and state mv inside one project, it changes state without a plan, so check both plans right after.

Set up a remote backend, with Terraform itself

State in a local terraform.tfstate file works for one person on one laptop. The moment a second person or a CI job runs Terraform, you need a remote backend with locking turned on for every writer, so two applies cannot write the same state at once. Not every backend locks, and on S3 it is opt-in, so check that yours does.

The classic chicken-and-egg problem: you want Terraform to create the bucket that will hold Terraform's state. The answer is two steps.

Step 1: create the bucket with local state. A small bootstrap configuration, applied once:

Bootstrap the state bucket

AWS S3

resource "aws_s3_bucket" "state" {
  bucket = "acme-terraform-state"
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration {
    status = "Enabled" # every state write becomes a recoverable version
  }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket = aws_s3_bucket.state.id
  block_public_acls = true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}

DigitalOcean Spaces

resource "digitalocean_spaces_bucket" "state" {
  name = "acme-terraform-state"
  region = "fra1"
  acl = "private"
}

Step 2: point the configuration at the bucket and migrate. Add a backend block:

Backend block

AWS S3

terraform {
  required_version = ">= 1.11"

  backend "s3" {
    bucket = "acme-terraform-state"
    key = "prod/network.tfstate"
    region = "us-east-1"
    encrypt = true
    use_lockfile = true # lock with a .tflock object in the bucket, no DynamoDB
  }
}

DigitalOcean Spaces

terraform {
  required_version = ">= 1.11"

  backend "s3" {
    endpoints = {
      s3 = "https://fra1.digitaloceanspaces.com"
    }
    bucket = "acme-terraform-state"
    key = "prod/network.tfstate"

    # Spaces speaks the S3 API; these skip AWS-only checks
    skip_credentials_validation = true
    skip_requesting_account_id = true
    skip_metadata_api_check = true
    skip_region_validation = true
    skip_s3_checksum = true
    region = "us-east-1" # required by the backend, not used by Spaces

    use_lockfile = true
  }
}

Then run terraform init -migrate-state. Terraform notices the backend changed, asks whether to copy the existing state to the new backend, and from then on reads and writes it remotely. The same command handles any backend change later: a new bucket, a new key, or moving from one provider to another. Here it is moving local state to a new location:

terraform init -migrate-state

$ terraform init -migrate-state -force-copy
Initializing the backend...
Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.
...
Terraform has been successfully initialized!

-force-copy answers yes to the copy prompt, which is what you want in a script. Run it without the flag the first time so you see the question.

For Spaces, the credentials are a Spaces access key and secret, passed as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment, not written in the backend block. DigitalOcean documents the full setup in Configure DigitalOcean Spaces as a Terraform Remote State Backend, including state locking with use_lockfile on Terraform 1.11 or later. If your state also holds secrets (it usually does), keep the bucket private and limit who has keys to it; that is the whole point of the next two sections.

For the errors people usually hit on this step, from the wrong region to missing permissions, see common S3 backend configuration errors, and if a lock gets stuck, how to unlock a locked state file.

The DynamoDB lock table is on its way out

For years, locking on the S3 backend meant a separate DynamoDB table with a LockID key. Terraform 1.10 added locking through S3 itself: at the start of an operation Terraform writes a .tflock object next to the state with a conditional write that fails if the object already exists, so a second apply is blocked. With use_lockfile = true you no longer need the table, and Terraform now says so when it sees the old setting:

terraform init with dynamodb_table

$ terraform init
Initializing the backend...
Warning: Deprecated Parameter
  on main.tf line 6, in terraform:
   6: dynamodb_table = "terraform-locks"
The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
instead.

If you are migrating an existing backend, you can set both use_lockfile = true and dynamodb_table for a while. A client with both settings takes both locks, so it excludes old clients that only know DynamoDB and new ones that only use the lock file. Retire the table only when every writer (people and CI jobs) has use_lockfile enabled, has permission to create and delete the .tflock object, and nothing depends on DynamoDB any more.

The "all attributes must be indexed" error

If you still build a DynamoDB table in Terraform, for locking or anything else, you will probably meet this error from the AWS provider sooner or later:

Error: all attributes must be indexed. Unused attributes: ["category"]

The message sounds like a query rule, but it is about the attribute blocks. In aws_dynamodb_table, an attribute block does not describe the item's fields. It declares the type of a key : the table's hash_key or range_key, or a key of a global or local secondary index. DynamoDB is schemaless for every other field, so an attribute that no key uses is an error.

The wrong way, declaring fields as if it were a SQL table:

resource "aws_dynamodb_table" "orders" {
  name = "orders"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "id"

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "category" # no key uses this: "all attributes must be indexed"
    type = "S"
  }
}

The right way: declare only key attributes. If you do need to query by category, make it a key of an index, and then its attribute block is valid:

resource "aws_dynamodb_table" "orders" {
  name = "orders"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "id"

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "category"
    type = "S"
  }

  global_secondary_index {
    name = "by-category"
    hash_key = "category"
    projection_type = "ALL"
  }
}

For a lock table, the rule is short: one attribute, LockID of type S, as the hash key, and nothing else.

Should .tfstate go in Git?

No, for three reasons that each matter on their own:

1. State can hold secrets in plain text. Database passwords, generated keys, and values marked sensitive are redacted in plan output, but sensitive does not keep them out of state. (Newer ephemeral values and write-only arguments do, where a provider supports them.) A state file in Git is a credential in Git, forever, in every clone and every fork.
2. Git cannot lock. Two people who run apply from their own checkouts each write a different state. The next merge picks one, and Terraform loses track of whatever the other one created.
3. State changes whenever an apply changes something. Committing it makes every infrastructure change a merge conflict waiting to happen.

What belongs where:

# .gitignore
*.tfstate
*.tfstate.*
.terraform/
crash.log
# saved plans can contain sensitive values: save them as *.tfplan
*.tfplan
# only if your variable files hold secrets; commit a non-secret example instead
*.tfvars
*.tfvars.json

Commit .terraform.lock.hcl, though. It pins the exact provider versions and checksums, so everyone and every CI run use the same provider build. And if state already made it into your history, rotating the secrets in it matters more than rewriting the history, because every existing clone still has the old file.

Where state should live is a bigger question than a backend block: who is allowed to run apply, and whose laptop has the keys. We wrote about that in Who owns the state file.

Summary

The pattern behind all of it: prefer changes that go through plan. The config blocks (removed, moved, import) turn state surgery into reviewable code, and the CLI commands are there for the rare case where that is not possible. If you want to go further with the language itself, Terraform variables, loops and outputs covers the rest, and the Terraform terminal simulator lets you practice init, plan, apply and state list in the browser.

Originally published at devops-daily.com.