Migrating Terraform State Between Backends: A Simple Guide

Migrating Terraform State Between Backends: A Simple Guide

Managing your Terraform state files is an important part of working with Infrastructure as Code tools like Terraform and Pulumi. These state files track your cloud resources, helping Terraform understand what exists, what needs to change, and how to manage your infrastructure. But as you progress, there might be situations where you need to switch your backend. For example, once you’re done working with your state file locally, you’ll often need to migrate it to a cloud provider like AWS S3 bucket or Google Cloud Storage to enable better collaboration between your teammates and make sure your infrastructure is managed centrally for the whole organisation.

In this blog, we’ll take a look at the scenarios where you might need to migrate your Terraform state, how you can migrate your Terraform state between different backends with real-world examples, and the best practices you should follow when doing so. You can consider this your ultimate guide to managing state migrations effectively.

Scenarios where you are required to migrate your state

Before we learn how to migrate your Terraform state, let’s take a look at what are the scenarios where you might need to migrate your state from one backend to another:

• Transitioning from Local to Remote State: When starting with Terraform, the state file is often stored on your local machine. But as more team members start collaborating on the same infrastructure, it’s important to migrate that state file to a remote backend. Remote backends (such as AWS S3, Google Cloud Storage, or Azure Blob Storage) make it easier for teammates to collaborate, allowing all team members to access and update the infrastructure efficiently.
• Switching Cloud Providers: You may want to switch the remote backend where your Terraform state is stored to another cloud provider, such as moving from AWS S3 to GCP Google Cloud Storage or Azure Blob Storage. This could be for reasons like cost savings or specific features your company needs. Note that Terraform can still manage resources in different providers, even if your state file is stored in a different cloud provider. Migrating the Terraform state file ensures that Terraform continues to manage your resources across providers without losing any track of the existing infrastructure.
• Handling Regional Failures: Sometimes things can go wrong within your infrastructure, such as an AWS S3 bucket that you were using as a remote backend for your Terraform project becoming unavailable due to a regional failure, which makes your Terraform state file inaccessible to your team. In such cases, you need a backup plan to keep everything running. Setting up cross-region replication for your S3 bucket can ensure that your state file is replicated in real-time to another region, providing a more immediate failover. Additionally, backing up your state file weekly or monthly to another S3 bucket or a different storage service will further help restore your state if needed. This way, even if one region is completely down, you still have a copy ready to use, making sure that your resources stay available and manageable for Terraform, even during such outages.
• Environment Cloning: When you’re working on testing or staging environments, you often need to ensure that your Terraform code accurately replicates your production setup. The code itself should be used to replicate environments, not the state file, which is used to track the live infrastructure. For example, if you’re preparing for a high-traffic event like a product launch, where you expect many users at the same time, you want your test environment to behave just like the production one. By ensuring that your Terraform code can provision the same infrastructure across environments, you can validate changes without affecting the live system. This approach keeps your production environment intact while ensuring testing or staging mirrors the live environment as closely as possible.

How Can You Migrate Terraform State Between Backends

Migrating your Terraform state between different remote backend stores, like S3 or Google Cloud Storage, needs to be done carefully to make sure nothing goes wrong, like losing track of the resources within your infrastructure. Just follow these steps to easily move your Terraform state from one backend to another.

Step 1: Understanding the Current and Target Backend

Before you start with your migration process, it’s crucial to know all the details of both your current and the target backend. The current backend is where your Terraform state file is stored right now, and the target backend is where you want to migrate it. This means taking a close look at configuration details like bucket names, keys (paths), regions, and any other settings that your backend might require. This helps makes sure a smooth transition without missing any configurations.

• Identify Resources in the Current State: Use the command terraform state list to check what resources are being tracked within your current state. This will show a list of all resources currently managed by Terraform in your current state file.

• Check Backend Configuration Details: Do a review of your existing main.tf or backend.tf file to understand more about your current backend settings. This file contains all the information about where your Terraform state is currently stored as a backend. Here’s an example configuration for an AWS S3 backend:

terraform {
  backend "s3" {
    bucket         = "infra-prod-01"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "infra-state-lock-01"
    encrypt        = true
  }
}

In this code:

• Bucket: This is the S3 bucket where your Terraform state file is currently stored. It acts as the main container for your state data.
• Key: This is the specific path or file name within the bucket where your state data is stored. It’s like the exact location of your state file within the bucket.
• DynamoDB Table: This table is used for state locking, ensuring that only one person or process can make changes to the state file at a time, preventing any accidental overwrites or conflicts.

Step 2: Backup Your Existing State

Before you start migrating your Terraform state, make sure to save a backup of your current state file. This backup file will help you in future if something goes wrong during the process and you dont want to avoid losing any data. To back up your state file, follow these steps based on your current backend setup:

• If you’re using an S3 backend: You can pull the terraform.tfstate file from your S3 bucket using the terraform state pull command.

terraform state pull > terraform-backup.tfstate

• For Local State: If your state file is stored in your local machine, simply make a copy of it by using this command:

cp terraform.tfstate terraform-backup.tfstate

Step 3: Update the Backend Configuration in Terraform Code

Next, you need to update your main.tf or backend.tf file to define the new backend block where you want to store your state. Make sure that you enter all the details correctly, such as the bucket name, key (state file path), and region. You’ll need to set environment variables for secure access, like AWS credentials if you’re using AWS, or equivalent credentials for other providers such as Google Cloud or Azure, depending on the backend you’re migrating to.

Step 4: Initialize the New Backend

After you’ve updated the backend configuration, you need to connect Terraform to the new backend. You can do this by running: :

terraform init -migrate-state

Here, the -migrate-state flag will make sure that Terraform migrates your existing state file to the new backend. When you run this command, Terraform will detect that you’ve changed the default backend configuration and ask if you want to migrate your state.

This step is required here because it migrates your state from the old backend to the new one. Make sure to double-check every setting within your configuration to be sure that the state migration is done correctly and that Terraform is now using the new backend to manage your infrastructure.

Step 5: Verify the State Migration

As we are done with the migration process, the next step is to verify that every resource migrated over correctly to the new backend. This will make sure that your resources are now managed by the new backend and nothing was missed during the migration.

Check the resources within your state data by running the command:

terraform state list

If the migration was successful, running the command should display all your resources in the terminal output, just as they were before. This confirms that the new backend has the complete state file stored correctly.

The next and last step would be to check if any resources were misconfigured. You can verify this by simply running terraform plan, and if everything was migrated properly, the output will not show any infrastructure changes or updates meaning your state file is in sync with your actual resources.

Usage Examples: Migrating Terraform State Between Backends

Now that we’ve covered all the main steps required for migrating the Terraform state between backends, let’s take a look at some examples of where we can implement these steps.

We’ll go through three scenarios: first, migrating your state from a local machine to a remote backend; second, migrating the state back from a remote backend to a local machine; and finally, switching your state between different environments, such as moving from development to production.

Scenario 1: Migrating State from Local to Remote

Here, we’ll move our Terraform state file from a local machine to an S3 backend. This is required and also considered as a best practice when your team grows or when more people start working on the same infrastructure.

Step 1: Create a Local Terraform Setup

Now, start by creating main.tf to create an AWS EC2 instance:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "terrateam-in" {
  ami           = "ami-0e86e20dae9224db8"
  instance_type = "t2.micro"
  subnet_id     = "subnet-02efa144df0a77c13"
  tags = {
    Name = "TerrateamInstance"
  }
}

Now, initialize Terraform and create the resources by running the following commands:

terraform init
terraform apply

This process will generate the terraform.tfstate file on your local machine.

Step 2: Set Up AWS S3 and DynamoDB

Next, we need to create an S3 bucket for the backend which will store the state file and a DynamoDB table for state locking.

For creating the S3 bucket, just run this command in your CLI:

aws s3api create-bucket --bucket infra-prod-01 --region us-east-1

Now, for creating the DynamoDB table use this command:

aws dynamodb create-table \
  --table-name infra-state-lock-01 \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \
  --region us-east-1

Step 3: Update the Terraform Backend Configuration

Next, we need to create a backend configuration in your main.tf. This backend configuration will tell Terraform on where to store the data source the state file, how to access it, and how to lock it when multiple members are making changes on the same state file.

By setting up this backend configuration, your state file will be stored in the S3 bucket you created, and DynamoDB will manage state locking.

terraform {
  backend "s3" {
    bucket         = "infra-prod-01"
    key            = "global/s3/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "infra-state-lock-01"
    encrypt        = true # Enable encryption
  }
}

Step 4: Migrate the State to S3

Now it’s time to migrate your state file to S3. You can run the following command:

terraform init -migrate-state

Your Terraform state is now stored in the AWS S3 backend. You can verify this change by logging into the AWS Console and navigating to your S3 bucket (terrateam-prod-01). Here, you should see a file named terraform.tfstate.

By opening this file will confirm that your state has been successfully migrated, and all the details of your infrastructure will be stored here.

Scenario 2: Migrating Terraform State from Remote Backend to Local Setup

In this scenario, we’ll migrate our Terraform state file from a remote backend back to our local machine. This isn’t a common practice for organizations and is usually not required also, but it can be helpful if you’re troubleshooting, working offline, or need more control over the state for testing or debugging. It allows you to work on the infrastructure without impacting the team’s shared state file on the remote backend, which makes it much easier to experiment or make temporary changes.

Step 1: Prepare Your Local Environment

Create a main.tf file and add the local backend configuration of the current remote backend from where you want to migrate the state file, along with your Terraform code.

terraform {
  backend "s3" {
    bucket         = "infra-prod-01"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "infra-state-lock-01"
    encrypt        = true
  }
}

Step 2: Initialize the Backend and Pull the State File Now, you need to initialize Terraform by running terraform init to connect it with the current remote backend. This step makes sure Terraform recognizes your existing state file stored in S3.

Step 3: Pull the State File to Your Local Setup

The next step is to bring the state file from the remote backend to your local machine. You can do this by running the following command to pull the state file:

terraform state pull > terraform.tfstate

This command fetches the state file from the S3 bucket and saves it as terraform.tfstate in your current directory. This means you now have a local copy of the state file that you can work with.

Step 4: Remove the Remote Backend Configuration Since you want to work with the state file locally, update your main.tf to remove the remote backend configuration.

Step 5: Verification To verify that Terraform is now using the local state file, simply run the following command:

terraform state list

This command will show all your resources as they were before, which confirms that the state file is now managed locally and have successfully migrated your Terraform state from a remote backend (AWS S3) to a local setup.

Scenario 3: Migrating State Between Environments

Now, we’ll move our Terraform state from a development environment to a production one. This use case is specifically for teams that started with a single state file and are now looking to separate it into different environments, like development, staging, and production, without rebuilding the resources.

This scenario is for teams that initially used a single state file for all environments (production, staging, and development) and now want to split it into multiple states for better management. If your production, staging, and development setups are all mixed into one state file, this method will help you separate them into distinct environments without needing to rebuild your infrastructure.

You can also use the terraform workspace select command before migrating the state to switch between environments. This approach works well for teams transitioning from a single state setup to separate environments, providing better control and management of each.

Step 1: Start with the Development Environment

Firstly, create a main.tf file with your development environment Terraform configuration:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "terrateam-server" {
  ami           = "ami-0e86e20dae9224db8"
  instance_type = "t2.micro"
  subnet_id     = "subnet-02efa144df0a77c13"
  tags = {
    Name        = "terrateam-server"
    Environment = "Development"
  }
}

terraform {
  backend "s3" {
    bucket         = "dev-state-23412"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "infra-state-lock-01"
    encrypt        = true
  }
}

Now, initialize and apply this configuration by running these commands:

terraform init
terraform apply

After running these commands, Your state file is now stored in the development S3 bucket as defined in your backend configuration.

Step 2: Update the backend configuration

Next, we’ll change the backend configuration to point to the production bucket:

terraform {
  backend "s3" {
    bucket         = "prod-state-23412"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "infra-state-lock-01"
    encrypt        = true
  }
}

Now that you’ve updated the backend configuration to point to the production bucket, you need to reinitialize Terraform so it connects to this new backend.

Run the following command:

terraform init -migrate-state

This command will prompt Terraform to recognize the change in the backend configuration and will handle moving the state file from the development S3 bucket to the production S3 bucket.

Step 3: Update Resource Attributes for Production

Now that we’ve switched to the production environment, we need to change our instance’s tags to match the production requirements.

resource "aws_instance" "terrateam-server" {
  ami           = "ami-0e86e20dae9224db8"
  instance_type = "t2.micro"
  subnet_id     = "subnet-02efa144df0a77c13"
  tags = {
    Name        = "terrateam-server"
    Environment = "Production"
    Owner       = "Saksham"
  }
}

Finally, run terraform apply to apply these changes and make sure your production setup is configured correctly.

After this, we have successfully migrated the backend from the development S3 bucket (dev-state-23412) to the production S3 bucket (prod-state-23412). The changes we made, such as updating the tags, were applied only to the existing instance, and the resource was not recreated. This verifies that Terraform managed the transition correctly without affecting the existing infrastructure.

Best Practices

Now, let’s take a look at some of the best practices you need to follow while migrating state backend:

• Always Backup State File: Before starting any migration, always back up your current state file. This makes sure that you have a fallback if something goes wrong during the process. You can back up the state file by using terraform state pull or downloading it directly from your backend storage, like S3.
• Enable Versioning on Remote Backends: When using a remote data store like AWS S3 or Google Cloud Storage, enable versioning on your bucket. This keeps a history of all the changes made to your state file, which makes it much easier to roll back to a previous version if needed. This is helpful if you accidentally make changes that break your setup, as you can quickly recover the previous state without losing data.
• Use State Locking Mechanisms: Always use a locking mechanism, such as a DynamoDB table, when using S3 as your backend. This helps prevent multiple people from making changes to the state file at the same time, making sure that only one person can make changes at a time.
• Avoid Manual Edits to the State File: Never manually edit the state file unless it’s absolutely necessary, and you know exactly what you’re doing. The state file is sensitive and can easily become corrupted if changed incorrectly, which can lead to serious issues with your infrastructure. Always let Terraform handle the state file to ensure its integrity.

Make Infrastructure Management Easy with Terrateam

Management of infrastructure changes can be tricky and time-consuming, especially as your infrastructure setup grows. You can ensure that there’s a simpler way to manage everything without the usual hassles by using Terrateam.

Terrateam offers a modern and easy way to handle your infrastructure. It’s designed to make your team’s workflow smoother with features like automatic pull request plans, centralized configurations, and support for multiple environments. To get started with Terrateam, you simply need to follow their setup guide, which walks you through the process of integrating Terrateam with your GitHub repository. Once set up is done, Terrateam will generate automated plans on every pull request.

Here, Terrateam has generated a plan and cost estimation associated with the pull request. After reviewing the plan, you can apply changes directly by commenting terrateam apply.

This integration simplifies your workflow and makes sure that all the infrastructure changes are applied accurately without the need for manual intervention.

Terrateam makes sure that every change goes through proper reviews, reducing the chances of errors. With its GitOps integration, it lets you control your infrastructure directly from your GitHub repository, ensuring all changes are traceable and stored in one place.

Ready to enhance your infrastructure management?

Try Terrateam today and experience seamless, automated directory and workspace locking for consistent and conflict-free collaboration.

Conclusion

Now, you should have a clear understanding of how to move your Terraform state between backends. Whether you’re going from local to remote, switching environments, or planning for any unexpected issues, following these steps will help you do it without problems. Always remember to back up your state, test the process in a safe environment first, and stick to the best practices. This way, you can manage your Terraform projects smoothly.