As an Infrastructure or DevOps engineer, one of the more annoying situations is when a resource in your infrastructure starts misbehaving. For example, an EC2 instance might keep crashing because it does not have enough memory or due to issues with the application running on it. A load balancer could stop routing traffic correctly because its health checks weren't updated when the application configuration changed. Or a database might fall out of sync with other services because of changes made outside Terraform.

The first instinct might be to replace that resource entirely. However, destroying it can cause downtime for your services and also affect other resources that are relying on it. Removing the resource from Terraform's state causes Terraform to lose track of it, meaning you would need to re-import it later. These options often feel too complicated to fix a very specific problem. This is where terraform taint comes into play. It allows you to flag such resources for recreation during the next terraform apply.

In this blog, we'll explain how terraform taint works, when to use it, and how it compares to commands like destroy and state rm.

Struggling with Terraform taint?

Our team specializes in helping teams tackle challenging Terraform problems.

What is Terraform Taint?

terraform taint is a command that allows you to mark a specific resource in Terraform's state for recreation. It ensures that the tainted resource is destroyed and rebuilt during the next terraform apply without requiring changes to your configuration files. This makes it a simple and controlled way to address issues like resource drift, corruption, or outdated configurations while keeping the rest of your infrastructure untouched.

When you use the taint command, Terraform flags the resource addresses in its state file as needing replacement. This does not modify your configuration but ensures that the resource is recreated according to the defined configurations within your code. For example, if an EC2 instance named web_server is experiencing issues, you can run:

terraform taint aws_instance.app_server

undefined

This tells Terraform to destroy and recreate the web_server instance during the next apply. The process ensures that the new instance is provisioned with the same configuration, keeping your infrastructure consistent without requiring any additional intervention from your end.

What is the key difference between using taint and not using taint?

Now, when handling issues with a single resource from within your infrastructure, the choice between using terraform taint or not is an important one. Below is a table comparing the two approaches to give you a clear understanding of their differences and when each might be applicable:

undefined

Using terraform taint gives you more control, especially when Terraform's drift detection through terraform plan or terraform apply does not catch the issue, or when you need to replace a resource before the issue expands.

When would you use Terraform Taint

There are specific scenarios where terraform taint is the best option for resolving resource issues without disturbing the rest of your infrastructure. Here are the most common use cases for terraform taint:

Fixing a Corrupted Resource: If an EC2 instance stops working because it's assigned an incorrect subnet or security group or fails to boot due to a missing dependency in its startup script, marking it as tainted tells Terraform to destroy and recreate the instance during the next apply. This can help resolve issues like a corrupted EBS volume or wrong configurations that prevent the instance from working, making sure it is set up correctly again.
Rebuilding Outdated Configurations: When an existing resource configuration no longer matches your Terraform code, such as a load balancer using outdated health checks or old SSL certificates, tainting ensures that the resource is replaced to reflect the updated configurations. This avoids the need to update the resource through the cloud provider's console or AWS CLI.
Resolving Resource Drift Caused by Manual Changes: Changes made outside Terraform, such as editing an IAM role policy through the AWS Console or modifying tags on an S3 bucket using the AWS CLI, can cause drift that Terraform does not always detect during a plan. Tainting forces Terraform to destroy and recreate the resource, making sure that it matches the state defined in your Terraform code.
Addressing Failing State Updates: In some cases, a resource might be created or updated with incorrect configurations, but Terraform marks the process as successful. For example, a DNS record might be created but points to the wrong IP address because the target IP was changed in the configuration but not applied properly during the deployment. By tainting the resource, you can force Terraform to destroy and recreate it during the next apply, ensuring that the correct resource configuration is applied as per your Terraform code.
Proactive Recreation for Maintenance or Testing: In staging or QA environments, resources like EC2 instances or databases might need to be recreated to test updates, such as applying a new version of an application or validating changes to configuration files. Using terraform taint lets you taint resource configurations for destruction and recreation during the next apply, making sure of a clean rebuild without having to delete the resources through the console and re-import them into Terraform.

These scenarios show how terraform taint can fix specific resource problems while keeping your infrastructure aligned with your Terraform code.

Comparing Terraform Taint to Other Commands

Now, when you are working with Terraform, different commands handle resource management in specific ways. Knowing when to use terraform taint, terraform destroy, terraform state rm, or just remove the resource from the configuration is important for maintaining control over your infrastructure. Each has a different purpose, and the choice depends on the scenario. Let's break them down:

Taint vs. Destroy

When working with terraform taint and terraform destroy, the key difference lies in how the resource is handled. terraform taint command marks the resource for recreation during the next terraform apply. The resource remains in the state file, allowing for a controlled replacement without impacting the rest of the infrastructure. For example, if an EC2 instance has an incorrect network configuration, such as being assigned to the wrong subnet, you can use terraform taint to flag the resource instance for recreation during the next apply. This ensures that the other resources like attached security groups or load balancers, remain untouched.

On the other hand, terraform destroy -target deletes the resource immediately from both the cloud and Terraform's state file. This command is particularly useful in urgent situations, such as removing a security group that allows unrestricted access to ports like SSH (22) or HTTP (80). It is also helpful for removing resources that were mistakenly deployed in the wrong environment, such as a database created in production instead of staging.

To see these commands in action, let's explore how to use them with the EC2 instance defined in your Terraform configuration.

If you want to recreate the instance due to an incorrect network configuration, such as being assigned to the wrong subnet, you can use terraform taint to mark it for recreation. Run the following commands:

terraform taint aws_instance.app_server

undefined

This process flags the instance for destruction and makes sure that it is recreated during the next apply.

To verify, you can run terraform plan to confirm that the resource is marked for recreation. The instance will be rebuilt with the correct configuration while leaving other resources, like security groups or load balancers, untouched.

undefined

Now, If you want to undo the taint action and keep the resource as-is, you can use the following command to untaint it:

terraform untaint aws_instance.app_server

undefined

Alternatively, if the resource needs to be removed immediately, such as a security group exposing sensitive ports like SSH (22), you can use terraform destroy -target to delete it. Execute the following command:

terraform destroy -target=aws_instance.app_server

undefined

This deletes the instance from both the cloud and Terraform's state file. After running the command, use terraform state list command to verify that the resource is no longer tracked by Terraform.

By implementing these steps, you can understand how terraform taint and terraform destroy handle resources differently, giving you control over your infrastructure based on the specific situation.

Taint vs. State RM

When comparing terraform taint and terraform state rm, the main difference is whether Terraform continues to manage the resource. Using terraform taint, the resource remains part of Terraform's state and is rebuilt during the next apply. For example, if an IAM role is missing a policy due to a drift caused by some changes done through AWS console, you can use terraform taint to make sure that it is destroyed and recreated with the correct configuration. This allows Terraform to realign the resource with the definitions in your .tf file.

Whereas terraform state rm removes the resource from Terraform's state without destroying it in the cloud. The resource remains active in your cloud environment but is no longer managed by Terraform. This is useful for scenarios where a resource needs to be transitioned to another team, tool, or management system. For example, if an EC2 instance is being moved to a different IaC tool like Pulumi, or transitioned to direct control by system administrators, you can remove it from Terraform's state while leaving it untouched in AWS.

Let's see how these commands work using the EC2 instance defined in your Terraform configuration.

If the instance needs to be recreated due to drift, you can use terraform taint. This will make sure that the resource is destroyed and rebuilt during the next apply, aligning it with the configuration in your .tf file.

On the other hand, if you want Terraform to stop managing the instance while keeping it active in AWS, you can use terraform state rm.

Before proceeding, it's a good idea to run terraform state list to see all the resources currently tracked by Terraform. This allows you to confirm the instance is being managed before removal and verify that it is no longer listed afterward.

undefined

To remove the resource from Terraform's state file without affecting its presence in the cloud, run the following command:

terraform state rm aws_instance.app_server

undefined

Now, you can confirm that Terraform no longer tracks the resource by using terraform state list.

undefined

The instance will stay active in AWS but will no longer be tracked by Terraform, making it possible for other tools or teams to manage it.

These commands let you either rebuild resources or remove them from Terraform's control while keeping them functional within your infrastructure.

Taint vs. Removing Configuration

When deciding between terraform taint and removing a resource from the configuration file, the choice depends on whether the resource is still required in your infrastructure or not. Using terraform taint, the resource remains part of your Terraform configuration and state. Tainting ensures that the resource is destroyed and recreated during the next apply without making changes to your .tf files. For example, if an EC2 instance is facing issues like being misconfigured with the wrong instance type or a missing tag, you can taint the resource using terraform taint aws_instance.web_server command.

On the other hand, removing a resource from the configuration entirely deletes it from both Terraform's state file and the cloud infrastructure. This is the approach to take when the resource is no longer needed. For example, if a database is retired from the production environment, you can remove its resource block from the .tf file.

However, directly removing the configuration is not always a good idea, especially in a production environment, as it can delete some of the important resources. However, for beginners, this is a common way to learn Terraform and understand how resource management and the lifecycle work.

Firstly, open your .tf file, remove the resource block for the EC2 instance, and run terraform apply.

After making these changes, Terraform will remove the resource from both the cloud and its state file. You can check this by running terraform state list to make sure that the resource is no longer tracked. You can also verify in your cloud provider's console to confirm that the resource has been deleted.

Using terraform taint lets you replace a resource that is still needed, making sure that it is rebuilt correctly. Removing a resource from the configuration is the best choice when the resource is no longer needed and should be permanently removed. Both methods help keep your infrastructure well-managed and aligned with your current needs.

By understanding these differences, you can choose the right approach based on your specific needs. Below is a comparison table summarizing how terraform taint, terraform destroy, terraform state rm, and removing a resource from the configuration differ in purpose and use cases:

undefined

By knowing how these commands work, you can make better choices to manage your infrastructure, which will help you focus on what matters most.

Conclusion

So far, we've covered how terraform taint works, its use cases, and how it compares to other Terraform commands like destroy, state rm, and removing resources from configuration. Using the right command helps manage your resources more effectively, whether it's recreating a resource, migrating it to another IaC tool, or removing it entirely from your infrastructure.

Features

Learn More

Learn

Connect