What is Terraform Cache?

Managing your cloud infrastructure with IAC tools like Terraform, Ansible, and Pulumi can sometimes be a bit challenging for DevOps Engineers. Let’s say you are working at an organization where multiple teams use Terraform to manage resources within their infrastructure. When you work on different Terraform projects, running terraform init can slow things down because it downloads the same provider plugins and modules again. This takes extra time, causing delays, especially if you have large setups or switch between projects often.

Downloading the same plugins and modules repeatedly can slow down the process, especially in projects with many dependencies. It also causes problems if the downloaded versions are different from what the project expects. For example, if a newer provider version changes how a resource is configured, Terraform might show errors like “unsupported argument” or fail to deploy resources correctly. This can slow down the infrastructure setup and requires additional fixes.

In this blog, we’ll explain how Terraform cache helps save time by avoiding repeated downloads. You’ll learn what the cache does, how it works, and the steps to set it up for your projects.

Introduction to Terraform Cache

Terraform cache helps fix the problem of repeated downloads by storing provider plugins and modules in a centralized location in the system. Without caching, running terraform init will download these files each time for a new project or after clearing the cache. This process can take extra time, especially when you have too many Terraform files or the infrastructure size is large, especially in projects with large configurations or teams working on multiple environments. For example, downloading a large provider plugin like AWS can take extra time during initialization. By saving these files locally, Terraform avoids downloading them again, making the process quicker and smoother.

Terraform searches the local directory for provider plugins and modules before downloading them from the internet. For example, when you run terraform init in a project that uses the AWS provider, Terraform downloads the plugin and saves it in the cache directory (e.g., .terraform or a centralized cache if configured). If you create another project that also uses the AWS provider, Terraform will load the plugin directly from the cache instead of downloading it again. This avoids unnecessary downloads and reduces the time it takes to initialize projects.

Caching is especially useful in deployments with large configurations or multiple environments. For example, in a project using AWS, the provider plugin might take a few minutes to download during terraform init. Once cached, the plugin is reused in other projects or environments, avoiding repeated downloads.This prevents problems with mismatched versions and saves time by ensuring that the same plugin version is used for each run. Caching prevents plugins from being downloaded repeatedly, which speeds up repetitive setups like staging and non-production environments with the same configurations.

To understand how caching helps in Terraform, it’s important to know what data Terraform stores in its cache. Terraform speeds up workflows by avoiding repetitive data downloads and keeping some files locally. Modules and provider plugins are the two main categories of data that are kept in the cache.

• Provider Plugins

Provider plugins are required for Terraform to share with cloud service providers like AWS, Azure, or GCP. Terraform downloads provider plugins when you run terraform init for the first time or when a required plugin is missing. These plugins are then saved in the cache, so they don’t need to be downloaded again in future runs. This reduces initialization time and ensures faster infrastructure setup.

• Module Downloads

Modules are reusable pieces of code that help organize Terraform configurations. When you use modules, whether from the Terraform registry or private sources, Terraform downloads them and stores them in the cache. This way, the modules don’t need to be downloaded again in later runs, saving time and ensuring the same version is used consistently across deployments.

How Cache is Used in Terraform

Caching allows Terraform to save provider plugins and modules locally, avoiding repeated downloads. This speeds up initialization and configuration, especially when working in environments with slow or unreliable networks.

How Terraform Cache is Generated

Terraform creates a cache when you run terraform init for the first time. It downloads the required provider plugins and modules and saves them in the .terraform directory in your project.

Let’s see how it affects Terraform’s performance.

• Faster Initialization:

Terraform does not have to download these files again if they are already in the cache .It saves time by using the .terraform cached files instead. For example, it may take several minutes to download the AWS provider plugin from the remote Terraform registry; however, if it is cached, this process can be completed in a matter of seconds.

• Consistent and Quick Resource Fetching:

When you run the terraform plan or terraform apply, Terraform may need details about the resources, like EC2 Instances or S3 Buckets, etc., in your configuration, such as provider metadata or module. Instead of downloading these modules and provider plugins again, Terraform fetches them from the local cache.

• Works Offline or with Limited Internet Access:

Terraform can still work using cached files even if there’s no internet connection. For example, if you’re in a remote location or the network is down, Terraform will use the saved provider plugins and modules from the local cache. This avoids delays and lets you continue running commands like terraform initor terraform apply.

• Improved Speed in Repeated Runs:

If you’re working on multiple projects that use the same provider or module, the cache will ensure you don’t have to download them repeatedly. For example, if you’ve already cached the Azure provider for one project, all other projects using Azure will load it directly from the cache.

Cache Storage Locations

Terraform improves operation workflow speed by using cache to store files like provider plugins and modules. However, where are these files kept? It is essential to understand storage locations and management strategies, particularly when working with large teams or on multiple projects. Let’s examine where Terraform stores its cached files and how to effectively handle them.

• Default Storage Locations

Terraform saves cached files in a folder named .terraform within your project directory. This folder contains provider plugins, modules, and other data required for Terraform to work. Each project has its .terraform directory.

• Plugin Cache Directory

Terraform allows you to set up a centralized cache directory for storing provider plugins. This is done by configuring the plugin_cache_dir in the terraform.rc file. For example, you can specify a directory like /home/user/.terraform.d/plugin-cache to store plugins. This allows Terraform to reuse the same plugins across multiple projects.

If you are working in a local, Terraform will also create a cache in the .terraform directory within that project. However, this cache is only for that specific project. If you stop using the project, the cache in its .terraform directory won’t be reused elsewhere. In contrast, a centralized cache configured in terraform.rc remains available for all your projects, saving time and avoiding repeated downloads.

• Managing Cache Across Workspaces or Projects:

When working with Terraform workspaces that use the same providers or modules, a centralized cache can help avoid downloading these files repeatedly. For example, if multiple workspaces use the AWS provider, a centralized cache allows Terraform to reuse the same plugin file for all workspaces instead of downloading it each time. This reduces download time and ensures consistent provider versions across workspaces.

Implementing a Centralized Terraform Plugin Cache Directory

A centralized cache directory in Terraform stores provider plugins in a single location, so they don’t need to be downloaded repeatedly. For example, you can configure a directory like /home/user/.terraform.d/plugin-cache using the terraform.rc file. Once set up, Terraform reuses plugins from this directory whenever needed, saving time and ensuring consistent plugin versions.

• Directory Structure for Caching:

Terraform uses a specific directory structure for centralized caching. This directory is located on most systems at AppData/Roaming/terraform.d Inside this folder, Terraform organizes cached files, including provider plugins.

• Overview of the plugin_cache Directory:

Within the terraform.d directory, the plugin_cache folder stores provider plugins. Each plugin is saved in a specific format based on its provider name and version. For example, the AWS provider plugin might be stored as plugin_cache/registry.terraform.io/hashicorp/aws/3.75.2. This structure allows Terraform to quickly find and reuse plugins without downloading them again.

Setting Up the terraform.rc Configuration File

To enable a centralized plugin cache in Terraform, you need to configure the terraform.rc file in the AppData / Roaming path in your directory. This file tells Terraform where to store and access cached provider plugins. By setting it up correctly, you can ensure that all projects use a shared cache directory, saving time and avoiding repeated downloads.

• Purpose of the terraform.rc File:

The terraform.rc file is a configuration file that lets you control specific Terraform settings, including where provider plugins are stored. One important setting is plugin_cache_dir, which specifies a centralized directory to store downloaded provider plugins. This allows Terraform to reuse the plugins for future runs, avoiding the need to download them again. For example, you can set plugin_cache_dir = "/home/user/.terraform.d/plugin-cache" to centralize all plugin files in one location.

The plugin_cache_dir setting in the terraform.rc file tells Terraform where to save downloaded provider plugins. This directory acts as a shared cache, allowing multiple projects to access the same plugins without downloading them repeatedly. For example, adding the following line to your terraform.rc file sets the cache location to “/home/user/.terraform.d/plugin-cache`:

• How plugin_cache_dir Works During Initialization:

When you run terraform init, Terraform checks the directory set in plugin_cache_dir for the required plugins. If the plugins are found, Terraform can use them without downloading them. If they are missing, Terraform downloads the plugins from the registry and saves them in the cache. This allows Terraform to use the saved plugins the next time you initialize a project that requires the same plugins, avoiding repeated downloads.

Hands-On: Configuring the terraform.rc File

Using a centralized cache in Terraform simplifies plugin management by storing them in a shared location. Here’s how to set up and test the terraform.rc file for a centralized cache.

Locate or Create the terraform.rc File:

The terraform.rc file is used to configure Terraform’s global configuration. You need to either locate this file or create it if it doesn’t exist:

• Windows: The file should be saved in the AppData\Roaming directory. You can find this folder by pressing Win + R, typing %AppData%, and opening the Roaming folder. Inside this folder, check for the terraform.rc file. If it’s not there, create a new file named terraform.rc in this directory.
• Linux/Mac: The file should be located in your home directory as .terraform.rc. Open a terminal and run the command:

To enable a centralized cache for Terraform plugins, you need to specify the directory where the plugins will be stored. This is done by adding a configuration in the terraform.rc file. Here’s how you can set it up: plugin_cache_dir = "/home/user/.terraform.d/plugin-cache"

If the directory doesn’t already exist, create it with the following command:

mkdir -p /home/user/.terraform.d/plugin-cache

Test the Centralized Cache Setup

Once the plugin_cache_dir is configured and the directory is created, you can test the setup by initializing a Terraform project:

• Navigate to a Terraform Project Directory Open a terminal and go to a directory containing your Terraform configuration files.
• Run terraform init:

After running terraform init, verify that the provider plugins were saved in the centralized cache directory. Use the following command to check the contents of the cache:

    ls /home/user/.terraform.d/plugin-cache/registry.terraform.io/

This command lists the plugins stored in the plugin_cache_dir. You should see subdirectories for the providers, such as hashicorp/aws or hashicorp/azurerm, which confirm that the cache is set up and working properly. If the directory is empty, ensure that terraform init is completed successfully and that the plugin_cache_dir path is in the terraform.rc file is correct.

Reuse Plugins in Another Project
Create a second Terraform project that uses the same provider and run terraform init.

By configuring a centralized cache, you ensure faster Terraform workflow and maintain consistent provider versions across all your projects.

Common Errors and Troubleshooting

Setting up a centralized Terraform cache can speed up workflows, but issues can arise if the cache path is incorrect or if the stored plugins are outdated. These problems might lead to Terraform failing to find the required plugins or using the wrong versions. Here’s how to identify and fix these issues.

Misconfigured Cache Paths

Issues with the plugin_cache_dir configuration in the terraform.rc file can disrupt Terraform’s ability to use the centralized cache effectively. Misconfigurations often lead to repeated downloads or errors during initialization. Let’s explore this in more detail.

Incorrect Paths in terraform.rc

If the path specified in the plugin_cache_dir attribute is incorrect or points to a non-existent directory, Terraform won’t use the cache. Instead, it may either download plugins again or create a new cache in the wrong location.

• Impact: Terraform fails to locate the cache directory and re-downloads the required plugins, defeating the purpose of a centralized cache. This can also result in slower initialization or duplicate cache directories.
• Solution: Check the plugin_cache_dir entry in the terraform.rc file and ensure it points to a valid and accessible directory.

Resolving Misconfigured Cache Paths

• If Terraform displays errors related to the cache directory:
  • Confirm the terraform.rc file is in the correct location (~/.terraformrc on Linux/Mac, AppData/Roaming/terraform.rc on Windows).
  • Ensure the directory path in plugin_cache_dir is spelled correctly and exists. Create the directory if needed:
• Issues with Corrupted or Old Plugins:

Sometimes, provider plugins in the cache can become corrupted or outdated, causing Terraform to fail during initialization or apply steps.

• Solution: Clear the cache and reinitialize:

rm -rf /home/user/.terraform.d/plugin-cache/

• Refreshing the Cache: If clearing the cache doesn’t resolve the issue, check the provider versions in your Terraform configuration and ensure they are compatible. Update the versions if necessary, and run terraform init -upgrade.

Conclusion

Setting up a Terraform cache speeds up workflows by avoiding repeated downloads and ensuring consistent plugin versions. A properly configured terraform.rc file and resolving issues like misconfigured paths or outdated plugins can maximize its benefits.