Terraform modules and automatic discovery

Terraform modules are a powerful way to encapsulate and reuse common infrastructure components across your Terraform projects. Terrateam supports working with Terraform modules seamlessly, allowing you to trigger plans and applies when changes are made to your modules or the directories that depend on them. Additionally, Terrateam’s automatic module discovery feature simplifies your workflow by automatically detecting module dependencies and configuring Terrateam accordingly.

Triggering on module updates

When using Terraform modules, you often want to trigger a plan or apply operation in the directories that depend on the module when the module itself is updated. Terrateam allows you to configure this behavior using the when_modified directive in your Terrateam configuration file (.terrateam/config.yml).

Here’s an example configuration that creates a dependency between the iam directory and the modules directory:

dirs:
  modules:
    when_modified:
      file_patterns: []
  iam:
    when_modified:
      file_patterns: ["iam/*.tf", "iam/*.tfvars", "modules/*.tf"]

In this configuration:

The modules directory is configured with an empty file_patterns list, which means that changes to the modules directory itself will not trigger any operations.
The iam directory is configured to trigger a plan operation when changes are made to files matching the patterns iam/*.tf, iam/*.tfvars, or modules/*.tf.

With this setup, when a change is made to a Terraform file in the modules directory, Terrateam will automatically trigger a plan operation for the iam directory, ensuring that any updates to the module are properly reflected in the dependent directory.

Automatic module discovery

Terrateam’s automatic module discovery feature takes the concept of module dependencies a step further by automatically detecting and configuring them for you. By enabling the code indexer, Terrateam can analyze your repository structure and Terraform code to determine which directories should be executed during plan and apply operations based on module dependencies.

How it works

When the code indexer is enabled, Terrateam creates a map of your repository, including:

Terraform modules and the directories that reference those modules
Symbolic links in the repository

Using this information, Terrateam automatically determines:

Module directories should not be run if they change
Directories referencing a module should be run if the module changes

This ensures that the appropriate directories are executed based on the module dependencies in your Terraform code, without requiring manual configuration.

Enabling automatic module discovery

To enable automatic module discovery, you need to configure the code indexer in your Terrateam configuration file (.terrateam/config.yml). Here’s an example configuration:

indexer:
  enabled: true

By setting enabled to true, you enable the code indexer, which automatically discovers module dependencies and generates the necessary Terrateam configuration.

Example workflow

Let’s walk through an example of how automatic module discovery works in practice:

Consider a repository with the following structure:
```
├── modules
│   └── vpc
│       └── main.tf
└── prod
    └── network
        └── main.tf
```
In this example, the prod/network directory references the vpc module in the modules directory.
Enable the code indexer in your .terrateam/config.yml file:
```
indexer:
  enabled: true
```
When a change is made to the modules/vpc directory, Terrateam automatically detects that the prod/network directory depends on this module.
Terrateam generates the necessary configuration to ensure that:
- The modules/vpc directory is not run directly, as it contains state information
- The prod/network directory is run when the modules/vpc directory changes
When a pull request is opened with changes to the modules/vpc directory, Terrateam automatically triggers a plan operation for the prod/network directory.
Review the plan output and collaborate with your team to ensure the changes are as expected.
Once the pull request is approved and merged, Terrateam automatically applies the changes to the prod/network directory based on the module dependency.

By leveraging automatic module discovery, Terrateam simplifies your Terraform workflow by automatically detecting and configuring module dependencies, reducing the need for manual configuration.

Using modules from a separate repository

Terraform allows you to reference modules from external sources, including separate Git repositories. Terrateam supports this workflow, enabling you to use modules from both public and private repositories.

Public repositories

To use a module from a public GitHub repository, you can simply reference the repository URL in your Terraform code:

module "consul" {
  source = "github.com/opentofu/example"
}

Terrateam automatically recognizes github.com URLs and will clone the repository as needed during plan and apply operations.

Private repositories

To use a module from a private GitHub repository, you need to configure SSH access for Terrateam to clone the repository. Here’s how you can set this up:

Install the Terrateam GitHub application on your private modules repository. This grants Terrateam the necessary permissions to clone the repository.

Generate a new SSH key pair:

ssh-keygen -t ed25519 -C "Terrateam SSH Key" -N "" -f ~/.ssh/terrateam-ssh-key

Add the SSH public key to your private modules repository as a deploy key:

gh repo deploy-key --repo "OWNER/TERRAFORM-MODULES-REPO" add ~/.ssh/terrateam-ssh-key.pub

Add the SSH private key to your main Terraform repository as a GitHub Actions secret:

gh secret --repo "OWNER/TERRAFORM-REPO" set TERRATEAM_SSH_KEY < ~/.ssh/terrateam-ssh-key

Update your Terraform code to reference the private repository using the SSH URL:

module "example_module" {
  source = "git::ssh://git@github.com/OWNER/TERRAFORM-MODULES-REPO.git"
}

Terrateam will automatically detect the TERRATEAM_SSH_KEY secret and configure the necessary SSH access to clone the private repository during plan and apply operations. To add more than one key, just create additional GitHub Actions secrets using the TERRATEAM_SSH_KEY_ prefix. For example, TERRATEAM_SSH_KEY_FOO and TERRATEAM_SSH_KEY_BAR.

Non-GitHub repositories

If your modules are stored in a non-GitHub repository, you can still use them with Terrateam by following a similar process:

Generate a new SSH key pair:

ssh-keygen -t ed25519 -C "Terrateam SSH Key" -N "" -f ~/.ssh/terrateam-ssh-key

Add the SSH public key to your Git hosting provider as an authorized key for your modules repository.

Add the SSH private key to your main Terraform repository as a GitHub Actions secret:

gh secret --repo "OWNER/TERRAFORM-REPO" set TERRATEAM_SSH_KEY < ~/.ssh/terrateam-ssh-key

Update your Terraform code to reference the repository using the SSH URL:

module "example_module" {
  source = "git::ssh://username@example.com/TERRAFORM-MODULES-REPO.git"
}

Configure Terrateam to perform SSH host key scanning before plan and apply operations by adding the following to your .terrateam/config.yml:
```
hooks:
  plan:
    pre:
      - type: run
        cmd: ['ssh-keyscan-pre-hook', 'example.com']
  apply:
    pre:
      - type: run
        cmd: ['ssh-keyscan-pre-hook', 'example.com']
```
The ssh-keyscan-pre-hook script is a simple wrapper for the ssh-keyscan command that adds the host keys to the known hosts file, ensuring that Terrateam can securely connect to your Git repository.

Best practices

When working with Terraform modules and Terrateam, consider the following best practices:

Keep your modules focused and modular, encapsulating specific functionality or resources.
Use semantic versioning for your modules, and reference specific versions in your Terraform code to ensure consistency and reproducibility.
If your modules repository doesn’t require Terrateam operations, disable them by adding the following to your .terrateam/config.yml in the repository’s default branch:
```
enabled: false
```
Use Terrateam’s when_modified directive to configure dependencies between your modules and the directories that use them, ensuring that changes to modules trigger the necessary plan and apply operations.
Secure your private modules repositories by using SSH deploy keys and GitHub Actions secrets, and regularly rotate these keys to maintain security.
Organize your repository structure in a clear and logical manner, separating modules and their respective directories.