Announcing Layered Runs

Layered Runs: Wield the Thor’s Hammer for Complex Infrastructure

In complex infrastructure setups, some changes require multiple rounds of planning and applying to complete. This could be due to regulatory requirements, such as needing to release a change to a development or staging environment before production. It could also be that the output of one directory is necessary before planning and applying another directory.

Previously in Terrateam, this could be accomplished by manually triggering a plan and apply process using tag queries to target those directories. For example, if the application directory depends on changes in the database, we would make a pull request and then comment: terrateam plan database followed by terrateam apply database, then terrateam plan app followed by terrateam apply app. However, this approach had several drawbacks:

1. The user needs to manually know and execute these relationships. As repositories grow in complexity, this becomes harder to manage accurately.

2. We want to trigger the application run even if no files have changed in the application directory. In Terrateam, we use a list of file patterns that match pull request diffs to trigger directory runs. This would require configuring the application directory to run whenever any database files are modified.

3. We want to chain these changes together automatically. If the database directory needs to run after networking configuration changes, the database file patterns need to trigger based on networking changes. Similarly, the application directory needs to trigger on both networking and database changes, even though it primarily relies on the database.

The reason you’re using Terrateam is so you don’t have to worry about these details. With the new Layered Runs feature, you no longer need to.

Layers are defined with a new attribute in when_modified called depends_on. This is a tag query, and if a change in the pull request matches this tag query, the directory will be triggered. The process continues until no more layers remain to be triggered.

How It Works

Let’s imagine we want to implement the following setup: our infrastructure is divided into three layers—network, database, and application. We want database to trigger if network changes, and we want application to trigger if database changes.

Here’s an example configuration (note that we don’t need to configure the network directory, as it will use the default configuration):

dirs:
  database:
    when_modified:
      depends_on: 'dir:network'
  application:
    when_modified:
      depends_on: 'dir:database'

Representation of dependency layers:

network
  ├── main.tf
database
  ├── main.tf
application
  ├── main.tf

If network changes, Terrateam will guide the user through planning and applying it, then move to planning and applying database, followed by application. The user doesn’t need to know the layers, their number, or their order—Terrateam handles it all. You simply tell Terrateam when you’re ready to plan and apply the next layer.

Enforcing Development Before Production

Another common requirement is ensuring the dev environment is planned and applied before the prod environment. In this example, we’ve defined all of our shared infrastructure in modules consumed by both dev and prod. The directory structure is as follows:

.
├── modules
│   ├── one
│   │   └── main.tf
│   ├── three
│   │   └── main.tf
│   └── two
│       └── main.tf
├── dev
│   └── main.tf
└── prod
    └── main.tf

We can enable the indexer in Terrateam’s configuration, which will automatically identify modules and which directories use them. Then, we specify that prod depends on dev:

indexer:
  enabled: true

dirs:
  dev:
    tags: [dev]
  prod:
    when_modified:
      depends_on: dev

Now, when a user makes changes to one of the modules, both dev and prod are triggered by the module change. However, prod will only run after dev.

depends_on: Tag Queries Everywhere

Terrateam leverages tags and tag queries extensively, allowing for flexible and powerful configuration. The depends_on attribute is no different. In the previous examples, we’ve only shown a single dependency per directory. But what if the application directory needs to depend on both database and network? It’s as simple as updating the tag query:

dirs:
  database:
    when_modified:
      depends_on: 'dir:network'
  application:
    when_modified:
      depends_on: 'dir:database or dir:network'

Why Use or Instead of and?

You might wonder why we use or for dependencies. Wouldn’t it be more accurate to say that application depends on both network and database? Actually, the term depends_on is a bit misleading—what it really means is “trigger this directory if any directory matching this tag query is triggered.” Hence, or is the correct operator.

Using relative_dir

The depends_on attribute comes with a useful predicate not available in other tag queries: relative_dir. This allows directories to be referenced relative to the directory being checked. For example:

depends_on: 'relative_dir:../network'

Unlocking Potential

What’s truly exciting about Layered Runs is how a small configuration change (just adding a few depends_on tags) can encode your entire infrastructure’s execution logic. Imagine a worst-case scenario: your infrastructure is wiped out by a ransomware attack, and you need to rebuild everything from scratch. It’s been a while since you set up the foundational infrastructure, and you’re unsure about the ordering. No worries—if your Terrateam configuration is up-to-date, just let Terrateam take over and reb…

In less dramatic scenarios, like deploying similar environments across different regions, Terrateam can still save you time and effort. The directory structure and services may be the same between environments, but the details vary. By encoding these relationships in Terrateam, it will guide you through the proper order of operations, no matter the complexity.

If all environments follow the same structure, you can even define the configuration once, and as new environments are added, Terrateam will automatically handle them correctly.

Example: Multi-Region Environments

We have several environments, each with a series of services: networking, block_storage, database, and application. The dependency ordering is as follows, from least to most dependent:

1. networking
2. database and block_storage
3. application

Here’s the layout of the repository:

.
└── envs
    ├── asia
    │   ├── application
    │   │   └── main.tf
    │   ├── block_storage
    │   │   └── main.tf
    │   ├── database
    │   │   └── main.tf
    │   └── networking
    │       └── main.tf
    ├── europe
    │   ├── application
    │   │   └── main.tf
    │   ├── block_storage
    │   │   └── main.tf
    │   ├── database
    │   │   └── main.tf
    │   └── networking
    │       └── main.tf
    ├── us-east
    │   ├── application
    │   │   └── main.tf
    │   ├── block_storage
    │   │   └── main.tf
    │   ├── database
    │   │   └── main.tf
    │   └── networking
    │       └── main.tf
    └── us-west
        ├── application
        │   └── main.tf
        ├── block_storage
        │   └── main.tf
        ├── database
        │   └── main.tf
        └── networking
            └── main.tf

We can create a single configuration that will automatically handle any new environments:

dirs:
  envs/*/application/*.tf:
    when_modified:
      depends_on: 'relative_dir:../database or relative_dir:../block_storage'
  envs/*/block_storage/*.tf:
    when_modified:
      depends_on: 'relative_dir:../networking'
  envs/*/database/*.tf:
    when_modified:
      depends_on: 'relative_dir:../networking'

Conclusion

Terrateam’s new Layered Runs functionality unlocks significant potential for encoding your infrastructure as code (IaC). Not only can you define the infrastructure itself as code, but you can also encode the relationships and dependencies between components. Committed to GitOps, all of your configurations are stored in code.