In this post we will compare GitHub Actions to Terrateam for managing a Terraform workflow. We’ll show that Terrateam provides a safer and more robust solution than using GitHub Actions alone.
Terrateam uses GitHub Actions to execute Terraform. However, the Terrateam backend provides the necessary functionality to ensure changes are applied safely.
We will compare GitHub Actions to Terrateam across the following:
Terrateam uses GitHub Actions, so one might wonder: what is this blog post about?
Almost no decision logic happens in the action. Instead, Terrateam has a backend service that listens to GitHub events, evaluates those events, then decides whether or not to run the GitHub Action.
The standalone GitHub Actions solution compared in this post is attempting to solve the same problem solely with GitHub Actions.
Therefore, when this post refers to “standalone GitHub Action” or just “GitHub Actions”, it means a Terraform automation workflow implemented using solely GitHub Actions.
The purpose of this post is not to dismiss a standalone GitHub Actions workflow for Terraform automation. This approach can take a team very far. However, there is a trade-off. As the complexity of a workflow grows and the team grows, a GitHub Actions solution starts to show its limitations. Some workflows are simply not possible using GitHub Actions without giving up on safety. A complicated GitHub Action is a piece of software that needs to be maintained like any other software and that is a burden a team should take into account.
Terrateam is able to simultaneously scale with the workflow and team while maintaining safety, robustness, and configurability. Because Terrateam is a service, users benefit from improvements and bug fixes without having that maintenance burden.
The basis for comparing Terrateam to GitHub Actions is the blog post: Elevate Your Terraform Workflow With GitHub Actions by Andrew Walker. This a great blog post that develops a GitHub Action that supports linting, planning, and applying. He also describes how to use GitHub Branch Protection to give some safety guarantees.
Terrateam excels at safety. Being a service, it easily tracks the state of the changes in the repository, which is imperative to ensuring changes are applied safely.
The GitHub Actions solution does provide some guarantees, but they are not nearly as strong as the Terrateam guarantees.
apply
if the plan does not
reflect the latest code.apply
after merge and uses Branch
Protection to control when a merge can be performed.apply
before or after a merge. Supports
configurable requirements around when an apply
can be performed.apply
has finished. In this case, multiple apply
operations would
execute at the same time, but with locking of Terraform state enabled, one
apply
would fail. There is no way to execute the failed apply
without
another code change.apply
is executing at a time. In
the case that a second pull request is merged before the previous apply
has finished, Terrateam will also prevent any future apply
from being
performed until the second pull request has been successfully applied.Terrateam, being a backend service, is able to make decisions using more information than just what is available in the pull request.
For example, Terrateam will fetch sensitive configuration, such as permissions, from the default branch rather than the branch in the pull request.
Terrateam can also verify that the GitHub Action workflow file is correct. A standalone GitHub Action workflow will blindly run whatever is in the branch without any external verification or validation that it is running the intended code.
To some degree, this is an unfair comparison.
Terrateam is a backend service that uses GitHub Actions to execute Terraform in a trusted environment. Being a backend service, Terrateam is written in a programming language with a transactional database.
On the other hand, GitHub Actions can call other programs specified by a workflow written in YAML which limits what can be expressed.
The theme for this category is the following:
Most things are possible in a GitHub Action but the devil is in the details. Implementation can often times be challenging and brittle.
plan
or apply
operation.
There are many situations where they may fail, not due to an error in the
code, such that retrying will resolve it.terrateam plan
or terrateam apply
on the pull request.
Terrateam also provides actionable feedback for some errors, and is adding
more on each release.plan
or apply
without a code change. In the
case of an outage, this is a bad experience to have to perform code changes
when needing to re-run an apply
.terrateam unlock
in a pull request will remove
any locks that pull request owns. This should not be part of everyday
usage, but in an emergency the system should not be an unnecessary
hindrance. Terrateam also supports manually re-running a plan
or apply
.
In the case of an outage, if an apply
fails due to reasons outside of the
code, it is trivial to re-run.This is where Terrateam really shines compared to GitHub Actions.
A goal of Terrateam is to fit into its users’ workflow, not impose a workflow on its users. This makes Terrateam very configurable. While it is very configurable, the default behaviors of Terrateam work out-of-the-box for most use-cases, requiring very little upfront setup. Reasonable defaults are configured against Terraform repositories with safety as the number one priority and usability as a close second.
For sophisticated workflows, some configuration will be required. Terrateam supports and continues to support more complex workflows through configurations such as hooks, workflows, tagging, and directory globbing.
Standalone GitHub Actions workflows don’t have any runtime Terraform defaults in place requiring lots of manual setup and configuration for even a simple workflow. Additionally, standalone GitHub Actions workflows don’t have off-the-shelf features to satisfy complex Terraform workflow requirements. Writing custom scripts and injecting other GitHub Actions into a workflow file are typical solutions which can lead to more complexity and fragility.
See how stark the difference is:
Before running any change with Terraform, lots of teams often times want to lint
the new code. The Terraform CLI makes this easy with the terraform fmt
command.
To configure Terrateam to run terraform fmt
:
workflows:
- tag_query: ""
plan:
- type: run
cmd: ["terraform", "fmt", "-check", "-diff", "."]
capture_output: true
- type: init
- type: plan
The configuration in the GitHub Action is a fair bit longer:
name: Plan / Test On PR
on:
pull_request:
branches:
- main
jobs:
lint:
name: Lint
runs-on: ubuntu-20.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.9
- name: Run terraform fmt check
run: terraform fmt -check -diff -recursive ./terraform
But these experiences are not the same. On a successful plan, Terrateam only provides the plan output on a pull request comment. It’s only when something fails that Terrateam shows more details in the comment. In the case of the GitHub Action, if it fails, one must go to the action output to investigate.
Additionally, what if we only want to run fmt
against specific directories?
In Terrateam this is easy using tagging
. The below configuration will tag any
file under the production
directory with the production
tag and only run
fmt
on those files.
dirs:
production/**:
tags: [production]
workflows:
- tag_query: production
plan:
- type: run
cmd: ["terraform", "fmt", "-check", "-diff", "."]
capture_output: true
- type: init
- type: plan
For this workflow, we’d like to run the following steps for each environment
(dev
, stage
, prod
):
To configure Terrateam:
Modify the Terrateam configuration .terrateam/config.yml
:
workflows:
- tag_query: ""
plan:
- type: run
cmd: ["terraform", "fmt", "-check", "-diff", "."]
capture_output: true
- type: env
name: TF_VAR_allowed_account_id
cmd: ["echo", "$ALLOWED_ACCOUNT_ID" ]
- type: init
- type: plan
To perform this in the GitHub Action:
name: Plan / Test On PR
on:
pull_request:
branches:
- main
jobs:
lint:
name: Lint
runs-on: ubuntu-20.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.9
- name: Run terraform fmt check
run: terraform fmt -check -diff -recursive ./terraform
plan:
name: Plan
env:
TF_VAR_allowed_account_id: ${{ secrets.ALLOWED_ACCOUNT_ID }}
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
path:
- dev
- stage
- prod
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.9
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: us-east-1
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- name: Initialize Terraform
run: |
cd terraform/${{ matrix.path }}
terraform init -input=false
- name: Plan Terraform
id: plan
continue-on-error: true
run: |
cd terraform/${{ matrix.path }}
terraform plan -input=false -no-color
- name: Post Plan to GitHub PR
uses: mshick/add-pr-comment@v1
with:
allow-repeats: true
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]'
message: |
## ${{ matrix.path }} plan
${{ steps.plan.outputs.stdout || steps.plan.outputs.stderr }}
But, again, these experiences are not the same. Some key differences:
dev
,
stage
, prod
) on every pull request, even if no change has been made to
those directories, or even to a Terraform file. This means updating the
README will result in a plan and apply of all environments.The blog post goes on to show various improvements around planning, such as
removing extraneous output and modifying the output to fit the diff
syntax.
All this comes at the cost of more lines of YAML.
Terrateam does this already, and more, combining all of the outputs into a concise single comment.
The output of Terrateam requires no configuration:
and the output of GitHub Action:
The configuration to achieve this output:
- name: Put Plan in Env Var
run: |
PLAN=$(cat plan.txt)
echo "PLAN<<EOF" >> $GITHUB_ENV
echo "$PLAN" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
- name: Plan Terraform
id: plan
continue-on-error: true
run: |
cd terraform/${{ matrix.path }}
terraform plan -input=false -no-color -out=tfplan \
&& terraform show -no-color tfplan
- name: Reformat Plan
run: |
echo '${{ steps.plan.outputs.stdout || steps.plan.outputs.stderr }}' \
| sed -E 's/^([[:space:]]+)([-+])/\2\1/g' > plan.txt
- name: Put Plan in Env Var
run: |
PLAN=$(cat plan.txt)
echo "PLAN<<EOF" >> $GITHUB_ENV
echo "$PLAN" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
- name: Post Plan to GitHub PR
uses: mshick/add-pr-comment@v1
with:
allow-repeats: true
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]'
message: |
## ${{ matrix.path }} plan
${{ env.PLAN }}
Terrateam supports pre-merge and post-merge apply workflows. Terrateam supports
running operations both on pull request create, update, and merge, but also
through commenting on the pull request terrateam plan
or terrateam apply
.
In a pre-merge workflow, Terrateam can also automatically merge the change after
all changes have been applied successfully.
The GitHub Actions workflow only supports post-merge apply, and it has to be this way to maintain the few safety guarantees it offers. As mentioned before, this solution uses Branch Protection rules to ensure that all pull requests have the latest changes before merging, which is how applies are guaranteed to always have the latest changes. But this means that there cannot be a pre-merge workflow because then an apply could be performed while the pull request is not mergeable.
The inability to perform an apply
without a code change is a serious
limitation for the GitHub Action workflow. Our customers commonly see an apply
fail for various reason unrelated to the code, for example an IAM permission not
being visible when the change using it executes.
Finally, the GitHub Action workflow does not store the plan between the plan
and apply
, and instead it performs a second plan before applying. The
consequence is the apply
that is performed may not be the plan that was
reviewed in the pull request. How big of a risk this is depends on your
organization, but this is a silent failure mode that will not be visible until
it’s too late.
Terrateam stores the plan and applies exactly that plan, one is guaranteed that the plan they reviewed is the plan they applied.
For the sake of completeness, here is what the Terrateam configuration looks like to in order to match the apply-after-merge workflow of the Github Action:
when_modified:
autoapply: true
workflows:
- tag_query: ""
plan:
- type: run
cmd: ["terraform", "fmt", "-check", "-diff", "."]
capture_output: true
- type: env
name: TF_VAR_allowed_account_id
cmd: ["echo", "$ALLOWED_ACCOUNT_ID" ]
- type: init
- type: plan
And here is what the GitHub Actions configuration has grown to:
name: Plan / Apply On Merge
on:
push:
branches:
- main
jobs:
inform_about_apply:
name: Inform About Apply
runs-on: ubuntu-20.04
steps:
- name: Inform on PR that Apply is Running
uses: mshick/add-pr-comment@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]'
message: |
***Running terraform apply***
Results will display here momentarily...
plan_and_apply:
name: Plan and Apply
env:
TF_VAR_allowed_account_id: ${{ secrets.ALLOWED_ACCOUNT_ID }}
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
path:
- dev
- stage
- prod
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.9
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: us-east-1
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- name: Initialize Terraform
run: |
cd terraform/${{ matrix.path }}
terraform init -input=false
- name: Plan Terraform
id: plan
continue-on-error: true
run: |
cd terraform/${{ matrix.path }}
terraform plan -input=false -no-color -out=tfplan \
&& terraform show -no-color tfplan
# Sed is taking all lines that begin with one or more spaces followed by a `+` or `-`.
# It stores the amount of spaces in `\1` and the +/- in `\2`.
# Then replace that portion of the line with `\2\1` (+/- followed by the number of matched spaces).
- name: Reformat Plan
if: steps.plan.outcome == 'success'
run: |
echo '${{ steps.plan.outputs.stdout || steps.plan.outputs.stderr }}' \
| sed -E 's/^([[:space:]]+)([-+])/\2\1/g' > plan.txt
- name: Put Plan in Env Var
if: steps.plan.outcome == 'success'
run: |
PLAN=$(cat plan.txt)
echo "PLAN<<EOF" >> $GITHUB_ENV
echo "$PLAN" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
- name: Apply Terraform
if: steps.plan.outcome == 'success'
id: apply
continue-on-error: true
run: |
cd terraform/${{ matrix.path }}
terraform apply \
-input=false \
-no-color \
tfplan
- name: Post Plan and Apply to GitHub PR
if: steps.plan.outcome == 'success' && steps.apply.outcome == 'success'
uses: mshick/add-pr-comment@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]'
message: |
Applying **${{ matrix.path }}**:
${{ env.PLAN }}
${{ steps.apply.outputs.stdout }}
- name: Post Plan Failure
if: steps.plan.outcome == 'failure'
uses: mshick/add-pr-comment@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]'
message: |
Plan failed for **${{ matrix.path }}**:
${{ steps.plan.outputs.stderr }}
- name: Post Apply Failure
if: steps.apply.outcome == 'failure'
uses: mshick/add-pr-comment@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]'
message: |
Apply failed for **${{ matrix.path }}**:
${{ steps.apply.outputs.stderr }}
Terrateam supports even more features for pull requests:
Cost estimation and static analysis can be added to the GitHub Action solution, but at the cost of more YAML, that has already grown quite large.
RBAC, on the other hand, is simply not feasible to implement in this way. Any RBAC solution in this YAML can be turned of in a pull request, circumventing its own protection.
One might think this comparison is unfair. The GitHub Actions solution implements some of what Terrateam does using only GitHub Actions, and this blog post has only shown the configuration file for Terrateam. Not only that, but the GitHub Actions solution is in a few hundred lines of YAML, surely Terrateam is more code than that.
And Terrateam is more lines of code. Terrateam is supporting a more diverse
range of workflows. It’s providing stronger safety guarantees with more
flexibility. But consider a small change to one’s workflow: wanting to apply
before merge. This would require a significant change to the GitHub Actions
YAML, and may not be possible to do safely without running an external service.
Standing up a homegrown solution with GitHub Actions can quickly lead to a
tangled web of workflow steps with very little safety or visibility.
And that is why we don’t dismiss a standalone GitHub Actions solution outright. A lot can be accomplished with it. Despite the strong words in this blog post, if the GitHub Actions workflow matches your needs, use it. If your workflow outgrows it, switching to Terrateam is as easy as removing the existing GitHub Actions workflows and replacing it with Terrateam.
With Terrateam, we have a lot of out-of-the-box functionality that will allow your team to make safe Terraform changes with very little required configuration.
Please give Terrateam a shot by following our getting-started guide.