Terraform Registry Cache

On this page

Introduction
What is a Terraform Registry Cache?
How does this work exactly?
Requirements
Nginx Configuration
Terraform Configuration
Example Usage
Conclusion

Introduction

I recently came across an r/terraform post about a GitHub project called terraform-registry-proxy.

This inspired me to create a similar project strictly using Nginx.

What is a Terraform Registry Cache?

By default, Terraform will pull down provider artifacts from https://releases.hashicorp.com. For the typical user this behavior is acceptable. For environments that are air gapped or require a local caching server, we can do better.

How does this work exactly?

Nginx will proxy requests for the following endpoints:

Response bodies are rewritten to update where provider artifacts are served from. Terraform will use the rewritten URL to fetch the artifact from the registry cache (Nginx).

Nginx will serve back a response from its local cache if possible and fallback to the official Hashicorp servers.

Requirements

An Nginx server with SSL configured. The Terraform CLI must be able to trust the certificate.
Two subdomains:
1. terraform-registry.example.com
2. terraform-releases.example.com

Nginx Configuration

This is an example nginx.conf. Modify as you see fit.

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    gzip  on;

    proxy_headers_hash_bucket_size 128;
    proxy_cache_path /tmp/terraform-cache levels=1:2 keys_zone=terraform_cache:10m max_size=10g inactive=7d use_temp_path=off;

    # Custom log format so we can validate cache HITS and MISSES
    log_format custom_cache_log '[$time_local] [Cache:$upstream_cache_status] [$host] [Remote_Addr: $remote_addr] - $remote_user - $server_name to: $upstream_addr: "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" ';

    # terraform-registry.example.com
    server {
        listen       443 ssl;
        server_name  terraform-registry.example.com;
        resolver 8.8.8.8 ipv6=off;
        access_log  /var/log/nginx/terraform-registry-cache.log custom_cache_log;

        # SSL
        ssl_certificate "/etc/letsencrypt/live/terraform-registry.example.com/fullchain.pem";
        ssl_certificate_key "/etc/letsencrypt/live/terraform-registry.example.com/privkey.pem";
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout 10m;
        ssl_prefer_server_ciphers on;

        location / {
            # Pull from the local cache then https://registry.terraform.io/
            proxy_cache terraform_cache;
            proxy_cache_use_stale error timeout;
            proxy_pass https://registry.terraform.io/;
            proxy_set_header Accept-Encoding "";
            proxy_set_header X-Forwarded-For $remote_addr;
            tcp_nodelay on;
            # Rewrite releases.hashicorp.com payloads so the Terraform CLI pulls artifacts from terraform-releases.example.com
            sub_filter_types application/json;
            sub_filter 'releases.hashicorp.com' 'terraform-releases.example.com';
        }
    }

    # terraform-releases.example.com
    server {
        listen       443 ssl;
        server_name  terraform-releases.example.com;
        resolver 8.8.8.8 ipv6=off;
        access_log  /var/log/nginx/terraform-releases-cache.log custom_cache_log;

        # SSL
        ssl_certificate "/etc/letsencrypt/live/terraform-releases.example.com/fullchain.pem";
        ssl_certificate_key "/etc/letsencrypt/live/terraform-releases.example.com/privkey.pem";
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout 10m;
        ssl_prefer_server_ciphers on;

        location / {
            # Pull from the local cache then https://releases.hashicorp.com/
            proxy_cache terraform_cache;
            proxy_cache_use_stale error timeout;
            proxy_pass https://releases.hashicorp.com/;
            proxy_set_header Accept-Encoding "";
            proxy_set_header X-Forwarded-For $remote_addr;
            tcp_nodelay on;
        }
    }
}

Terraform Configuration

Inside your main.tf, set the global source address against the provider:

terraform {
  required_providers {
    aws = {
      source = "terraform-registry.example.com/hashicorp/aws"
    }
  }
}

Example Usage

Inside the directory with your main.tf changes:

$ terraform init
$ terraform init

josh@ringo:~/$ tail -f /var/log/nginx/terraform-registry-cache.log
[08/Apr/2022:12:44:39 +0000] [Cache:MISS] [registry.terrateam.io] [Remote_Addr: 213.93.7.141] - - - registry.terrateam.io to: 151.101.86.49:443: "GET /.well-known/terraform.json HTTP/1.1" 200 73 "-" "HashiCorp Terraform/1.1.7 (+https://www.terraform.io)"
[08/Apr/2022:12:44:39 +0000] [Cache:MISS] [registry.terrateam.io] [Remote_Addr: 213.93.7.141] - - - registry.terrateam.io to: 151.101.86.49:443: "GET /v1/providers/hashicorp/aws/versions HTTP/1.1" 200 96922 "-" "Terraform/1.1.7"
[08/Apr/2022:12:44:39 +0000] [Cache:MISS] [registry.terrateam.io] [Remote_Addr: 213.93.7.141] - - - registry.terrateam.io to: 151.101.86.49:443: "GET /v1/providers/hashicorp/aws/4.9.0/download/linux/amd64 HTTP/1.1" 200 8598 "-" "Terraform/1.1.7"
[08/Apr/2022:12:44:55 +0000] [Cache:HIT] [registry.terrateam.io] [Remote_Addr: 213.93.7.141] - - - registry.terrateam.io to: -: "GET /.well-known/terraform.json HTTP/1.1" 200 73 "-" "HashiCorp Terraform/1.1.7 (+https://www.terraform.io)"
[08/Apr/2022:12:44:55 +0000] [Cache:HIT] [registry.terrateam.io] [Remote_Addr: 213.93.7.141] - - - registry.terrateam.io to: -: "GET /v1/providers/hashicorp/aws/versions HTTP/1.1" 200 96882 "-" "Terraform/1.1.7"
[08/Apr/2022:12:44:55 +0000] [Cache:HIT] [registry.terrateam.io] [Remote_Addr: 213.93.7.141] - - - registry.terrateam.io to: -: "GET /v1/providers/hashicorp/aws/4.9.0/download/linux/amd64 HTTP/1.1" 200 8591 "-" "Terraform/1.1.7"
josh@ringo:~/terraform-registry-cache$

josh@ringo:~/$ tail -f /var/log/nginx/terraform-releases-cache.log
[08/Apr/2022:12:44:42 +0000] [Cache:MISS] [releases.terrateam.io] [Remote_Addr: 213.93.7.141] - - - releases.terrateam.io to: 151.101.85.183:443: "GET /terraform-provider-aws/4.9.0/terraform-provider-aws_4.9.0_linux_amd64.zip HTTP/1.1" 200 53259564 "-" "Terraform/1.1.7"
[08/Apr/2022:12:44:58 +0000] [Cache:HIT] [releases.terrateam.io] [Remote_Addr: 213.93.7.141] - - - releases.terrateam.io to: -: "GET /terraform-provider-aws/4.9.0/terraform-provider-aws_4.9.0_linux_amd64.zip HTTP/1.1" 200 53259564 "-" "Terraform/1.1.7"
josh@ringo:~/terraform-registry-cache$

You’ll notice I ran the init command two times.

This is to show that the first time Nginx served the request produced a Cache:MISS. The second time Nginx was able to pull from the local cache resulting in a Cache:HIT.

Conclusion

This Nginx setup could easily be integrated against environments that don’t have a direct connection to the Internet or company policy requiring the use of proxy servers.