Locks and concurrency

Terrateam’s locks and concurrency feature ensures that changes to your infrastructure are safely applied by preventing concurrent modifications to the same resources. Locks guarantee that only a single change to a resource can be applied in a pull request at any given time, avoiding conflicts and maintaining consistency between your Terraform configuration and the real-world state of your infrastructure.

Concurrent collaboration

When multiple team members are working on the same Terraform project, locks play a crucial role in coordinating their efforts and preventing conflicting changes.

Here’s how Terrateam’s locks help facilitate collaboration among team members:

1. Coordination: When a team member applies a change to a resource in a pull request, Terrateam acquires a lock on that resource. This lock ensures that no other team member can concurrently apply changes to the same resource, preventing conflicts and inconsistencies.

2. Invalidation: If a team member attempts to apply a change to a resource that is already locked by another apply operation, Terrateam will deny the change. Once the lock is released, the change can be re-planned and applied.

3. Consistency: By preventing concurrent modifications to the same resources, locks help maintain consistency between your Terraform configuration and the actual state of your infrastructure. This consistency is crucial when multiple team members are working on the same project.

What is a lock?

A lock is a mechanism that ensures only a single change to a resource can be applied in a pull request at any given time. When a lock is acquired on a resource, changes to that resource in other pull requests cannot be concurrently applied.

Note

• A lock is owned by a pull request.
• A lock applies to the dirspace (a combination of a directory and a workspace).

When is a lock acquired?

By default, a lock is acquired when executing an apply operation or merging a pull request without applying the changes. The lock is acquired on the dirspaces targeted in the pull request.

Note

• A lock is acquired regardless of the success or failure of an apply.
• Once a single dirspace has been applied in a pull request, that pull request owns locks on all targeted dirspaces associated with the pull request.
• A lock is only released after a successful apply and merge.
• A lock can be forcefully unlocked by commenting terrateam unlock on the pull request.

When is a lock not acquired?

If a pull request is created but never applied or merged, no lock is acquired. Locks are only acquired when executing an apply or merging a pull request without applying the changes.

Example 1: applying

Consider a pull request with changes in dir1 and dir2 (using only the default workspace):

1. Alice runs terrateam plan to plan all changes in the pull request.

2. Alice runs terrateam apply dir:dir1 to apply only the changes in dir1.

The pull request now owns locks on the following dirspaces:

• (dir1, default)
• (dir2, default)

Example 2: merging

Consider the same pull request from Example 1. Instead of executing a terrateam apply, Alice merges the pull request with unapplied changes.

The pull request now owns locks on the following Dirspaces:

• (dir1, default)
• (dir2, default)

When is a lock released?

A lock is released when a change has been both applied and merged.

Unlocking

In certain scenarios, you may want to force an unlock. To explicitly remove a lock, comment terrateam unlock in the pull request that owns the lock.

• Executing terrateam unlock only applies to locks acquired prior to performing the unlock.
• Performing a merge or apply after an unlock will acquire any necessary locks again.

Caution

The terrateam unlock command is very powerful.

• Executing this command will unlock all locks in the pull request.
• If terrateam apply is issued in multiple pull requests, Terrateam will ensure that only a single apply is executed.
• If terrateam unlock is issued during an in-flight apply operation, Terrateam will unlock all locks in the pull request and subsequently allow an additional apply operation in a separate pull request. If both pull requests have changes against the same set of Terraform resources, the additional apply could result in unintended consequences.

With great power comes great responsibility

Benefits of locks

Drift protection

Locks are released when a change is both merged and applied to ensure that your default branch matches the real-world state of your infrastructure.

Consider a pull request where a change is applied, but the pull request is accidentally closed instead of merged. Without the pull request merged, future pull requests would not be created on top of the applied change. With locks, future changes to the same dirspace could not be applied because the closed pull request still owns the lock. The change would need to be explicitly unlocked or the pull request would have to be re-opened and merged.

Preventing conflicts

Locks help prevent conflicts when multiple pull requests modify the same Terraform resource. Without locks, this can cause the state defined in your configuration to be different from the real-world state of your infrastructure.

Consider two open pull requests without the ability to lock:

• PR1 created by Alice
• PR2 created by Bob

1. Alice and Bob both make changes to dir1.

2. Alice applies her changes in PR1 but does not merge.

3. Bob applies his change in PR2, changing the same resources that Alice applied against.

4. Bob merges his pull request before Alice merges her pull request.

Examining the repository, one could assume that the latest changes in the default branch match the real-world state of the infrastructure (Alice’s changes). However, in reality, the real-world state matches Bob’s changes.

With locks, Terrateam guarantees that changes are merged in the order they are applied, preventing such conflicts.

Merging without applying

Terrateam will not prevent users from merging a pull request without issuing a terrateam apply. However, Terrateam will recognize this scenario, and users will not be able to apply any new changes until locks have been released from the merged pull request that was not applied.

Locking policies

The lock_policy option under workflows in your Terrateam configuration file instructs Terrateam under what situations it should acquire a lock.

• strict (default): Acquire a lock if a user comments terrateam apply in the pull request or the change is merged. The lock is released when the complimentary operation is performed. Recommended for production environments.
• apply: Acquire a lock only if the directory has been applied in Terrateam (terrateam apply). The lock is released once the change is merged.
• merge: Acquire a lock only if the directory has been merged. The lock is released when the change is applied (terrateam apply). Useful for development pull requests that are used as a playground and closed when done, rather than merging.
• none: Never acquire a lock.

Example scenario

Consider the following scenario:

• A repository named repo
• Alice creates pull request PR1 against repo
• Bob creates pull request PR2 against repo
• Both pull requests have a change in directory dir (using only the default workspace)

1. Alice applies her change by commenting terrateam apply but does not merge the pull request. PR1 now owns a lock on the Dirspace (dir, default).

2. Bob tries to execute terrateam apply on PR2 but receives an error stating that the plan cannot be applied because PR1 owns the lock.

3. Alice merges PR1. The lock is released, and a terrateam apply can be executed against PR2.

4. Bob tries to execute terrateam apply on PR2 again but receives an error stating that the plan has been invalidated. The terrateam apply Alice executed in PR1 changed the state, marking Bob’s plan as invalid. Bob now needs to run terrateam plan again before running terrateam apply.

Best practices

When working with locks in a team environment, consider the following best practices:

• Communicate and coordinate with your team members when working on shared resources to avoid conflicts and delays.
• Regularly review the lock status and address any long-standing locks that may be blocking other team members’ progress.
• If necessary, use Terrateam’s override mechanisms (e.g., terrateam unlock) judiciously and with caution, ensuring that all team members are aware of the implications.
• Establish clear guidelines and processes within your team for managing locks, resolving conflicts, and ensuring smooth collaboration.