Remote state and working in a team

Where you are

At the end of Lesson 7, you created a real server on DigitalOcean, then destroyed it. Its state, the record of what Terraform made, lived in a terraform.tfstate file in your terraform-cloud folder, on your own computer.

For one person learning, that's fine. For anything real, it's a problem. If your laptop is lost, so is Terraform's record of your infrastructure, and Lesson 3 showed what Terraform does without it. A colleague can't make a change, because they don't have the state. And if you both somehow had copies and applied at the same time, each of you would be working from a different idea of what exists. This lesson moves state somewhere safe and shared.

Locking: one change at a time

A single-person bathroom has a lock for a reason: two people at once doesn't end well, and the lock means the second one waits. Terraform's state works the same way. Whenever a command might change the state, Terraform first takes a state lock, and anyone else who tries in the meantime is told to wait.

You can see it on your own computer, with the terraform-practice folder from Lessons 1 to 6. Open two terminals, and cd into terraform-practice in both. In the first, run:

terraform apply

and when it asks Enter a value:, don't answer: leave it waiting. Terraform is holding the lock while it waits for you. In the second terminal, run:

terraform plan

Error: Error acquiring the state lock

Error message: resource temporarily unavailable
Lock Info:
  ID:        3cf217ad-765d-59f6-451d-6476c2e2936b
  Path:      terraform.tfstate
  Operation: OperationTypeApply
  Who:       you@your-computer

followed by a few more lines, and an explanation that Terraform locks the state so it can't be written by two runs at the same time. The Lock Info says who has the lock, and what they're doing: here, an apply. The details will be your own. Go back to the first terminal and type no: the apply is cancelled, the lock is released, and the plan in the second terminal works if you run it again.

The message also mentions -lock=false, a way to skip the lock. Treat it the way you'd treat breaking into an occupied bathroom: if a lock is in the way, the right move is to find out whose run is holding it, not to ignore it.

Backends: where state lives

Where Terraform keeps the state is called the backend. Until now, you've used the local backend: a file in your folder. For a team, state goes somewhere everyone can reach, that's backed up, that supports locking, and that keeps it encrypted, since it holds secrets. Common choices:

• A storage bucket in your cloud, like Amazon S3, Google Cloud Storage, Azure Storage or DigitalOcean Spaces. You set up the bucket and its permissions yourself.
• HCP Terraform (formerly Terraform Cloud), HashiCorp's service. It stores the state, locks it, keeps every version, and can run Terraform for you as well.

This lesson uses HCP Terraform, because it needs no bucket to set up first, and it's what CivicGrid uses. When this lesson was written, HashiCorp's Free tier included up to 500 managed resources and unlimited users at no cost; HashiCorp has changed its plans before, so check the current details on its pricing page. This lesson manages one.

Moving a project to HCP Terraform

You'll move the terraform-cloud project from Lesson 7. It has nothing in it right now, since you destroyed the server, which makes it a clean start.

Step 1. Sign up and make an organization. Create a free account at app.terraform.io (opens in a new tab). When asked, create an organization: the name your team's projects live under, like your username. Organization names are unique across all of HCP Terraform, so you may need to add something to yours.

Step 2. Log in from your terminal.

terraform login

Terraform explains that it will save a token for app.terraform.io on your computer, and asks you to confirm with yes. It then opens a page in your browser where you create the token; copy it, and paste it at the prompt in your terminal. Like read -s, nothing appears as you paste. Terraform saves the token in a credentials file in your home folder, outside any project, so it never goes near your code.

Step 3. Point the project at HCP Terraform. In terraform-cloud/versions.tf, add a cloud block inside the terraform block, above required_providers, using your own organization name:

terraform {
  cloud {
    organization = "your-organization-name"

    workspaces {
      name = "terraform-cloud-practice"
    }
  }

  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.106"
    }
  }
}

A workspace is HCP Terraform's home for one set of state: one project, or one copy of it, like production. Then run:

terraform init

Terraform connects to HCP Terraform and creates the terraform-cloud-practice workspace. If your folder still had a local state file, it would offer to move it into the workspace; type yes if it asks. From now on, this project's state lives there, not in your folder.

Runs happen there too

Run terraform plan. This time it starts differently:

Running plan in HCP Terraform. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.

By default, HCP Terraform doesn't just store the state: it runs Terraform on its own servers, and streams the output back to your terminal. That matters for secrets. Your DIGITALOCEAN_TOKEN is an environment variable in your terminal, and the run isn't happening in your terminal. So the plan fails to authenticate with DigitalOcean, just like the no-token break-it in Lesson 7.

Step 4. Give the workspace the token. The token has to live where the run happens:

1. In HCP Terraform, open your organization, then the terraform-cloud-practice workspace, then Variables.
2. Add a variable. Choose the Environment variable category, with the key DIGITALOCEAN_TOKEN, and paste your DigitalOcean token as the value. (If you deleted it at the end of Lesson 7, create a new one the same way, with a short expiry.)
3. Tick Sensitive. Once saved, a sensitive value can't be read back, by you or anyone else, in the web interface or the API; it can only be replaced.

Run terraform plan again, and it works: one Droplet to add. Now:

terraform apply

Type yes in your terminal, as usual. Open the workspace in your browser while it runs: you'll see the run, the plan, who started it, and the apply's progress. When it's done, the outputs appear in your terminal, and in the workspace, with your server's URL; give the startup script a minute, and the page loads, as in Lesson 7.

Look around the workspace: the States tab keeps every version of the state, with the run that made it, so you can see exactly what changed and when. That's the team's shared memory, and it's encrypted, locked and backed up for you.

Then clean up, as in Lesson 7: run terraform destroy, type yes, and check the Droplets page in DigitalOcean is empty.

How teams change things

With shared state, the usual way a team changes infrastructure looks like the Git track's pull requests:

1. Someone changes the .tf files on a branch, and opens a pull request.
2. A plan runs for that change, and its output is shown on the pull request, so everyone can read exactly what would happen.
3. A reviewer reads the plan, not just the code, because the plan is what will really happen.
4. Once approved and merged, the change is applied, by a person or automatically.

HCP Terraform can do steps 2 and 4 for you, by connecting a workspace to a Git repository; that's called the VCS-driven workflow, where VCS means version control system. What you've used in this lesson, starting runs from your terminal, is the CLI-driven workflow, where CLI means command-line interface. Both use the same state, locks and variables.

Break it on purpose

Two runs at once. You saw the lock on your own computer earlier. In HCP Terraform, it works as a queue. Start terraform apply in the terraform-cloud folder, and leave it at the yes prompt. In a second terminal, in the same folder, start another terraform apply. Instead of an error, the second run waits its turn: open the workspace in your browser, and you'll see it listed behind the first. Type no in the first terminal, and the second one moves up; type no there too. (A plain terraform plan doesn't change anything, so HCP Terraform can run it without waiting.)

Not logged in. If you ever work on a computer where you haven't run terraform login, terraform init on a project with a cloud block stops with an error saying a token for app.terraform.io couldn't be found, and tells you to run terraform login. You don't need to make this happen; just recognise it when it does. It's the CLI telling you it doesn't know who you are, the same idea as a missing DigitalOcean token.

Delete the secret. In the workspace's Variables, delete DIGITALOCEAN_TOKEN, and run terraform plan. It fails to authenticate with DigitalOcean, even though nothing in your code changed and the token may still be in your terminal. The run happens in the workspace, so the workspace's settings are what count. Add the variable back, ticking Sensitive again.

When you're done, make sure terraform destroy has been run and the Droplets page is empty. You can keep the HCP Terraform account and workspace for Lesson 9, or delete the workspace under its Settings, then Destruction and Deletion. Delete the DigitalOcean token too, unless you're going straight on.

Where you are now

Your state is in a shared, encrypted, locked place, your runs happen there too, and your secret lives in the workspace instead of anyone's terminal. You've used every piece of Terraform a team relies on day to day. Lesson 9, the last, is about keeping a growing configuration tidy: modules for reuse, and the habits that stop a project from becoming a mess.

Check yourself

Key terms

• Remote state: state kept somewhere shared and safe, instead of a file on one computer.
• State lock: a lock taken by any run that might change the state, so two can't do it at once.
• Backend: where Terraform keeps the state, like a local file, a storage bucket, or HCP Terraform.
• HCP Terraform: HashiCorp's service for storing state and running Terraform. It used to be called Terraform Cloud.
• Organization: your team's space in HCP Terraform.
• Workspace: HCP Terraform's home for one set of state, with its runs and variables.
• Workspace variable: a setting or secret stored in the workspace, for runs that happen there.
• CLI-driven and VCS-driven workflows: starting runs from your terminal, or automatically from changes in a Git repository.

Go deeper

• HCP Terraform (opens in a new tab) documentation, including workspaces, variables and runs.
• State locking (opens in a new tab) in the Terraform documentation.
• HashiCorp's post on the current Free tier (opens in a new tab).