by Adam Brett

A Practical Introduction to Terraform

This article was published on Thursday, October 22, 2015 which was more than 18 months ago , this means the content may be out of date or no longer relevant. You should verify that the technical information in this article is still current before relying upon it for your own purposes.

It's been a good few months since I've written anything here (or anywhere), I've tried to respond to all of the emails I've gotten from readers, but my twitter presence has been somewhat lacking, so sorry about that.

The reason for my abscence has been a very busy year, including starting a new job as a DevOps Engineer for a major UK Energy company. The transition was a fairly natural one, as a Lead Developer I was focusing more on automating the delivery pipeline for the team, and increasing the overall quality and reliability of what we were delivering. Writing code had taken a backseat to the DevOps type stuff for a while (as can be seen by the type of things I've written about on here!), and my new role now focuses on that full time.

Forge

I've been using Laravel Forge for my side company, Voreti Ltd., pretty much since it was released, and it's awesome. I've only provisioned around 10 servers with it, each with up to about 8 apps or sites on, but found it a pleasure to use that entire time.

But with my current busy schedule in mind, I've not had time to do many side projects, and as a result have logged into Forge maybe 3 or 4 times in the last year.

As my annual renewal is up very soon, I can't really justify paying for it for another year, so I want to replicate that infrastructure in a nice automated way.

Laravel Forge is a fairly comprehensive solution, so my goal isn't to replicate it fully, but only the core functionality that I use. I also plan to improve some areas that I feel Forge is lacking, and luckily the tool-chain to do that has really matured over the last 12-18 months.

Requirements

The main features of Forge that I plan to replicate are:

  • Managing Digital Ocean Droplets
  • Provisioning VMs with a PHP Stack
  • Managing CRON entries
  • Managing Firewall rules
  • Creating new nginx virtual hosts
  • Automated deployments on git push to master

The things I plan to improve are:

  • Automatically creating a user per site for better access control and isolation
  • Automatically creating databases
  • Automatically updating DNS records in CloudFlare
  • Automated backups

Fortunately, I can do all of this with just two tools, which I'm already familiar with: Terraform and Ansible.

Terraform

Terraform1 is a tool for creating your underlying infrastructure, mostly virtual machines and networks, in the cloud provider of your choice. In their own words:

Terraform provides a common configuration to launch infrastructure — from physical and virtual servers to email and DNS providers. Once launched, Terraform safely and efficiently changes infrastructure as the configuration is evolved.

Simple file based configuration gives you a single view of your entire infrastructure.

Ansible

Ansible, you will probably already be familiar with, it takes over where Terraform leaves off, and allows you to provision the software on your servers in a reliable and consistent way.

Terraform

Installing

Installing both tools is fairly simple, so I won't cover that here. Head over to their respective websites23 and follow the instructions.

Creating Digital Ocean Droplets With Terraform

We'll start with creating the new servers using Terraform, as we need to create them before we can install anything on them.

Let's start by creating a new directory for our project, I'd recommend you add this to git and store it online somewhere private (I use BitBucket4), but I'll leave that up to you.

mkdir -p ~/Projects/forge && cd ~/Projects/forge

In our new project directory, we'll want two more directories, one to hold our Terraform Configuration Files, and another to hold our Ansible Playbooks.

mkdir {terraform,ansible}

Terraform works by loading all *.tf files from a single directory. There is no entry-point and no other naming requirements, if the file ends with .tf, Terraform will load and run it. .tf files can be in one of two formats, Hashicorp Configuration Language (HCL), or JSON. As we will be writing these by hand, we will be using the HCL format, as it's much easier to read and write. If you wanted to create an actual Forge clone, you could use Terraform as the backend, and have your application generate .tf files in JSON format, so that both Terraform and your application can understand them easily.

Currently in Forge I have three types of servers (but I'll only be creating two types for this tutorial):

  • Hosting servers

This is where I host one or more client websites, usually simple CMS or static websites.

  • App servers

These are for web-apps, they ususally only run a single web-application, but may have a few non-Forge services running like nodejs or elasticsearch.

We'll start by creating a file called servers.tf in our terraform directory, with the following content:

resource "digitalocean_droplet" "hosting" {
    count = "${var.servers_hosting_count}"
    image = "ubuntu-14-04-x64"
    name = "hosting-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "512mb"
    backups = true
    ipv6 = true
}

resource "digitalocean_droplet" "app" {
    count = "${var.servers_app_count}"
    image = "ubuntu-14-04-x64"
    name = "app-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "512mb"
    backups = true
    ipv6 = true
}

This is an example of HCL. The first thing you should know about HCL is that it's a declaration language, it describes something, like HTML does, so there is no logic or flow control included.

What we've done here is define two Resources. In Terraform, everything is a Resource. A Resource belongs to a Provider (which is usually the cloud service provider or cloud type e.g. AWS, Digital Ocean, OpenStack), and has two parameters:

  • Type
  • Name

The type is the first part, and relates to the type of resource you want Terraform to create, here, we're creating a droplet resource using the digitalocean Provider5. All Digital Ocean resources are prefixed with digitalocean_, and all Amazon Web Service resources are prefixed with aws_ and so on.

The types of resource you can create are very well documented on the Terraform website, and as Terraform is so new, they're constantly adding new ones. If you find Terraform doesn't support a type you need, you can write your own provider with it's own resources.

The second part of the resource is it's name. The name of a resource is arbitrary, but it must be unique amongst all other resources of it's Type (until we get to Modules later). That means you can have a digitalocean_droplet called "my-website", and an aws_instance called "my-website", but you can't have two aws_instance resources both called "my-website".

The third part of the resource, inside the curly braces {...} are the Properties of the resource. These are usually just strings that will get passed to the Provider's API behind the scenes, but can include very simple functions or variables, as I've done here.

Let's take a look at the first resource resource "digitalocean_droplet" "app" line-by-line, and try to break down what's happening:

resource "digitalocean_droplet" "hosting" { ... }

This is the definition of our resource, the name, as I said previously, has to be unique amongst resources of this type, in this Module, but can be pretty much anything you want. You will use this name to reference this Resource in other Resources later.

count = "${var.servers_hosting_count}"

Count is a magic Property and isn't usually included in the documentation for a Resource. Every Resource can have a count Property, and it just means "create this many resources with these properties". If you exclude the count Property, it will default to 1.

In our example, the value for count is set to a variable. So here we have an example of Variable Interpolation. All variables are global, and always start with the var. prefix. We'll look at adding the server_hosting_count variable in a minute, but it's always a good idea to make your count a variable, because when creating linked resources later, Terraform has no way of counting the number of resources it's created, so if you want to do something like create a DNS record for every server, you'll want to reference the count variable in both places so you get the right amount.

image = "ubuntu-14-04-x64"
name = "hosting-${format("%02d", count.index+1)}"
region = "lon1"
size = "512mb"
backups = true
ipv6 = true

The rest of our Properties are unique to this resource, as they usually are. They are well documented for each Resource on the Terraform website. Here, the image tells the Digital Ocean API what type of Droplet to create. The name is what we want to call it in our Digital Ocean Control Panel, the region is which Digital Ocean data centre to create it in, the size is the size of the Droplet, which has to be one of the standard sizes Digital Ocean provides, and backups and ipv6 are both booleans that tell Digital Ocean we want to enable backups for this Droplet, and IPv6.

The only line in there that needs more explanation is this one:

name = "hosting-${format("%02d", count.index+1)}"

Here, we have two things going on:

First, when you add a count Property to a Resource in Terraform you get a magic count variable, with an index property (count.index) which is kind-of like a loop, in that it references the current Resource Terraform is creating. count.index is zero-based, which is why I'm adding 1 to it here.

That means if you want to create 3 digital ocean Droplets, when it's creating the first Droplet, this value will be substituted with 0, when it's creating the second, 1, and the third, 3. I'm adding 1 here to make it return, 1, 2, and 3 respectively.

The second thing we're doing is calling one of the very few functions that Terraform has, format. Think of format as an sprintf type function, here, I'm saying I always want two digits, with a leading 0. This whole line will give me names like "hosting-01", "hosting-02", and "hosting-03".

Making a Plan

If all you want to do is create some Digital Ocean Droplets, then you're pretty much done, it's time to run Terraform.

When you want to Terraform some infrastructure, it should always be a two step process. First, we plan, to see what changes Terraform plans to make, and then we apply those changes. You should never just apply, because Terraform has the power to be destructive, and you could accidentally destroy half your servers if you mistakenly changed 25 to 15 instead of 21, for example.

To make a plan with our two servers here, cd to the root of our project, then assuming you installed Terraform on your $PATH, simply run terraform plan terraform/. If you're running terraform from the same directory as your *.tf files, you can omit the last terraform/, that's just the path to the directory that contains your *.tf files, and if it's the current directory then it's not necessary to specify it. As we see later, Terraform generates some config files in the directory that it's run from, so I think it's always better to store your *.tf files in a subdirectory, to keep them separate and clear. Here's what the output of that command should look like:

Terraform Variables Error

If Terraform detects an error in your configuration files, it will tell you before it tries to do anything (this actually also applies to terraform apply). Here, we can see we've forgotten to add our count variables from before, so let's fix that now.

Variables

As we said before, Terraform will run all of the *.tf files in your target directory, so it's current best practice to include all of your variables in a file called variables.tf. This gives you a single point of reference later on for everything that you can configure in your infrastructure.

Create a file called variables.tf in the terraform directory, and add the following content:

variable "servers_hosting_count" {
    default = 1
    description = "The number of hosting servers to create"
}

variable "servers_app_count" {
    default = 1
    description = "The number of app servers to create"
}

Variables are defined using HCL, just like any other resource. The main difference is that you use the variable keyword instead of resource, and that you don't need a type for the variable.

Variables have two properties, both of which are optional:

  • A default value
  • A description

The default value is used when you don't supply a value anywhere else (more important when we look at Modules later), and a description, to remind you later what this variable is for.

As variables are global in Terraform, I like to name them in a consistent format, so I know where they are primarily used: {file}_{resource name}_{property}. This doesn't scale massively well for more complex variable uses, but it's a pretty good start, and really easy to refactor later using modern IDE's if you end up moving things around.

Let's try running our plan again now that we've fixed the missing variables error:

Missing provider token

Here we've run into another common error. Terraform uses the API's of the various providers to create your resources, so it needs to know your API credentials. We'll fix that next.

Provider Credentials

There are two ways to give Terraform your provider credentials:

  • Provider configuration blocks
  • Environment variables

Provider configuration blocks look just like variables or resources, except they start with provider:

provider "digitalocean" {
    token = "${var.do_token}"
}

It's a good idea to make your provider credentials variables, so you can swap them out easily. Terraform also allows you to pass in variables as environment variables, in the form: TF_VAR_variable_name=abc123. This means that we can commit our code to git using variables, and don't have to worry about storing API keys and other sensitive information in the source repo.

This is not technically necessary though, as you can by-pass configuring the provider completely, and just store your keys in environment variables, like so:

export DIGITALOCEAN_TOKEN=abc123
terraform [command]

This is the path we're going to take here, because I don't like writing unnecessary code.

Head over to your Digital Ocean control panel and get yourself an API key, making sure that you give it both read and write permissions, then type the following in your terminal:

export DIGITALOCEAN_TOKEN=your-api-key-here

Now that you've exported that variable, it will be available for the remainder of this shell session, and once you close this shell session, or start a new one, it won't know about your DIGITALOCEAN_TOKEN variable, so it's much more secure than committing it to version control.

After exporting your API key variable, type terraform plan terraform/ from the root of your project again, and this time you should see something more like this:

Our first plan

This is the standard plan output for Terraform. You should take some time to familiarise yourself with the messages on the screen, and read everything over carefully. In this example, we can see that Terraform is going to create two new droplets for us.

Any value that says <computed> is something Terraform won't know until after the resource has been created. If you reference any of these values in another resource on subsequent runs, Terraform will know the value, as it's saved in your state file.

Terraform State

Terraform is not smart enough to inspect your account and work out what resources you already have created and start managing them. I'm sure this will come one day, but it is a long way off. Currently, this is both a curse and a blessing.

The curse is two-fold, firstly, if you want to transition your existing infrastructure to Terraform, you will have to re-create everything from scratch, then delete your old resources manually.

The second part of the curse is that Terraform has to store a record of all of the resources that it's aware of and their details. If these details change, or you do anything manually outside of Terraform, this can potentially mess up the state and things will stop working (this isn't always the case, as we will see later).

Your state file (called terraform.tfstate, and should have been created in the root of your project when you ran terraform plan) needs to be stored and looked after carefully. If you lose it there is currently no way to re-create it, Terraform will not be able to manage those resources anymore.

There are two ways to maintain your state file, both appropriate for different situations.

The first is the simplest: commit it to source control and version it just like any other file. It's a simple JSON file, so git can handle it just fine. The downside of this is if you're working in a team it will be possible for two developers to run terraform apply, change the state file, then commit their changes to master and end up with conflicts Terraform won't be able to resolve. You could get around this by having a build-server run your apply and then commit the changes back to master, and making sure your developers/ops team don't ever run Terraform themselves (only ever producing plans).

The second way to store files is the preferable way for teams, and that's with Remote State. You configure Terraform to store your state remotely, then before each plan or apply it will download the current version of the state file, so your local state is always up-to-date, and then after an apply it will update the remote state file. This means there is always one-true-master state file, and everyone always gets a copy of it before doing anything.

Currently storing your state remotely is very limited, Amazon S3, Hashicorp Atlas, Hashicorp Consul, etcd, or a custom HTTP endpoint are the only methods of storing remote state. This is fine if you use any of those tools, but if you don't then you either have to set them up just for this, or write your own HTTP endpoint. None of these are ideal. I'd like to see more remote-state back-ends added in the future (e.g. a git repo or scp). Fortunately at work we use AWS, so simply store the remote state in an S3 bucket, and it works pretty well.

In our example here, we'll be the only people running Terraform, so it's safe to simply commit it to the git repo.

The benefit (although I think it's accidental) of Terraform only being able to manage resources that it creates and knows about via the state-file is that it can't mess up your existing infrastructure, which means it won't do anything destructive and you can continue using the same cloud accounts to create Terraform resources and, and continue to create resources via any other tools (e.g. cli tools or web interfaces like Foreman).

Applying a Plan

Now that we're ready to create our resources, let's generate a plan file. Generating a plan file is separate to creating a plan, although it can be done in the same command. Generating a plan file is saying: "This is the exact thing that's going to happen when we apply". If you don't generate a plan file, you can still run apply, but there's no guarantee that changes haven't snuck in since your last plan.

Run the following from the root of your project:

terraform plan -out=terraform.tfplan terraform

You should see output similar to the following:

Saved Plan

This looks just like our plan from before, except that it tells us the plan has been saved to the file we specified. The filename itself isn't important, but to me terraform.tfplan makes sense. plan.txt would work just as well, but the file isn't human readable.

Now we can safely change our configuration files, and as long as we don't re-create our plan, we can be sure that this is all that's going to be applied.

Let's finally give that a go, run the following from the root of your project:

terraform apply terraform.tfplan

If we had specified the path to our configuration directory (like we did with terraform plan), then Terraform would have applied whatever changes are in there without a plan, but because we specified the path to our plan file instead, Terraform is going to use that to create our resources, which is much safer!

Creating a plan also means you can commit your plan file, then have a pull request/review process with a build server that runs terraform apply terraform.tfplan on merge to master, being safe in the knowledge that those are the only changes it will apply.

If everything was sucessful, you should see something like this:

Terraform Apply Success

Refresh your Digital Ocean control panel in your browser and you should see your new droplets have been created there.

Inspecting Your Infrastructure

You don't need to login to your various control panels to see the details for your infrastructure. At the moment, we're only using Digital Ocean, so that's not too bad. In a real infrastructure project, you could be using Digital Ocean Droplets, CloudFlare DNS records, S3 Buckets, and a whole combination of other services.

To make inspecting your current infrastructure simple, terraform provides a show command. Run terraform show from the root of your project:

Terraform Show

We didn't need to specify the path to our terraform/ directory, because terraform show uses the terraform.tfstate file, which should be saved in the root of our project.

Here you can see all of the vital information about your infrastructure, the most important for these droplets is probably the IP Address, but for other resources their relevant information will be shown here too.

Making Changes

Making changes to your Terraformed infrastructure is as simple as changing a value in your configuration file, then running plan and apply.

Let's rename our "app" servers to "webapps", and increase the number to 3. We can make more than one change at a time, and Terraform will handle it just fine. It is probably better practice to only make one change at a time, but for the sake of this tutorial I'll show you two at once.

In servers.tf, change the line that says name = "app-${format("%02d", count.index+1)}" to read: name = "webapps-${format("%02d", count.index+1)}", then open variables.tf, and change the value for servers_app_count from default = 1 to default = 3. Now if you save those changes, drop back to your terminal, and run terraform plan terraform/, you should see the following plan:

Terraform Changes

You can see at the top of the output that Terraform says it's refreshing state. This is a small inspection is does to see if the state of the resource has been changed manually (e.g. via a web ui), and if it has, Terraform will reset the manual changes, so that the resource matches the definition in your Terraform Configuration files. It can't do this for all changes, however, so be careful what you change manually, and always prefer to use Terraform if you can.

Notice how I didn't specify an -out parameter this time. That's because this is the first time I've run a plan for these changes, so I want to inspect them first before I generate a plan, to make sure I'm happy with them.

I'm not going to apply these changes, but if you want to, you can run terraform plan -out=terraform.tfplan terraform/, and then terraform apply terraform.tfplan and your changes should go through as before.

Destructive Changes

Sometimes, when making changes, the change you want to make Terraform can't do without destroying and re-creating a resource. Terraform will warn you in the plan when this is going to happen (with a -/+ next to the resource instead of the ~ above), so make sure you inspect your plan carefully.

In these cases it is better to create a new resource, migrate your users across, and then remove the old resource. You should only really let Terraform re-create the resource for you if you're in the development phase and the downtime won't affect anything.

Destroying Your Infrastructure

If you ever want to get rid of your entire Terraformed infrastructure, you can do that with the command terraform destroy terraform/. Once it's up and running, and you're happy, you're unlikely to ever want to do that. Whilst we're in development though, it can be useful to destroy it and start again. In production, it's better to incrementally add and remove resources to make sure you don't interrupt your service.

It could be useful to use destroy if you want to temporarily create a clone of your entire infrastructure for some sort of investigation (e.g. load testing), and then remove it later to save on the costs of the instances, just be careful which account credentials you're using and don't destroy production!

You should also try to limit how many times you create and destroy your infrastructure when you're developing with certain providers. Some of them charge you for a full hour, no matter how little of it you use. If you destroy and recreate 10 servers 10 times in an hour, you'll be charged for 100 hours of usage, even if they're only active for a few seconds.

I'm going to run a destroy now, so I can demonstrate adding Cloudflare DNS records at the same time as creating a server. We could do this incrementally (as you'll see later), but this way I get to show you what a destroy looks like too:

Terraform Destroy

Adding DNS records

One thing Forge doesn't do, but I like to do for all of my servers, is create a new DNS record at CloudFlare. Normally, I'd have to get the IP Address from Forge, then login to CloudFlare and manually add the record for the hostname. With Terraform, thanks to the CloudFlare Provider, that's not necessary anymore, and can be fully automated.

Let's add a new file in the terraform directory, called domains.tf. In there, add the following:

resource "cloudflare_record" "hosting-ipv4" {
    count = "${var.servers_hosting_count}"
    domain = "${var.domains_hosting_domain}"
    name = "${element(digitalocean_droplet.hosting.*.name, count.index)}"
    value = "${element(digitalocean_droplet.hosting.*.ipv4_address, count.index)}"
    type = "A"
    ttl = 3600
}

resource "cloudflare_record" "hosting-ipv6" {
    count = "${var.servers_hosting_count}"
    domain = "${var.domains_hosting_domain}"
    name = "${element(digitalocean_droplet.hosting.*.name, count.index)}"
    value = "${element(digitalocean_droplet.hosting.*.ipv6_address, count.index)}"
    type = "AAAA"
    ttl = 3600
}

resource "cloudflare_record" "app-ipv4" {
    count = "${var.servers_app_count}"
    domain = "${var.domains_app_domain}"
    name = "${element(digitalocean_droplet.app.*.name, count.index)}"
    value = "${element(digitalocean_droplet.app.*.ipv4_address, count.index)}"
    type = "A"
    ttl = 3600
}

resource "cloudflare_record" "app-ipv6" {
    count = "${var.servers_app_count}"
    domain = "${var.domains_app_domain}"
    name = "${element(digitalocean_droplet.app.*.name, count.index)}"
    value = "${element(digitalocean_droplet.app.*.ipv6_address, count.index)}"
    type = "AAAA"
    ttl = 3600
}

You might be able to work out what's going on here, but there are a few things that need explaining.

What we're doing is adding an IPv4, and an IPv6 DNS record for each server that we create. We're using the ${var.servers_hosting_count} and ${var.servers_app_count} variables that we created earlier, so that we create the correct number of records for each server.

With the next line:

domain = "${var.domains_hosting_domain}"

We're going to set a new variable in variables.tf to let Terraform know what domain we want to create each of these records under. We'll get on to that in a moment.

name = "${element(digitalocean_droplet.hosting.*.name, count.index)}"
value = "${element(digitalocean_droplet.hosting.*.ipv6_address, count.index)}"

These two lines are a bit more complicated. This is using another of the very few functions available in Terraform: element. This function takes two parameters, a list, and an index, and then returns the value from the list that appears at that index. We're using count.index which we've seen before, to make sure we incrementally get one element from the list for each resource we create, and then a splat, which is giving us a list of values from a particular resource.

In this example, we're saying get a list of all digitalocean_droplet.hosting.* resources, and that we only want the list to contain the name attribute. This can be tricky to get your head around at first, but it does make sense, and Terraform won't let you do it in any other way, it will just throw an error, so don't be worried about messing it up.

When getting a list from a resource using a splat, it will return the resources in the order they were created, maintaining their count.index value. This means if we use count.index here, we can be sure that the domain and droplet resources will match up.

Let's add our new variables to variables.tf:

variable "servers_hosting_count" {
    default = 1
    description = "The number of hosting servers to create"
}

variable "servers_app_count" {
    default = 1
    description = "The number of app servers to create"
}

variable "domains_hosting_domain" {
    default = "example.org"
    description = "The domain to use for our hosting servers"
}

variable "domains_app_domain" {
    default = "example.org"
    description = "The domain to use for our app servers"
}

I've used the same value twice here, for illustration purposes, but you could use two different domains, or just a single variable. Now run another terraform plan terraform/ to see what's going to happen.

Sourcing Your Environment

Hopefully, you should have another error. This is similar to the error we got the first time we ran plan. We need to provide Terraform with our credentials for CloudFlare, and to do that, we need two new environment variables, an email and a token. Login to your CloudFlare account and get your API token. The email address you login with will be the value you need for email.

export CLOUDFLARE_EMAIL=[email protected]
export CLOUDFLARE_TOKEN=abc123

We now have three variables we need to export in our shell any time we want to run Terraform, and that can be tedious to lookup and remember to do.

What I like to do is include all of my exports in a file called .env in the root of my project, which I add to my .gitignore so they don't get committed accidentally. Then whenever I want to work with Terraform, I simply have to run source .env and all of my tokens and keys are loaded for that session. This has the added benefit of removing them from your bash history, which is a bit more secure than using export directly.

A New Plan

Let's run terraform plan terraform/ again, from the root of your project. You should now see it wants to re-create the servers we destroyed earlier, and that it also wants to add two dns records to CloudFlare for every server.

As this is what we're expecting, I'm happy to apply this plan. As this is our second plan, you might think it's a good idea to keep a record of the plans we've applied, rather than overwriting the last one, kind of like database migrations.

This is a bad idea.

A plan in Terraform is created from the current state. This means if you re-apply a plan you've already applied it will assume it was in the state it was in before it was applied, and instead of re-reading your current state, then only applying the difference, it will just re-apply the whole plan, re-creating all of your existing resources, and losing the reference to the original ones in your terraform.tfstate, so you will end up with orphaned resources that you will need to delete manually. Depending on how many resources you have, this will be an absolute nightmare.

Let's create our new plan, ensuring we overwrite the old one:

rm -f terraform.tfplan
terraform plan -out terraform.tfplan terraform

You should have a new file in your plans directory, which we can now apply:

terraform apply terraform.tfplan

Some More Errors

If your domain already exists in your CloudFlare account, you should have seen everything go successfully. The following information will still be useful to you, just skip the practical steps. If it didn't work, you should see some error output similar to the following:

Terraform Zone Does Not Exist

A current limitation (that as far as I'm aware isn't being fixed any time soon), is that Terraform will only create records in a DNS Zone (domain name), it can't add a new one to your account for you.

I don't know if this is down to the CloudFlare API, or just how new Terraform is (and no-one has written a resource for it yet), but either way, we need to go in to CloudFlare and manually add the domain before we can create records on it, so head over to your control panel and do that now.

Ok, that should be the only manual interaction we need. Once your domain is all setup on CloudFlare, adding and removing records with Terraform should work just fine. Let's make a new plan and run it. First, we should run the plan without creating a plan file:

Terraform Plan Domains

That looks good, so let's generate a new plan file:

rm -f terraform.tfplan
terraform plan -out terraform.tfplan terraform

The rm isn't really necessary, Terraform should overwrite it for you, but I want to make sure I see the plan file disappear before the new one is created, just so I can be sure.

Apply it with another terraform apply terraform.tfplan, and all should go successfully. Head over to your CloudFlare account and you should see the records created there, and pointing to the servers terraform created earlier:

Terraform Created DNS Records

Refactoring

Refactoring in Terraform is currently virtually impossible. Once the name of a resource changes, Terraform will destroy it and then re-create it. They are working on ways to prevent this, and detect or trigger a rename, but currently it's not possible without understanding and hacking your terraform.tfstate.

If you want to refactor anything (such as extracting common patterns into a module), you have to create the module, re-create the resources, then migrate your users, then remove the old resources.

This is a pain, but if you're still in dev and don't care about resources being destroyed and re-created, refactor away to your heart's content.

I'm going to demonstrate the production-safe way next.

Your First Module

Terraform doesn't include *.tf files in subdirectories, it will only include the files at the root of the path you pass to plan or apply. This is a real pain, and a total oversight as far as I'm concerned, but they don't plan to change it, and there is a way around it. That's by creating a module.

In Terraform, a module is a collection of *.tf files, with specified inputs (variables), and predefined outputs.

A module is included in your project just like configuring any other resource:

module "name" {
    property = value
}

A module has only one required property: source. The source of a module can be one of:

  • A local file path (relative or absolute)
  • A git or mercurial repository
  • A subdirectory in a git or mercurial repository (this is very useful, as I'll explain later)
  • A HTTP url (I don't see how this works, it's not as simple as an archive download, so I've not used it)

We're going to start with a local path. As you write more generic modules and share them across projects, or release them as open source code, git and mercurial are the best ways to include modules.

As each provider is different, if you were to write a generic "mesos-cluster" module, for example, and then release it as open-source, you would want it to work with Digital Ocean just as easily as it does with AWS. In order to do this, it's current best practice to include subdirectories inside your module, one for each provider you support, which is each in turn a stand-alone module. You can then tell Terraform to include the module subdirectory, instead of the module itself, by adding a double slash to the end of the git/hg url, and then the path to the folder, like this:

module "example" {
    source = "git.your-company.org/example//subdir"
}

This is pretty advanced usage of Terraform, so don't worry about it for now, we're not going to use it here, it's just a cool feature and I thought I would highlight it.

Let's get started with our local module.

You might have noticed that we've repeated ourselves quite a bit so far. In servers.tf, our two declarations are almost identical, except for the names:

resource "digitalocean_droplet" "hosting" {
    count = "${var.servers_hosting_count}"
    image = "ubuntu-14-04-x64"
    name = "hosting-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "512mb"
    backups = true
    ipv6 = true
}

resource "digitalocean_droplet" "app" {
    count = "${var.servers_app_count}"
    image = "ubuntu-14-04-x64"
    name = "webapps-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "512mb"
    backups = true
    ipv6 = true
}

We have the same repetition in our domains.tf file. This is a perfect candidate for a Terraform Module. A Module in Terraform is similar but quite distinct from a Module when you're writing software.

In Terraform, think of a Module as a way to reproduce the same output over and over again, like a template for rendering HTML. The output is pretty much always the same, but you pass in different variables to substitute in certain places. There is no logic nor conditionals in a Terraform Module (besides simple count loops), just as there should be non in a HTML template.

In this instance, think of the output as: "Every time I create a new server, I also want to add a DNS record". If we wanted to do something else every time we created a new server (e.g. add a pingdom or nagios check), that should probably live in this module too.

Let's create a new directory to hold our Terraform Module, again from the root of your project, run:

mkdir -p terraform/modules/server

I've created a directory to hold our modules, and then a directory to hold this specific module, which I've called server. I've created a modules subdirectory as I'm anticipating that we'll create more at some point in the future, but I have no plan to for this tutorial.

Our new server module will now act just like our top level terraform directory does, it's just a collection of *.tf files, all of which will be run and included, but this time we'll make it more generic, and add some more variables which we can use in our main terraform directory.

Don't worry, it's about to make sense.

Create a new file in the server directory called droplet.tf, and add the following:

resource "digitalocean_droplet" "droplet" {
    count = "${var.droplet_count}"
    image = "ubuntu-14-04-x64"
    name = "${var.droplet_type}-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "${var.droplet_size}"
    backups = true
    ipv6 = true
}

This is almost identical to the definition in our original servers.tf, except we only need the one, and we've included a couple of extra variables. A side benefit of using modules is that you can now have to resources of the same type using the same name, as long as they reside in different modules.

Next, create a dns_records.tf file in the same server directory, and add this:

resource "cloudflare_record" "ipv4" {
    count = "${var.droplet_count}"
    domain = "${var.dns_record_domain}"
    name = "${element(digitalocean_droplet.droplet.*.name, count.index)}"
    value = "${element(digitalocean_droplet.droplet.*.ipv4_address, count.index)}"
    type = "A"
    ttl = 3600
}

resource "cloudflare_record" "ipv6" {
    count = "${var.droplet_count}"
    domain = "${var.dns_record_domain}"
    name = "${element(digitalocean_droplet.droplet.*.name, count.index)}"
    value = "${element(digitalocean_droplet.droplet.*.ipv6_address, count.index)}"
    type = "AAAA"
    ttl = 3600
}

Finally, we need to add a variables.tf (again, in the same server directory), and add all of the variables we've used in this module:

variable "droplet_count" {
    default = 1
    description = "The number of droplets to create"
}

variable "droplet_type" {
    description = "The droplet type to create"
}

variable "droplet_size" {
    default = "512mb"
    description = "The size of the droplet to create"
}

variable "dns_record_domain" {
    default = "example.org"
    description = "The domain to use for our servers"
}

I've added in droplet_type, and left out the default value, which makes it a required variable, and I've added in droplet_size too, so we can still create servers of different sizes if we need to scale to meet demand.

And that's it. That's all there is to creating a module in Terraform. It's pretty simple. Check that your directory structure matches the tree below, and then comes the harder part: Migrating our existing infrastructure.

.
├── ansible
├── terraform
│   ├── domains.tf
│   ├── modules
│   │   └── server
│   │       ├── dns_records.tf
│   │       ├── droplet.tf
│   │       └── variables.tf
│   ├── servers.tf
│   └── variables.tf
├── terraform.tfplan
├── terraform.tfstate
├── terraform.tfstate.backup

Migrating Resources

As I said previously, if you're just in dev, and don't care about destroying resources, you can remove them straight away from your original servers.tf, delete your domains.tf and variables.tf, then just include the new module parts below, but if you want to learn how to migrate to new infrastructure, then keep reading.

Open up servers.tf from our original terraform directory, and add the following:

module "hosting" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "hosting"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

module "app" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "webapp"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

Now, your whole file should look like this, with your original declarations at the top (the order doesn't actually matter):

resource "digitalocean_droplet" "hosting" {
    count = "${var.servers_hosting_count}"
    image = "ubuntu-14-04-x64"
    name = "hosting-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "512mb"
    backups = true
    ipv6 = true
}

resource "digitalocean_droplet" "app" {
    count = "${var.servers_app_count}"
    image = "ubuntu-14-04-x64"
    name = "webapps-${format("%02d", count.index+1)}"
    region = "lon1"
    size = "512mb"
    backups = true
    ipv6 = true
}

module "hosting" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "hosting"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

module "app" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "webapp"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

As you can see here, our variables from variables.tf become properties of our module, this is pretty cool. It doesn't just include variables from variables.tf, but variables defined anywhere in our module using variable { ... } blocks. This is another reasons that it's a good idea to include everything in variables.tf, so that you have a single point of reference for everything you can configure in your module.

That's all there is to using a module. Let's check the plan and see what happens:

Terraform Module Download

This is why Modules aren't a substitute for subdirectories. Even if you're using a local file path, you need to tell Terraform to download your Module before it will use it, which you do with terraform get [path-to-config-files]. You point get to your config files, not the module you want to download. Terraform will parse your config files and then download all of the modules you reference.

Terraform Get

Terraform will download a copy of a Module for every time you reference that Module, which is why it's downloaded it twice here. It will do that whether it's a local file, or a git repository that references the exact same commit hash. I'm sure there's a reason for it, but I think it's poor design, either way, it's what we're stuck with.

If you feel that way inclined, you can inspect what it's downloaded in a hidden directory called, .terraform in the root of your project. Whilst we're here, you should probably add that directory to your .gitignore, too.

Now if we try to run our plan again, it should work:

Terraform Get

By default, Terraform won't tell you what's going to happen inside your modules. This is annoying, but we can fix it by adding the -module-depth parameter, and setting it to 1, like this:

terraform plan -module-depth=1 terraform

If your modules include other modules (you can nest essentially infinitely), you can increase the -module-depth parameter to 2, 3, 4, or however deep you want to go to see the plan for those modules too.

You should now see the output of your plan like normal. And everything looks good! Or does it? If we look closely, Terraform wants to create some new DNS records with the same names as our existing records. Terraform isn't smart enough to detect this conflict, so it will just overwrite them, which would totally mess up our existing live infrastructure.

What we'll have to do, is temporarily create these new resources with different names, then once our users are migrated across, revert the names back to the originals.

Update your module definitions in your servers.tf file to match the following:

module "hosting" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "hosting-new"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

module "app" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "webapp-new"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

Now you should be able to re-run your plan, check that all of the values are as you expect, then create a plan file and apply it. Scroll back up if you can't remember how.

Removing Old Infrastructure

Now we've setup our new infrastructure, and I'll pretend we've migrated our users across to it, it's time to remove the old infrastructure.

Doing that is pretty simple. Remove the resource declarations from servers.tf, leaving only your two modules, delete domains.tf, and variables.tf, and finally, update the droplet-type variables for your modules to remove the -new, once it looks like the below, then we're good to go:

module "hosting" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "hosting"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

module "app" {
    source = "./modules/server"
    droplet_count = 1
    droplet_type = "webapp"
    droplet_size = "512mb"
    dns_record_domain = "example.org"
}

And the directory structure:

.
├── ansible
├── terraform
│   ├── modules
│   │   └── server
│   │       ├── dns_records.tf
│   │       ├── droplet.tf
│   │       └── variables.tf
│   └── servers.tf
├── terraform.tfplan
├── terraform.tfstate
├── terraform.tfstate.backup

Let's check the plan, and see what's going on:

Terraform Plan Remove

Terraform seems to have a bug at the moment with the case-sensitivity of IPv6 addresses, which is why it's saying that it will change them. Don't worry about that, look closely and you will see the two values are exactly the same.

Create one final plan file with terraform plan -module-depth=1 -out=terraform.tfplan terraform/, and then apply it using terraform apply terraform.tfplan, and we're all done with the Terraform portion of the tutorial. Your infrastructure should be created, and if you want to add any new server types, simply replicate your module blocks, updating the properties as necessary, or to add new servers of a specific type, simply increase the droplet_count property.

Hopefully this post has given you a good grounding in Terraform, it's usage, short comings, and pitfalls to avoid so you can start replicating your own infrastructure using it.

In the next part, we're going to look at how to automatically provision our servers with a modern PHP stack as they're created (just like Forge does), which we will do using ansible.

I'll then go on to show you how to setup multiple projects on those servers, including cloning from git, adding databases, isolated user accounts, cron jobs, and so on, and finally, how to make them automatically deploy on push, with blue-green deployments for zero downtime, just like Laravel's Envoyer.

For exclusive content, including screen-casts, videos, and early beta access to my projects, subscribe to my email list below.


I love discussion, but not blog comments. If you want to comment on what's written above, head over to twitter.