If you were hired as an Operations Engineer — your job was to procure hardware, install it at the data-center, configure the networking and firewalls, and then keep things running — "up", and that meant keeping them stable and minimizing change. What the Ops team was running was typically a custom-made web application, which Ops folks did not understand and could not fix when they broke. And engineers often times weren’t even on call. These practices created a lot of distrust between ops and engineering.

At the same time, lean startup methodology was telling founders to iterate and fail quickly, as to learn and pivot, rinse repeat until your business model works. All of that meant that until we were successful, the risk of crashing the site was acceptable to business if it meant continuous and rapid delivery of new features.

This tension was well described in the most excellent book on Site Reliability Engineering: How Google Runs Production Systems, which contains much additional wisdom from Google’s point of view.

In the years 2012-2015 when I was a CTO at Wanelo we went all the way Chef.

We created three non-overlapping environments: dev, staging and production.

Staging and Production were identical, except staging was way smaller. We could try the new Chef cookbook on Staging, and only after confirming it works would we apply it to production.

This often required cloning the repo three times (one per environment) and configuring Chef’s environment authentication per folder. That was a bit clunky, if I am being honest.

As it turned out, having a central database of all of your servers, their roles, and cookbooks that installed various components, made completely automatic configuration of the newly booted servers possible.

But, and this is an important but — we had to first provision the EC2 server using aws CLI or web UI, and only then could Chef connect and install itself, and then run pulling all the tasks from the cental server, and then running them until the machine came up with the application (if that was its role) or some other software dictated by the role assigned to the machine.

This ability to search for a particular role, automate routing of all requests in the system using Chef was a pinnacle of my personal DevOps experience.

Since then, I re-created similar Chef-based deployments of Rails applications at least three more times, at three different companies. There was not a single thing in the cloud that was manually configured. Everything was done via Chef, and the codebase was git-committed, peer-reviewed and even had automated tests.

I felt that 100% automation that Chef gave us was incredibly awesome, especially considering manual ways before it. But Chef was not without its own problems.

One big complaint about Chef is that it mutates infrastructure. Chef runs once every 30 minutes, and in theory is able to modify the servers it was running on.

On top of that, Chef had no knowledge of any manual modifications performed to any server. This sometimes created drift between servers, and we’d have to replace some of the instances.

The second thing that Chef really sucked at was provisioning new resources, even as simple as EC2 instances. We had to use aws CLI to provision our servers, and Chef’s bootstrap mechanism to get Chef installed there, and subsequently whatever configuration is appropriate for that role(s).

This gap between configuration management and infrastructure provisioning is exactly what Terraform was designed to solve.

We’ll build out a diagram that, hopefully, will assist in understanding how Terraform fits into the picture.

This first diagram shows how Terraform binary, using a "plugin architecture" design pattern, is able to load a specific adaptor for any specific cloud. Terraform allows us to define desired resources in a consistent declarative language, HCL. That achieves some uniformity among infrastructure as code efforts.

In reality, however, each cloud is done so differently that there is no universal Terraform language that is Cloud-agnostic. Terraform can speak to every cloud for which a provider has been written. But Terraform can only deal with resources and actions that are available in that particular cloud, and supported by the given provider.

It’s worthwhile to say what Terraform isn’t:

One of the main distinguishing aspects of Terraform compared to pretty much any other "previous-generation" DevOps automation is that terraform is a compiled binary executable written in Go. It’s main purpose is to "leverage the rich Cloud Native APIs that are provided by the cloud providers".

For instance, for AWS Terraform will download an "aws" module which contains AWS-specific resources, together with the code that knows how to use AWS API to create, modify, and delete those resources.

Terraform never goes out to SSH into the host, like Chef did. It sits outside of your Cloud environment, and controls your infrastructure by making API calls to the cloud provider, and comparing it to the state file.

Whenever you modify Terraform resource descriptions in your *.tf files and run terraform plan, it will compare three things:

Terraform will then compare 1 & 3 to show you the difference (and it will warn you about any differences between 1 & 2).

Any changes you made locally should show up as difference. Running terraform apply will use AWS API to modify appropriate resources, this syncing the state file with the actual state of the infrastructure and the HCL template.

Terraform keeps track of the state of the infrastructure it manages. This state is stored in a file named terraform.tfstate. This file is created after the first terraform apply command is run, and is updated every time Terraform is run.

For teams larger than one, it is recommended to store the state file on S3. For more information on the best practices related to state management please read about How to Manage Terraform S3 Backend on the Spacelift blog.

I’ve skipped the very basics on how to do plan and apply, as well as init because there are a ton of literature on how to get that setup.

Below is a very simple example of Terraform configuration file that creates an AWS EC2 instance and a security group:

What’s cool about the above declaration is that can be used as a module to create instances of any type that belong the hardcoded VPC.

We can then use this module in another project (directory) where we would invoke it while setting its variables to the desired values:

When you look at the above example it seems pretty clear. What doesn’t seem clear is how to grow this systems architecture. What if we want to add a load balancer, a database, a cache, and a few more instances?

If the module is, say, AWS-specific, then you may also find files such as:

Each directory containing *.tf files maintains its own separate state file. This is a crucial concept that isn’t immediately obvious:

This means when you run terraform apply in the networking/ directory, it only knows about and manages the resources defined in that directory’s *.tf files. It has no knowledge of resources defined in database/ or compute/ directories.

This isolation creates a fundamental challenge: How do you handle dependencies between directories?

Consider a typical infrastructure setup: 1. VPC and Subnets (networking layer) 2. RDS Database (needs VPC subnets) 3. EC2 Instances (needs VPC subnets and database connection info)

Since each directory has its own state, how does the database/ directory know which VPC subnets were created by the networking/ directory?

Terraform provides terraform_remote_state data sources to reference outputs from other state files:

networking/outputs.tf:

database/main.tf:

With separate directories and state files, you now face the dependency ordering problem:

Manual Dependency Management:

This manual process is error-prone and doesn’t scale well for complex infrastructure with many interdependencies.

You could put everything in a single directory with one state file, but this creates other problems:

This is where the limitations of vanilla Terraform become apparent. Managing dependencies manually across multiple directories is:

This dependency challenge is exactly what tools like Terragrunt were created to solve.

Terragrunt introduces a terragrunt.hcl configuration file that can specify dependencies between modules:

File Structure with Terragrunt:

networking/terragrunt.hcl:

database/terragrunt.hcl:

With Terragrunt, you can now run:

Terragrunt automatically:

DRY Configuration: Terragrunt can generate provider configuration, avoiding repetition:

Mock Outputs for Development:

The mock_outputs feature allows you to work on dependent modules even when dependencies aren’t deployed yet.

Built-in Remote State Management:

Terragrunt automatically configures S3 backends with DynamoDB locking, eliminating boilerplate.

Use Terragrunt when:

Stick with vanilla Terraform when: