Infrastructure as Code (IaC) with Terraform — MyTake
The first time I heard about Infrastructure as code my eyes lit up, I was like, really? I wondered if Infrastructure guys have an option of coding their work and put it under version control and not having to do manual steps or write complicated scripts, things haven’t been easier even if we want to make things automated for setting up systems and make life better in maintaining one of the most difficult and underrated parts of IT i.e infrastructure. It is not like there is no automation on the infrastructure side, we have always written code to make parts of environments better, it is just that it was never fully automated in a way such that you can literally build a whole environment with the execution of code, and infrastructure as code paves the way to achieve that goal.
Imp Note:
- Before we start our small overview, I would like to update that there are some assumptions made here in terms of you knowing certain things like what is AWS or Azure cloud or at least having a basic idea of what it is, as I will not delve deep into those topics.
2. I by no means say, everything mentioned here is a line curved on the stone on this topic, however, I do have a good amount of experience of using terraform with two different clouds. I can try my best to answer any question if I was not clear in my writing, please reach out. This is all part of sharing my experience and I am happy to get educated if someone knows it better.
Index
- What is Infrastructure As Code (IAC)
- What is Terraform
- How it Works
- Providers
- Setting Up Environment
- Terraform Stages of Execution
- Terraform State
- Terraform Modules
- Terraform Implementation
- Terraform with Python
- Reference Links
Infrastructure as Code (Iac)
- IaC is the managing and provisioning of infrastructure through code instead of using a manual process to configure devices or systems.
- The objective of IaC is to ensure that we write code once use it multiple times.
- Architecture and Engineering teams aim is to create an API style of code model/modules so that it makes it easier for everyone to participates in building and maintaining the infrastructure for the application. Also, allow some non-infra teams to build some basic things like a VM without knowing the complete underlying details and just supplying parameters needed to create their resource.
- Keeping in mind the hybrid cloud architecture that organizations are adopting in recent times, terraform in my opinion is one of the best choices as a cloud-agnostic tool.
What is Terraform
- Terraform users define infrastructure in a simple, human-readable configuration language called HCL (HashiCorp Configuration Language).
- Terraform is an IaC tool made by HashiCorp, it lets you provision, manage, and maintain cloud resources like Server, Networking and Storage, etc.
- Terraform can act like a tool that we can run and execute from CLI to build infrastructure, however, it is also a language that defines those changes.
- Terraform is something that helps you build resources that you want on the cloud, however, it does not directly define what’s on it.
- It allows you to build an Immutable infrastructure.
- It also allows us to have a disposable infrastructure.
- We can even build a CI/CD using terraform.
- The latest version of terraform is 0.13. The industry is still adapting to 0.12 and maybe (this is just a guess) a lot of folks might be still using 0.11. I do see the benefit of moving to a higher version though, it might be a rough path if the code is not standardized, the advantages of using the latest constructs of an evolving language make your job easier.
- The newer version provides good programming constructs like conditional operators, Looping options, and the creation of objects which are more powerful and allow us to code in a more efficient way.
- Terraform is not a tool for config management. Tools like Ansible when used independently or along with terraform can help you achieve better management of servers, updates, etc.
How it Works
- To set it up, you need to install and setup terraform software on your local to get started. Hashicorp distributes terraform as a binary package which makes it easier to set up using package managers.
- Hashicorp provides packages for various OS
https://www.terraform.io/downloads.html - You need an editor to write code in editors like eg, ‘vi’, ‘Sublime’, ‘VisualStudioCode’ or IntelliJ, etc.
- All terraform code files are have a ‘.tf’ extension and for environment variables definition, ‘.tfvars’ should be specified.
- Terraform maintains a state(memory) of your infrastructure in the cloud or on your local system.
Providers
Providers are the ones that help terraform deliver a single unified workflow of what modern infrastructure needs.
IaaS
- It provides information about how it connects to the various clouds like
Amazon, Azure, Google, Oracle, Alibaba, etc - Can be on-prem infrastructure OpenStack or VMWare
PaaS
- Terraform can be leveraged as Paas for Kubernetes, Heroku, or Lambdas
SaaS
- We can use it at SaaS with monitoring solutions like DataDog, Fastly for CDN, or managing Github Team.
- More about different providers
https://registry.terraform.io/browse/providers
Setting Up Your Environment:
- To Run Terraform with Azure.
You would need to provide the below-mentioned information to run terraform from your local or in the secrets of your CI/CD setup.
Setup your bash_profile under the user’s home profile with the below values for your local setup.
#Azure Terraform Secrets in local ~/.bash_profile
export TF_VAR_client_id=”8**************************c”
export TF_VAR_client_secret= ”n***********************I”
export TF_VAR_tenant_id=”6***************************7”
export TF_VAR_subscription_id=”6**********************4"
Note: to get client_id, client_secret, and tenant_id you should be creating Service Principle in Azure portal, which can be done via portal /cli/powershell. Subcription_ID can be pulled from the azure subscription where you want your resources to live.
Use the below link(Azure Documentation) for a reference as to how to create the Service principle.
NOT RECOMMENDED option but you can pass above parameter in code as well.
provider “azurerm” {
version = “1.35”
client_id = 8d********************************c
Client_secret = nZ*********************************I
tenant _id = 6********************************c
subscription_id = 6********************************4
}
Again would like to re-iterate, it is not recommended to pass these values in the code, we can set these as environment variables as Terraform Environment variables locally. An even better way would be setting it up in Azure Vault or ideally set it up in the secrets of your CI/CD environment variables.
- To Run Terraform with AWS.
You would need to create an IAM user that can provide a programmatic access for AWS API’s for Terraform. Security credentials for IAM user AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will be used to authenticate with AWS by Terraform.
The below-mentioned link(from amazon docs) will guide you through the process of doing the same.
Once you have the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY you will have to configure these secrets with your CI/CD for terraform build and deploy. Or if you want to set it up locally you can put it under the bash profile.
Setup your bash_profile under the user's home profile with the below values.
AWS_ACCESS_KEY_ID=A*************************E
AWS_SECRET_ACCESS_KEY=j*********************************b
Or you can pass it in terraform code, NOT RECOMMENDED.
variable “AWS_ACCESS_KEY” {
default = “A**********************E”
}
variable “AWS_SECRET_ACCESS_KEY” {
default = “j*******************************************b”
}
Terraform stages of execution
- terraform refresh ⇒ It reconciles the terraform view against the real-world infrastructure. This can be used to detect any drift from the last-known state, and to update the state file.
- terraform init ⇒ Initializes provider plugins/modules and downloads it(required to execute if the terraform state of infrastructure is stored in s3 or azure storage as you need to locally initialize the same state)
- terraform plan ⇒ It is the desired state of the infrastructure that will be if applied.
- terraform apply ⇒ Provisioning changes to your infrastructure.
- terraform destroy ⇒ Destroy the resources based on the information in the state file.
The most used commands in your play with terraform are going to be “terraform plan” and “terraform apply”.
terraform plan
Terraform Plan is like a preview of what will be created when the changes are applied to the infrastructure. It is like an alien mystery that we are about to uncover.
Once you have seen through the plan and understand what will be created, destroyed, or changed, you have a type of confirmation as to what to expect from the next step. It is similar to a compile of code, you can explicitly output the plan and use it while applying the changes.
terraform apply
This is when we put the plan to work and start seeing a change in our infrastructure
Terraform State (Azure and AWS)
- Default terraform state location
By Default terraform state is located on your computer where your terraform .tf files are placed. However, it is not the best place to keep it as we work in multiuser environments where multiple contributors will make changes to the code and try to make changes in the infrastructure. There is a better way to do it, using a remote state where we can track the state of the project, and not one person controls it. - Remote State
1. Azure Remote State
Terraform state file can be stored in azure, however, we would need to ensure the following components are in place to store the remote state in Azure.
terraform {
backend “azurerm” {
resource_group_name = “rg-myproject-cus-dev”
storage_account_name = “storage0myproject0dev”
container_name = “storagecontainer0myproject0dev”
key = “infra-myproject-cus-dev.tfstate”
access_key = “AjBc********************************==”
}
}
Note: Resource Group b. Storage Account c. Storage_Container should be created manually by using Console/CLI/Powershell beforehand as it is like the “chicken and egg problem” as we call it. Naming Storage account and container is tricky, so please follow guidelines.
2. Terraform Remote State using S3.
In AWS the standard way of storing a state file is existing s3 buckets and state locking using the existing dynamodb table. It is also recommended to have bucket versioning to allow for state recovery in the case of accidental deletions and human error.
Note: The S3 Bucket and DynamoDB table will have to be created beforehand.
Code snippet for creating s3 backend
terraform {
backend “s3” {
bucket = “terraformstate-myproject”
key = “terraform/mykey”
}
}
S3 Bucket Permissions
Terraform will need the following AWS IAM permissions on the target backend bucket:
s3:ListBucketonarn:aws:s3:::mybuckets3:GetObjectonarn:aws:s3:::mybucket/path/to/my/keys3:PutObjectonarn:aws:s3:::mybucket/path/to/my/key
This is seen in the following AWS IAM Statement:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::mybucket"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::mybucket/path/to/my/key"
}
]
}DynamoDB Table Permissions
If you are using state locking, Terraform will need the following AWS IAM permissions on the DynamoDB table (arn:aws:dynamodb:::table/mytable):
This is seen in the following AWS IAM Statement:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem"
],
"Resource": "arn:aws:dynamodb:*:*:table/mytable"
}
]
}3. Remote State using Terraform Enterprise or Terraform Cloud
This is feature is provided by Hashicorp in their Enterprise and Cloud version It takes care of the remote state and management for you.
Terraform Modules:
One of the great structures of programming languages I love is being able to create custom API’s that allows you to have the freedom of “writing once and use it everywhere applicable” i.e having a repeatable code used wherever and whenever you want is the whole idea behind terraform providing modules to make life easier for coders and add some standardization to it. Terraform modules is nothing but a directory/folder that will have your generic terraform code for repeatable usage
For. eg: We can create a module for Cloudwatch Alerts in AWS or one for windows_virtual machine in Azure. We can reuse this code as a reference API for all different projects and avoid rewriting the same code again.
Terraform Implementation:
We will see a simple example of creating a virtual machine using terraform implementation in AWS.
main.tf is the file where most of your code resides, you can name files anything but need to ensure they either have “.tf” extension for code and “.tfvars” for env, we will go by some generic naming here
main.tf
resource “aws_instance” “myinstance” {
ami = var.AMIS[var.AWS_REGION]
instance_type = “t2.micro”
}
vars.tf(again just use this option of supplying IAM crendentials to get your first code running then move this to bash(as mentioned above) or CI/CD secrets)
variable “AWS_ACCESS_KEY” {
default = “A*****************E”
}
variable “AWS_SECRET_ACCESS_KEY” {
default = “j******************************b”
}
variable “AWS_REGION” {
default = “us-east-1”
}
variable “AMIS” {
type = map(string)
default = {
us-east-1 = “ami-13be557e”
us-west-2 = “ami-06b94666”
eu-west-1 = “ami-0d729a60”
}
}
cloud_provider.tf
provider “aws” {
access_key = var.AWS_ACCESS_KEY
secret_key = var.AWS_SECRET_KEY
region = var.AWS_REGION
}
terraform_versions.tf
terraform {
required_version = “>= 0.12”
}
If you want to implement azure with terraform then follow belolink.
Terraform with Python
Terraform with Python(New feature or way — not matured enough to be adopted, but great to see movement in this area)
Hashicorp recently announced the usage of programming languages like Python or using TypeScript to build infrastructure. This will make the infrastructure build look more programmatic by having constructs like classes.
I tried my hand at it, here is the GitHub link for my public repo
Reference Links
- How to Download Terraform
https://www.terraform.io/downloads.html - How to get started with IAC from Hashicorp
https://learn.hashicorp.com/terraform/getting-started/install - Terraform Registry and discover all the providers and related information
https://registry.terraform.io/ - Complete Terraform Documentation https://www.terraform.io/docs/index.html
- Security Bulletin checks
https://checkpoint.hashicorp.com/ - Common terraform commands can be referenced from the below link
https://www.terraform.io/docs/commands/index.html
