Create a multi-stage pipeline for Terraform in Azure Devops - Deploying to multiple environments

Similar to my previous post How to Create Multiple Environments Using Terraform Workspaces - Virtual Network Creation Part 1 (phiptech.com). Instead of using Terraform workspaces for multiple environments, I have created a multi-stage pipeline using Azure DevOps with YAML.

Creating a multi-state pipeline for Terraform is really a must-need in any DevOps project. It can help automate the deployment process and ensure consistency and reliability in infrastructure deployment. It also provides a repeatable process which can save time and reduce errors.

Consider this scenario. You want to deploy infrastructure which includes virtual networks, subnets, and load balancers to multiple environments such as dev, test and prod. To achieve this approach, you can create a multi-stage pipeline.

This can look like the following:

• Build Stage - This stage can validate Terraform code and generate the Terraform plan which is then packaged into a deployment artifact.
• Dev Stage - This stage can deploy infrastructure to the Dev environment. It can be triggered manually or automatically after the build stage
• Test Stage - Same as the dev stage but is triggered after the dev stage. You can use the same terraform commands to apply terraform plan
• Prod Stage - This stage is triggered after the Test Stage automatically or manually.

The triggers mentioned above is referring to approval gates. You generally want another pair of eyes to oversee the pipeline you are running. This also enforces good deployment practices.

In this post, I have attempted to implement the approach above in Azure DevOps.

Prerequisites/Knowledge:

• Azure DevOps
• Yaml
• Terraform
• Azure CLI

Below is the workflow of what we'll be building:

This guide assumes you have a service connection connected to your Azure DevOps Project. You will need to grant this service principal access to your subscription with contributor access.

Let's get started.

Create Environment

We first need to create environments in Azure DevOps. This will be used for pre-deployment approvals when the pipeline is run.

1. Navigate to Azure DevOps-->Pipelines-->Environments

2. Once you've created the environment, click on dev and then the 3 dots on the top right corner and click on Approvals and Checks.

3. Add the approvals in the next step

4. Now add the user who is going to approve it. I will add myself in this example.

5. Repeat steps 1-4 to do the same for test and prod

Create Repo & Folder structure

6. Next, we want to create the repo and folder structure for our environments. I've created a pipeline in a folder called terraform_test and put all the files in there. This is what it'll contain. Go ahead and create all these files

TERRAFORM_TEST
│ azure-pipelines.yml
│ readme.md
│ variables.tf
│ vm.tf
│
├───azure-resource-group
│ main.tf
│ variables.tf
│
├───azure-windows-vm
│ main.tf
│ variables.tf
│
└───environments
│ dev.tfvars
│ prod.tfvars

Folders explained:

azure-resource-group - Module to create a resource group

azure-windows-vm - Module to create azure windows vm

environments folder - contains the variables for each environment

Create Variable Groups

Before digging into the Azure pipeline, we also need to set up preset variables which will be used within the yaml file. These variables will be used to authenticate to Azure and connect the backend storage account for Terraform to store the state files.

ARM_TENANT_ID

ARM_CLIENT_ID

ARM_CLIENT_SECRET

TF_STORAGE_ACCOUNT

TF_STORAGE_ACCOUNT_KEY

TF_STORAGE_BLOB_CONTAINER

TF_STORAGE_BLOB_NAME

TF_STORAGE_RG

Pipeline

parameters:
  - name: env
    displayName: Environment
    type: string
    default: dev
    values:
      - dev
      - prod

trigger:
  - feature*

variables:
  - group: Terraform-Pipeline-Test

In the code snippet above, an environment parameter is defined to establish various environments. This is significant because it allows us to select the desired environment to run when it is triggered. I am creating a trigger based on a feature branch. This will automatically trigger a pipeline run each time the code has been committed to the branch.

Build stage - Authentication

The first stage of the pipeline is the authentication stage although not necessary. This stage checks whether the permissions are working

stages:
  - stage: Authenticate
    jobs:
      - job: Authenticate
        pool:
          vmImage: ubuntu-latest
        steps:
          - task: AzureCLI@2
            displayName: 'Authenticate with Azure'
            inputs:
              azureSubscription: 'terraform-sp' # Service Connection
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |
                subscriptionId=$(az account show --query 'id' -o tsv)
                echo "Selected subscription: $subscriptionId"
                az account set --subscription $subscriptionId
            env:
              ARM_TENANT_ID: $(tenantId)
              ARM_CLIENT_ID: $(servicePrincipalId)
              ARM_CLIENT_SECRET: $(servicePrincipalKey)
              subscriptionId: $(subscriptionId)
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: 'Install Terraform'
            inputs:
              terraformVersion: '1.4.4'

Build stage - Terraform Init & Plan

The second stage is the terraform init & plan stage. This is known as the build stage. This stage will produce the plan and then publish the artifacts for the next stage.

To use different backends for multiple environments, we use the parameter "env" which I've defined above. This will create separate state files based on the environment.

This is the line we're talking about:

backend-config=key=${{ parameters.env }}.tfstate

  - stage: TerraformPlan
    dependsOn: Authenticate
    displayName: 'Terraform Plan'
    jobs:
      - job: Plan
        displayName: 'Plan'
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: AzureCLI@2
            displayName: Terraform Init
            inputs:
              workingDirectory: $(System.DefaultWorkingDirectory)/terraform_test/
              azureSubscription:  'terraform-sp' # Service Connection
              scriptType: bash
              scriptLocation: inlineScript
              addSpnToEnvironment: true
              inlineScript: |
                export AAD_USE_MICROSOFT_GRAPH=1
                export ARM_CLIENT_ID=$servicePrincipalId
                export ARM_CLIENT_SECRET=$servicePrincipalKey
                export ARM_TENANT_ID=$tenantId
                export TF_IN_AUTOMATION=true

                terraform init \
                -input=false \
                -backend-config="resource_group_name=${TF_STORAGE_RG}" \
                -backend-config="storage_account_name=${TF_STORAGE_ACCOUNT}" \
                -backend-config="container_name=${TF_STORAGE_BLOB_CONTAINER}" \
                -backend-config=key=${{ parameters.env }}.tfstate 

          - task: AzureCLI@2
            displayName: Terraform Plan
            inputs:
              workingDirectory: $(System.DefaultWorkingDirectory)/terraform_test/
              azureSubscription:  'terraform-sp' # Service Connection
              scriptType: bash
              scriptLocation: inlineScript
              addSpnToEnvironment: true
              displayName: Check artifact staging directory
              inlineScript: |
                export AAD_USE_MICROSOFT_GRAPH=1
                export ARM_CLIENT_ID=$servicePrincipalId
                export ARM_CLIENT_SECRET=$servicePrincipalKey
                export ARM_TENANT_ID=$tenantId
                export TF_IN_AUTOMATION=true

                chmod -R +x .terraform
                terraform plan \
                -input=false \
                -var-file=environments/${{ parameters.env }}.tfvars \
                -out=${{ parameters.TfPlanPath }}

          - task: PublishPipelineArtifact@1
            displayName: Publish Plan artifact
            inputs:
              targetPath: $(System.DefaultWorkingDirectory)/terraform_test/
              artifactName: 'terraform_plan'
              publishLocation: 'pipeline'

Dev & Prod Deployment Stage - Terraform Apply

And lastly, we have the deployment stage. This is where it applies the plan and deploys to Azure. Before it deploys to the environments, I've added a deployment job that sets an approval gate before this stage is run. The condition means that it only runs when the previous stage is successful and is only triggered if a build is committed to the repo.

jobs:
  - deployment: terraform
    condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
    displayName: 'Terraform Apply'
    pool:
      vmImage: ubuntu-latest
    environment: ${{ parameters.env }}
    strategy:
      runOnce:
        deploy:

  - stage: Terraform_Apply
    jobs:
      - deployment: terraform
        condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
        displayName: 'Terraform Apply'
        pool:
          vmImage: ubuntu-latest
        environment: ${{ parameters.env }}
        strategy:
          runOnce:
            deploy:
              steps:
              - task: DownloadPipelineArtifact@2
                inputs:
                  artifactName: 'terraform_plan'
                  path: $(System.DefaultWorkingDirectory)/terraform_test/
         
              - task: AzureCLI@2
                displayName: Terraform Apply
                inputs:
                  workingDirectory: $(System.DefaultWorkingDirectory)/terraform_test/
                  azureSubscription: 'terraform-sp' # Service Connection
                  scriptType: bash
                  scriptLocation: inlineScript
                  addSpnToEnvironment: true
                  inlineScript: |
                    ls -la
                    export AAD_USE_MICROSOFT_GRAPH=1
                    export ARM_CLIENT_ID=$servicePrincipalId
                    export ARM_CLIENT_SECRET=$servicePrincipalKey
                    export ARM_TENANT_ID=$tenantId
                    export TF_IN_AUTOMATION=true

                    chmod -R +x .terraform
                    terraform version
                    terraform apply \
                    -input=false \
                    -var-file=environments/${{ parameters.env }}.tfvars \
                    -auto-approve
                  terraformVersion: '1.4.4'

Modules - Terraform

I've provided modules for building a Windows machine and resource group. Note, I didn't create the first resource group "my-resource-group" in this originally as I had an existing resource group already. You can easily add this in vm.tf

TERRAFORM_TEST
│ vm.tf

provider "azurerm" {
  features {}
}

module "rg" {
  source              = "./azure-resource-group"
  resource_group_name = "my-resource-group2"
  location            = "australiaeast"
}
module "windows_vm" {
  source              = "./azure-windows-vm/"
  vm_name             = var.vm_name
  vm_size             = var.vm_size
  admin_username      = var.admin_username
  admin_password      = var.admin_password
  # rdp_port          = var.rdp_port
  resource_group_name = var.resource_group_name
  location            = var.location
  subnet_id           = var.subnet_id
  vnet_name           = var.vnet_name
}

│ variables.tf

# This is an example variables file for creating a virtual machine

variable "resource_group_name" {
  type        = string
  description = "The name of the resource group to create the virtual machine in"
}

variable "location" {
  type        = string
  description = "The location to create the virtual machine in"
}

variable "vm_name" {
  type        = string
  description = "The name of the virtual machine"
  default     = "my-windows-vm"
}

variable "vm_size" {
  type        = string
  description = "The size of the virtual machine"
  default     = "Standard_DS1_v2"
}

variable "admin_username" {
  type        = string
  description = "The username for the virtual machine's administrator account"
  default     = "myadminuser"
}

variable "admin_password" {
  type        = string
  description = "The password for the virtual machine's administrator account"
  default     = "myadminpassword"
}

variable "rdp_port" {
  type        = number
  description = "The port used for RDP traffic"
  default     = 3389
}

variable "subnet_id" {
  type        = string
  description = "The ID of the subnet to create the virtual machine in"
}

variable "vnet_name" {
  type        = string
  description = "The name of the VNet to create the virtual machine in"
}

# New environment variable for specifying the environment type
variable "environment" {
  type        = string
  description = "The type of environment to create the virtual machine in"
  default     = "dev"
}

├───azure-resource-group
│ main.tf

resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}

│ variables.tf

variable "resource_group_name" {
  description = "The name of the resource group to create"
}

variable "location" {
  description = "The location to create the resource group in"
}

│
├───azure-windows-vm
│ main.tf

resource "azurerm_windows_virtual_machine" "this" {
  name                = var.vm_name
  resource_group_name = var.resource_group_name
  location            = var.location
  size                = var.vm_size
  admin_username      = var.admin_username
  admin_password      = var.admin_password
  network_interface_ids = [
    azurerm_network_interface.nic.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2019-Datacenter"
    version   = "latest"
  }
}

#Build Network Layer
resource "azurerm_virtual_network" "main" {
  name                = var.vnet_name
  address_space       = ["10.0.0.0/16"]
  location            = var.location
  resource_group_name = var.resource_group_name
}

resource "azurerm_subnet" "internal" {
  name                 = "internal"
  resource_group_name  = var.resource_group_name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.1.0/24"]
}


resource "azurerm_public_ip" "main" {
  name                = "${var.vm_name}-pip"
  location            = var.location
  resource_group_name = var.resource_group_name
  allocation_method   = "Dynamic"

}

resource "azurerm_network_interface" "nic" {
  name                = "${var.vm_name}-nic"
  location            = var.location
  resource_group_name = var.resource_group_name

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = azurerm_subnet.internal.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.main.id
  }
}

│ variables.tf

variable "vm_name" {
  description = "The name of the virtual machine"
}

variable "vm_size" {
  description = "The size of the virtual machine"
  default     = "Standard_B2s"
}

variable "admin_username" {
  description = "The username for the virtual machine's administrator account"
}

variable "admin_password" {
  description = "The password for the virtual machine's administrator account"
}

variable "winrm_http_port" {
  description = "The port used for WinRM HTTP traffic"
  default     = 5985
}

variable "winrm_https_port" {
  description = "The port used for WinRM HTTPS traffic"
  default     = 5986
}

variable "resource_group_name" {
  description = "The name of the resource group to create the virtual machine in"
}

variable "location" {
  description = "The location to create the virtual machine in"
}

variable "subnet_id" {
  description = "The ID of the subnet to create the virtual machine in"
}

variable "vnet_name" {
  description = "The name of the VNet to create the virtual machine in"
}

└───environments
│ dev.tfvars

resource_group_name = "my-resource-group"
location            = "australiaeast"
vm_name             = "my-windows-vm"
vm_size             = "Standard_B2s"
admin_username      = "admin_user"
admin_password      = "P@$$w0rd1SC**%xa"
rdp_port            = 3389
subnet_id           = "/subscriptions//resourceGroups/my-resource-group/providers/Microsoft.Network/virtualNetworks/my-vnet/subnets/internal"
vnet_name           = "my-vnet"

│ prod.tfvars

resource_group_name = "my-resource-group2"
location            = "australiaeast"
vm_name             = "prod-my-windows-vm"
vm_size             = "Standard_B2s"
admin_username      = "admin_user"
admin_password      = "P@$$w0rd1234!HH$@@(**XX"
rdp_port            = 3389
subnet_id           = "/subscriptions//resourceGroups/my-resource-group2/providers/Microsoft.Network/virtualNetworks/prod-vnet/subnets/internal"
vnet_name           = "prod-vnet"

The Result

Pipeline:

State Files:

If you have any suggestions or improvements, please do let me know. There is so much more we can add to this but for this purpose, we'll keep it simple. Hopefully, it's not too hard to comprehend.