How do you do blue/green deployments, rolling deployments, and canary deployments with Terraform?

[brikis98]

This is a question we got from a customer. I'll share our recommendation below.

How can I use Terraform to do blue/green deployments, rolling deployments, and canary deployments? How would I integrate this sort of thing into my CI / CD pipelines?

---

Tracked in ticket #110721

[brikis98]

As discussed in How to use Terraform as a Team, Terraform does not have support built into the language or its standard workflows for deployment strategies such as blue/green deployment, rolling deployment, or canary deployments. However, depending on what you're deploying, there are several ways to accomplish these deployment strategies:

Option 1: deployment strategies built into specific resources/services

Some of the services that you manage with Terraform may natively support various deployment strategies if you configure the corresponding Terraform resources correctly. Here are a few examples:

• EKS/K8S: Kubernetes has built-in support for rolling deployments using the Deployment object. Whenever you update a Deployment, Kubernetes automatically rolls out the changes by deploying new Pods, waiting for them to come up, and then undeploying the old Pods. So if you're using a Terraform module to manage your Kubernetes Deployment, such as the kubernetes-service module, you get zero-downtime rolling deployment automatically any time you make a change and run apply. The kubernetes-service module also supports canary deployments by using the canary_image and desired_number_of_canary_pods parameters.
• ECS: ECS also has built-in support for rolling deployment. If you manage your ECS services with a Terraform module, such as the ecs-service module, then you can configure the rolling deployment using the deployment_maximum_percent and deployment_minimum_healthy_percent parameters: e.g., if you set deployment_maximum_percent to 200, and you have 3 tasks running, then when you update your service configuration and run apply, ECS will spin up 3 new tasks, wait for them to be healthy, and will then tear down the 3 old tasks. The ecs-service module also supports canary deployments using the desired_number_of_canary_tasks_to_run and ecs_task_definition_canary parameters.
• EC2/ASG: AWS supports rolling deployments for EC2 instances in Auto Scaling Groups (ASG) using instance refresh. If you manage your EC2 instances using a Terraform module, such as the asg-instance-refresh module, every time you make a change and run apply, AWS will automatically spin up a new ASG, wait for it to be healthy, and then tear down the old ASG.

Option 2: implement deployment strategies in custom scripts

Not all Terraform resources have built-in deployment strategies (in fact, most don't) or if they have them, they might not support the deployment strategy you want. In some cases, there's nothing you can do about it: e.g., you can't really do a blue/green deployment for a VPC route table update; either you update it, or you don't.

For some resources, you can implement the deployment strategy yourself using custom scripts. Here are a few examples:

• EKS/K8S: To do a rolling deployment on the worker nodes of an EKS cluster, we implemented an eks deploy command in our open source CLI tool kubergrunt.
• ECS: To do a rolling deployment on the EC2 instances in an ECS cluster, we implemented a roll-out-ecs-cluster-update.py script. Note: ECS just recently added support for managed instance draining, so we may be able to replace this script with native resource support soon!

Option 3: use tools other than Terraform

While Terraform is a great fit for managing your infrastructure, for managing the apps that run on top of that infrastructure, dedicated tools can sometimes be a better fit.

• EKS/K8S: If you install the Argo Rollouts controller in your Kubernetes cluster, you can use the rollout object to do a blue/green deployment by configuring the BlueGreen rollout strategy. Combined with ArgoCD, which you can install using the eks-k8s-argocd module, this can be a way to get blue/green deployments for your Kubernetes services.