AWS CodeDeploy, Blue/Green Deployments, and CloudFormation

If you’re the type to read the documentation, you may have noticed, buried inside the CloudFormation for CodeDeploy documentation, this sentence:

AWS CloudFormation supports blue/green deployments on AWS Lambda compute platforms only.

That’s rubbish, we say. Why not on EC2 instances? Well, OK then, but there’s nothing stopping me from manually changing the configuration of the CodeDeploy deployment group later on. After all, that is what AWS’s own example does!

It turns out that there’s actually a very good reason why this is not a supported configuration. CodeDeploy is going to be referring to an Auto Scaling Group, and it’s likely that you have defined that ASG in your CloudFormation template, and used !Ref to link the deployment group to it.

The problem comes when you do your first blue-green deployment. CodeDeploy does a blue-green deployment by cloning the ASG to make a new one with the same parameters, but whose instances run the newer version of your application. Once the new version is deployed and healthy, it winds down the old ASG. Then, it updates the deployment group configuration to point to the new ASG, and deletes the old ASG.

This is fine until you go back to your CloudFormation template and need to make a change that touches the ASG. CloudFormation will try to update the ASG that it created, only to find it doesn’t exist any more. Boom, template update failure and rollback.

My reluctant conclusion has been to say that the ASG needs to be managed manually. I have two CloudFormation templates - one creates everything up to but excluding the ASG (such as the VPCs and subnets, and the launch configuration that the ASG will use), and another one which contains the CodeDeploy configuration and everything else required. The second takes a parameter which is the name of the manually-created ASG.

After this point, care still needs to be taken - if I need to update the second stack, I will need to update the parameter to provide the current name of the ASG, which will have changed if any blue-green deployments have happened.

If anybody else has a smarter solution, please leave a comment. But I suspect that we will have to leave this up to the AWS CloudFormation team to solve.