Some choice extractions from Configuration Management is an AntiPattern:
.... you’ve got an operations team that bottlenecks all changes in production. They’re the only ones with commit access to the configuration management repository, and they have to make all changes in production. This is the anti-DevOps way of doing things.
The other option is that you expect developers to run configuration management on the clusters that they are responsible for. This is the much more DevOps way of doing things, but it suffers from a different problem. Now you have to teach all your engineers the DSL of your configuration management software of choice, and depending on how you deploy your code to production, every developer now has the power to take down your whole system with a poorly written configuration change.
... you just have a separate configuration management branch for every cluster you run in production, and restrict developers to running code on only the systems they manage ...
configuration management promises that you’ll know the complete state of your infrastructure, but it never works that way
The alternative is immutable infrastructure. If you’re running in the cloud, this means baking your AMI with your application code already on it. See, imagine a world where you finish writing the code, you push it into git, and it gets built into an RPM or Debian package. That package is installed on a Base-AMI that was carefully crafted by your security and performance engineers, and then an AMI, with your software installed, along with its dependencies is pushed out
Start with a base, or foundation image — this is either your handcrafted, optimized image or just the default image from an upstream vendor. ... Your base image should have the latest security updates as well as any base infrastructure packages that are run platform-wide. Things like your monitoring packages, or your service discovery. Now, you could use a configuration management tool to build your base image, but you could also use OS packages and a little python to install and configure your base image.
This enables continuous deployments because new code just goes through your pipeline and you don’t have to deal with old versions of libraries or dependencies or configuration that might have been left around.
Moving to immutable allows you to take advantage of some cool release strategies. The first one is a rolling release, where nodes are replaced one at a time. This can be handy when you have state on your cluster that needs to be preserved.
My favorite release strategy is the Blue/Green push. You have 100 nodes, you start up 100 new nodes, and then you move traffic to the new nodes. If you have issues, you can quickly move traffic back to the old nodes. After a reasonable window, say 1–3 hours, you shutdown the old nodes.