Version Control... for Servers
I wanted to pass on an excellent idea that I read from Martin Fowler's Blog. He calls it Immutable Servers, but I claim, if you think about it properly, it is merely the application of version control to systems administration.
Everyone understands just how much version control has transformed the development of software code. It enables developers make changes freely, rolling back changes if they need to. It enables them to look back in history and find out how things stood at any point in time, what was changed on a certain date, or when a given change was introduced. And with advanced usage, it allows "branching", where one can experiment with a group of changes for a long time (while still working on the original branch as well) then merge them together later.
These features aren't just for code. They are great for text documents that get edited frequently. They are a great idea for file systems. And system administrators are familiar with the idea of keeping all of their system administration scripts in a version control system. But some things are extremely difficult to put under version control. Databases are notoriously difficult to version (although Capital One 360 manages it). And servers, being pieces of physical hardware, are impossible to check into Git.
Except that they're not. Servers are not pieces of physical hardware anymore... they were until the last decade, but in recent years that has changed. The vast majority of the servers in our data center either are run or can be run on virtual servers. The current buzzword is "cloud computing", but whatever you call it, we have the technology to spin up and deploy servers from a template in a matter of minutes. (The fact that it takes weeks to get a server set up for your project has nothing to do with technical problems... that's just our own failure to take full advantage of the technology that we own.)
So, given that the servers are probably running on a virtual machine anyway, it's a good idea to keep a virtual machine template with the correct configuration (for quickly restoring the machine). Of course, if you do this you will need to update the template every time you make a significant configuration change. Updating the image doesn't necessarily mean you launch a virtual machine each time, make the change, then save a new image -- you can use tools like Puppet or Chef as part of the image deployment process so often it is just a matter of editing a configuration file.
For the final step, Martin Fowler proposes that you take this to its logical conclusion. If every change needs to be made on the real server AND on the template, why not simplify your workflow (and make it more reliable at the same time) by making the changes directly to the image and deploying a new copy each time. You never change the production server, just roll out a new one each time. This sounds crazy to anyone who hasn't yet drunk the "cloud computing" cool-aid, to anyone for whom creating a new instance of a server takes more than a couple of minutes, but if you DO have an environment that flexible, then you might get all the benefits of version control but for servers. Netflix is one example of a major company that has taken this approach quite successfully.