Testing configuration files before they go live - with Ansible

One of the great tips I picked up recently was how to test or validate your templated configuration files in Ansible.
see http://jpmens.net/2015/02/26/validate-ansible-templates. (I came across this in the excellent Briefs on Ansible newsletter btw - subscribe! https://devopsu.com/newsletters/ansible-weekly-newsletter.html)

The template module

The template module enables you to take a text file such as a configuration file and replace values with variables using Jinja2 syntax.
This file is then copied from the source template location (via the templating process) to the remote host (via an Ansible temporary folder on the server).
This is particularly useful for setting configuration values for applications on the host such as SSH, Apache, Nginx, PHP, etc.
One issue with this approach though is that the templated file may contain an error and when the service that relies upon it is restarted it may fail. Another good parameter of the template module is 'backup' - which will at least enable you to roll back to the last known good version.
What this post (http://jpmens.net/2015/02/26/validate-ansible-templates) showed me was that the template module has a setting 'validate' that will run the templated file through a configuration test to ensure that it will work - you supply the syntax of the test.
Why do we need this? Imagine you are putting up an Apache Vhost file and then restarting the Apache server, if the file has errors then Apache will not restart and you'll be in the dark (literally) with the web server down.
If you were doing this manually you would run;

( -t for short) and this would notify you of any errors in your config files; giving line numbers and some explanation - very handy.
With the -t option, apachectl will test the configuration files that Apache will use if restarted. If you use the -f option you can pass in a file to test - great stuff - so we can use this to test our file with the template module prior to it being copied to production.

Validate your templates

So with the template module this is a great find! and once found I intended to update all of my tasks to use it.

This way tasks would fail and configuration wouldn't go live if invalid - nice. By the way the actual check resolves to the following:

a long path to a temp copy of the file templated prior to being copied into place.
Well not quite so - as although this works great for many other tasks (checking sshd config for example), it doesn't work for Apache. (Note this is not a failing of Ansible or apachectl!)
You will get something along the lines of:

That scuppered me for some time - as if I put the configuration live the test was fine. In fact I checked it on several Apache versions on several OS - still no joy. What's more the syntax IS fine (the exact error will vary btw - here it is complaining about the line Order deny, allow (and no - it's not Apache 2.4 - I had been using Apache 2.4. when I came across the error and tested it back on 2.2, still the same). You will get this error with Apache 2.4 if you use the old 2.2 syntax - but that is another story.
Stumped (and I have been using Apache for some time and testing config files prior to a reload/restart- although admittedly not using the -f flag to pass in a config to the test facility of apachectl).
In this case it is complaining about the Order option - which requires authz_host or one of the other auth modules which were enabled (they are enabled by default!) - so it appeared that apachectl in test mode is not loading all of the enabled modules.
At first I though this must be a bug with Apachectl and investigated further - until *spark* realizing that apachectl is doing exactly what it says - testing your config file - and only your config file and not all of the other config files that are loaded via apache.conf (httpd.conf), conf_enabled (conf.d), mods_enabled/* etc etc etc
To test this I added the relevant includes to the test vhost and the syntax tested OK.
So you can't use validate in this case, as apachectl doesn't accept a list of config files to use and only tests yours with no reference to the other configs you may expect to be there.

A solution - pattern

To solve this I came up with the following pattern:

1. write your config as normal (into sites_available)
2. template a test file that contains includes for conf_enabled , mods_enabled and includes your config file
3. validate that and if that succeeds link your vhost into the sites_enabled
4. otherwise report a failure and then clean up the temp config test file.

See the gist below.

Update: I have added the line ErrorLog /var/log/apache2/error_log to the config test file as Apache is compiled (in some cases) with a default log location of:

and that folder is unlikely to exist - odd really as the standard envars file uses /var/logs/apache2/
So in some cases you may get:

If you don't add the env setting for the log file.

Back to the source

In fact, by chance, searching for further clues to this issue, I went back to the original article and since my initial visit two commenters had come across the same issue (the only references to this I could find anywhere - everywhere else on the web everyone thinks this works!!).
One commenter has a similar solution and I have updated the gist above to add some of his suggestions - such as a nice debug message.
This solution offered by Mark (http://pastebin.com/WnPKVuqU) does a similar thing except the patterns is:

1. put up the configuration
2. test the configuration (apachectl -t)
3. if it fails roll back the configuration and report

The advantage this has over my approach is that it tests all of the configuration that will be live - the disadvantage is that your configuration is live (albeit not loaded) and could be left so if one step failed.
The approach offered by Markdphillips is to update the standard handler to restart Apache to use the shell module to test the config prior to reloading

This would also leave the config in place in the meantime.
The pattern I use has the advantage of your config doesn't go live until tested (but the disadvantage that you are only testing it against the core configs).

There is good list of ways to test configuration files for a range of applications here - http://www.cyberciti.biz/tips/check-unix-linux-configuration-file-for-sy...