Provisioning Sensu with Puppet

In my last post we have installed RabbitMQ with Puppet on our CentOS 6.2 box. Today, we continue by installing Sensu, a pretty cool monitoring framework all the hippsters are going for these days.

This post is going through the Sensu installation guide and will provide the pointers to my GitHub repository where I have committed my puppet configuration. For the first time, am not repeating how to build up your box with vagrant and so forth. Just check my previous posts.

For a change I will not use the command line other than adapting and extending my puppet configuration.

So, you are logged into your box as root? Fine. Ready to go!

Before We Get Started #

There is a left-over from our RabbitMQ installation. We need to remove a directory and create some symbolic links.

{% highlight none %} cd /etc/puppet rm -rf manifests ln -s /vagrant/manifests ln -s /vagrant/modules {% endhighlight %}

You need that in case you want to invoke your provisioning run from within the virtual machine with puppet apply. If not, you can always do vagrant provision from outside your box on your host system.

In case you start with my vagrant box, you also need to install git

{% highlight none %} yum -y install git {% endhighlight %}

Generating and Installing SSL-Certificates #

There is another left-over. I did not add the SSL certificates to the puppet configuration. Since I already did the manual procedure here, the rest was a piece of cake.

First, the certificates need to be gnerated with

{% highlight none %} git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git cd joemiller.me-intro-to-sensu/ ./ssl_certs.sh clean ./ssl_certs.sh generate {% endhighlight %}

Then I have created the following directory

{% highlight none %} /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/ {% endhighlight %}

and copied the following files into it

{% highlight none %} cp server_key.pem /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/ cp server_cert.pem /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/ cp testca/cacert.pem /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/ {% endhighlight %}

Now, we need to extend our puppet configuration in the file

{% highlight none %} /vagrant/modules/rabbitmq/manifests/init.pp {% endhighlight %}

For a test I did only one file.

{% highlight none %} file { ‘/etc/rabbitmq/server_key.pem’: source => ‘/etc/puppet/modules/rabbitmq/files/etc/rabbitmq/ssl/server_key.pem’, owner => ‘root’, group => ‘root’, mode => ‘644’, notify => Service[‘rabbitmq-server’], require => Package[‘rabbitmq-server’], } {% endhighlight %}

We also need to tell puppet, that we need the directory /etc/rabbitmq/ssl, since it does not exist yet.

{% highlight none %} file { ‘/etc/rabbitmq/ssl/’: source => ‘/etc/puppet/modules/rabbitmq/files/etc/rabbitmq/ssl/’, owner => ‘root’, group => ‘root’, mode => ‘644’, } {% endhighlight %}

Also add this file resource in the require part of the service resource.

Let’s try our new configuration without changing anything by providing the --noop parameter.

{% highlight none %} cd /etc/puppet puppet apply –verbose manifests/site.pp –noop {% endhighlight %}

Then I have added the other files as well and did another test run. Everything went fine, so I did a real provisioning run.

One more thing we need to do for RabbitMQ, we need a config file that tells RabbitMQ to use SSL and where to find those certificates.

{% highlight none %} vim /etc/puppet/modules/rabbitmq/files/etc/rabbitmq/rabbitmq.conf {% endhighlight %}

and paste the following into it

{% highlight none %} [ {rabbit, [ {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"}, {certfile,"/etc/rabbitmq/ssl/server_cert.pem"}, {keyfile,"/etc/rabbitmq/ssl/server_key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,true}]} ]} ]. {% endhighlight %}

Now add another file resource entry for this configuration file to your init.pp of your puppet RabbitMQ module and provision it.

To make sure, let’s check our target directory.

You know what? That thing really works! We have just did that with one server, but we could have easily done it with plenty more servers. Would have been no deal at all.

Okay, after all this I committed everything and pushed it up to GitHub on my host system.

{% highlight none %} git add . git commit -m “Added the use of SSL to RabbitMQ” git push origin master {% endhighlight %}

Sensu Puppet Configuration #

Finally I have added a new module for Sensu to my puppet configuration and added it in my nodes.pp that looks now like this

{% highlight none %} node dev_ops_central { include cron include apache include rabbitmq include redis include sensu } {% endhighlight %}

Then I worked myself through the installation guide, including the howto of adding a check and adding a handler and translated that into a puppet configuration. Am not going to repeat everything here, since now it is your time to work yourself through the puppet code.

Finally I was able to fire up everything, but somehow it did not work. When stopping cron, I did not get any notification in my sensu frontend. As it turned out, I had some issues with my already existing RabbitMQ installation. Actually, it was not using SSL, since the config file had a wrong name. It is NOT rabbitmq.conf but rabbitmq.config. It took me a while to find that out with rabbitmqctl status saying that it is ignoring my rabbitmq.conf and using its own configuration.

In the meantime, while writing and polishing this post, my sensu code has been refactored by @linuxaddicted. Uuh, wait a minute … He also added some RSpec tests! Perfect! Thanks a lot. More on this RSpec thing soon.

Beware of the Brownfield #

For installing the sensu-plugin with

{% highlight none %} gem install sensu-plugin {% endhighlight %}

I first had to install the ruby development libs with

{% highlight none %} yum install ruby-devel {% endhighlight %}

I will make that part of my box and not deploy it separately.

Additionally, I had to install gcc. Not an option on potential production boxes.

Feature-Branching with Git #

For the first time I was using a feature branch called sensu, since I did not want to commit half-baked code to my master branch. Actually, I love the idea of Continuous Delivery, where you actually should not do feature branches, but need to find my way to that approach.

Here is the “process” of how to create a branch with git, add your code, commit it and push it to GitHub.

{% highlight none %} git checkout -b sensu git branch git commit -am “my message” git push origin sensu {% endhighlight %}

When I was sure everything worked fine, I merged everything back to the master branch and removed the branch sensu locally and remote on GitHub.

{% highlight none %} git checkout master git merge sensu {% endhighlight %}

Now, there is no need any more to keep the sensu branch, so I have removed it locally and remotely with

{% highlight none %} git branch -d sensu git push origin :sensu {% endhighlight %}

Having done all this bunch of work, it was also a good time to tag our master branch.

{% highlight none %} git tag provisioning_sensu_with_puppet git push –tags {% endhighlight %}

Wrapping It Up #

We almost have a complete automation from box creation to provisioning RabbitMQ and Sensu with Puppet. We can now extend Sensu with more checks and handlers to improve our monitoring and alerting mechanisms. But this is just the beginning of the story. There is so much to explore and to learn. Stay tuned. But for now …

Done for today!