By Jussi Heinonen

About

This project provides development and testing environment for applications utilising RabbitMQ message brokers (rabbitmq.com).

The goal for the project is to offer easy-to-use environment that can be quickly destroyed and rebuilt on variety of platforms (Windows, Mac and Linux).

The default configuration deploys 4 virtual machines: A load balancer node and 3 RabbitMQ Server nodes that forms a message broker cluster (1 disk node + 2 ram nodes).

Virtualisation layer is provided by Oracle Virtualbox (virtualbox.org) and configuration management is controlled by Puppet (puppetlabs.com).

The overall provisioning process is coordinated by Vagrant deployment framework (vagrantup.com).

Environment Diagram

Prerequisites

Internet connectivity (to access software package repositories)
Virtualbox 4.1.6 or later (provides virtualisation layer)
Ruby 1.9.x + Vagrant gem (for environment provisioning)
Python 2 + Pika AMQP client library (to run Python test scripts)
Git (download environment updates from source control)

One-off Configurations 

To get started with your own RabbitMQ development and testing environment ensure you fulfill the requirements as listed in the Prerequisites paragraph. 

Virtualbox

First thing to do is to download Oracle Virtualbox from virtualbox.org and install the software.

If your operating system is Linux it is recommended to check if Virtualbox is available in package repositories.

Platform specific installation instructions are available on Virtualbox website at https://www.virtualbox.org/manual/ch02.html.

Ruby 1.9.x and Vagrant gem

For Mac and Linux users Ruby environment is available in package repositories and you can install Ruby 1.9.x using preferred package manager.

For example on Ubuntu Ruby can be installed by running command sudo apt-get install ruby1.9.1-full.

Windows users do not enjoy the convenience of package repositories so you have to download and install Ruby from rubyinstaller.org/downloads. Download and install the latest 1.9.x release as well as the Development Kit package. Devkit is required to build and use native C/C++ extensions on Windows.

Once Ruby 1.9.x (and DevKit for Windows uesrs) is installed you should install the Vagrant gem. Ruby 1.9.x comes with built-in gem support so you can simply run command gem install vagrant to install the software package. 

To check that the Vagrant gem was successfully installed you can run command gem list –local and this should show locally installed gems such as vagrant and its dependencies.

Python and Pika AMQP Library

Python environment and Pika AMQP Library is only required to run test scripts (available in directory /rabbitmq-on-vagrant/scripts once Git project has been checked out locally). In case you use other tools for testing you may skip this paragraph.

Like Ruby also Python is available in package repositories for Mac and Linux users. For example to install Python and Pika on Ubuntu you need to run the following 2 commands: sudo install python python-pip then sudo pip install pika.

For Windows users it is recommended to use one of the Python installers such as ActivePython available in www.activestate.com/activepython/downloads. Python package manager called PyPM is included in ActivePython distribution so you may install Pika library by opening the Windows Command Prompt and running command pypm install pika.

Git installation and checkout

Git is required to get the latest environment configuration files from the Github project. If you intend to do checkout only once then you can skip the Git installation and download the latest files as .zip bundle available in https://github.com/jussiheinonen/rabbitmq-on-vagrant/downloads.

Git installation using distribution specific package repositories is as simple as installing Ruby or Python. For example on Ubuntu you can run command sudo install git. 

Git installer for Windows is available on Git website at http://git-scm.com. Do the usual double-clicking to start the installer and hit Next button until you see the Finish button. 

Once Git is installed you can create a local copy of the remote Gitrepository by running command git clone git://github.com/jussiheinonen/rabbitmq-on-vagrant.git (or git clone https://git@github.com/jussiheinonen/rabbitmq-on-vagrant.git).

This command will create a directory called rabbitmq-on-vagrant in the current work directory.

Download Virtual Machine Image

The final task is to get a copy of a virtual machine image from vagrantbox.es website. Vagrantbox.es contains good variety of virtual machine images that are compatible with Vagrant provisioning framework. 

This project has been developed and tested on Ubuntu 10.04 so it is recommended to select an image of this flavour and version.
Compatible images are:

http://files.vagrantup.com/lucid32.box– Ubuntu 10.04 32-bit

http://files.vagrantup.com/lucid64.box – Ubuntu 10.04 64-bit

Add Virtual Machine Image to Vagrant virtual machine inventory

Once virtual machine image has been downloaded it is time to add virtual machine in the inventory. This is done using vagrant.
On command line change to directory where downloaded machine image resides.

In this directory run command vagrant box add lucid32-1 <machine.box> (replace the <machine.box> with the name of the image file).
The display name lucid32-1 has important meaning to Vagrant.

This name must match the value of parameter node_config.vm.box specified in Vagrantfile located in directory /rabbitmq-on-vagrant/vagrant/environments/rabbitmq/. 

If you add box with different display name remember to update the file /rabbitmq-on-vagrant/vagrant/environments/rabbitmq/Vagrantfile accordingly.

Adding machine image to inventory should complete quickly and the output of the command when successful should look something of the following:

[vagrant] Downloading with Vagrant::Downloaders::File...
[vagrant] Copying box to temporary location...
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...

Create and destroy the environment

Environment preparation is now complete and we are ready to spin-up the environment. 

Vagrant up

Go to directory /rabbitmq-on-vagrant/vagrant/environments/rabbitmq/ and run command vagrant up.
Command vagrant up will start the environment provisioning process which takes about 10 minutes to complete.

Vagrant destroy, vagrant halt/resume

To shutdown the environment run command vagrant destroy. This command will shutdown the environment and erase all data stored on the virtual machines. 

To shutdown the environment without destroying the data you can run commad vagrant halt and to start the environment again run command vagrant resume.

Accessing Virtual Machines using Vagrant SSH command

You can use command vagrant ssh <vmname> to access Virtual Machines on command line.

For example to connect to haproxy Virtual Machine use command vagrant ssh haproxy. You can view a list of known Virtual Machines by running command vagrant status.

SSH on Windows

Windows users are unable to access Virtual Machines directly from Command Prompt using vagrant ssh command. In case you run Vagrant on Windows platform you should use Putty or similar SSH client software to access Virtual Machines. Use the following IP addresses to connect to Virtual Machines:

haproxy => 192.168.2.2
rabbitmq-server-a => 192.168.2.10
rabbitmq-server-b => 192.168.2.11
rabbitmq-server-c => 192.168.2.12

VM template comes with preconfigured user account called vagrant that uses a password vagrant. If you prefer not having to type in the username and password every time you log in you can configure public key authentication using SSL keys provided in the Vagrant gem package.

Default SSL keys are located in <ruby_gems_dir>/vagrant-1.0.1/keys/.

Accessing HAProxy status page

Haproxy VM that runs haproxy load balancer service has a status page available at http://192.168.2.2. Status page provides useful information on the state of the VM nodes within the cluster.

Testing

Once all RabbitMQ server nodes are showing status up (green colour) on HAProxy status page you can test the cluster functionality.

In project checkout directory you can find a subdirectory called scripts. In this directory you can find 2 Python script files:

1. rabbitmq_consumer.py – A RabbitMQ consumer script that connects to port 5672 on IP 192.168.2.2 (haproxy server address). Script consumes messages from RabbitMQ message queue and prints messages in standard out. Script disconnects from RabbitMQ Server when procer sends a quit message.Testing

2. rabbitmq_producer.py – A RabbitMQ producer script that connects to port 5672 on IP 192.168.2.2 (haproxy server address).

When running script without parameters script creates 30 messages with a sequence number as a payload ( 1st message payload “1”... last message payload “30”) and sends messages to RabbitMQ message queue.

When running the script with parameter quit it produces a single message that is sent into queue with payload quit which is a signal to consumer to disconnect from the server.

Run both scripts with prefix python, eg. # python ./rabbitmq_consumer.py and # python ./rabbitmq_producer.py

Customising the environment

The default environment configuration provides you with a load balancer and 3 RabbitMQ nodes.

Depending on the requirements of your application you may want to build a RabbitMQ cluster that contains more than 3 RabbitMQ nodes or you may want to add a new node that is dedicated to run some other application.

Adding and removing nodes in the RabbitMQ cluster

Number of Virtual Machines is defined in the Vagrantfile that resides in directory rabbitmq-on-vagrant/vagrant/environments/rabbitmq/.
Virtual Machines are declared in the array called nodes (starting on line 10 in Vagrantfile).

Line 10: nodes = [
Line 11: {:hostname => 'haproxy', :ip => '192.168.2.2',:fwdhost => 5672, :fwdguest => 5672},
Line 12: {:hostname => 'rabbitmq-server-a', :ip => '192.168.2.10'},
Line 13: {:hostname => 'rabbitmq-server-b', :ip => '192.168.2.11'},
Line 14: {:hostname => 'rabbitmq-server-c',:ip => '192.168.2.12'},
Line 15: ]

Removing a node

To remove a node from the cluster you can simply comment out (#) lines that you wish Vagrant to ignore.

For example to run a single instance of RabbitMQ server with no load balancing you should comment out lines 11 (haproxy), 13 (rabbitmq-server-b) and 14 (rabbitmq-server-c).

Run vagrant up to start your cluster in single-node mode.

To remove a node from the cluster that is already up and running you can simply run command vagrant destroy <:hostname> to destroy a node.

Adding a node

To add new RabbitMQ node into cluster the easiest method is to take a copy of one of the lines that declares a rabbitmq-server node.

For example take a copy of line 14 (rabbitmq-server-c) and add a new line before line 15 and modify parameters :hostname and :ip so that parameters won't conflict with parameters associated with other nodes. 

When declaring a new RabbitMQ node it is important that :hostname parameter starts with string rabbitmq-server (eg. rabbitmq-server-d) for reason because :hostname is used by Puppet configuration management tool and :hostname rabbitmq-server acts as a signal to Puppet to configure the node as a RabbitMQ server and join it into cluster with other RabbitMQ nodes. 

Run vagrant up to start your cluster with newly added node. If you are adding a node into cluster that is already running you should also run command vagrant provision haproxy which is a signal to Puppet to reconfigure HAProxy by adding a new node into load balancer configuration.

Puppet

Puppet configuration management tool takes care of the installation and configuration of the software stacks. 

Puppet manifests reside in directory called puppet that is found in the root checkout directory. This directory has the following structure:

puppet
├── manifests (the efault manifest directory, contains a file called site.pp)
├── modules (core modules, a collection of manifests used for building software stacks)
└── services (service configuration , contains a file called rabbit/nodes/nodes.pp)

When system starts up and Puppet daemon is run Puppet executes manifests in the following order. 

Read the file site.pp under the manifests directory. Site.pp instructs puppet to look for nodes.pp files under services directory.

File services/rabbit/nodes/nodes.pp contains node definitions that specify a collection of Puppet modules, located in modules directory, that are to be loaded on the nodes.

Runtime configuration using Puppet facts

Puppet Facts provides a mechanism to extract information from the runtime environment and use this information with Puppet manifests and templates to apply configuration that is specific to a node.

In addition to Puppet built-in facts the project has 3 custom facts that are contained within modules under directory.

Fact: rabbitnodes

A file modules/haproxy/lib/facter/nodes.rb contains a fact that extracts information related to RabbitMQ nodes specified in the Vagrantfile.

When facter is called it returns a list of hostnames and ip addresses of the RabbitMQ nodes.

The list is processed in ERB template file under modules/haproxy/templates/haproxy.cfg.erb and it generates HAProxy configuration file into /etc/haproxy.cfg on HAProxy node.

Fact hostentries

To create RabbitMQ cluster each RabbitMQ node has to be aware of one another. Fact hostentries in modules/rabbitmq/lib/hostentries.rb extracts hostname and IP address information from Vagrantfile and list of hosts is processed by ERB template under modules/rabbitmq/templates/hosts.erb.

Template generates a file /etc/hosts on RabbitMQ nodes which enables them to communicate using hostnames instead of IP addresses.

Fact: rabbitcoordinator

Located in the directory modules/rabbitmq/lib is a file called rabbitcoordinator.rb that declares a fact called rabbitcoordinator. When fact is called it returns a name of the first RabbitMQ node specified in the Vagrarantfile.

The hostname value returned by the rabbitcoordinator fact is used to detect whether the node that is running the fact is the first RabbitMQ node in the cluster (rabbitmq-server-a) or whether to join the node into existing RabbitMQ cluster (rabbitmq-server-b and c).

Logic for decision making is implemented in the manifest file modules/rabbitmq/manifests/init.pp.

Geppetto

For Puppet manifest and fact development there is a handy Eclipse plug-in called Geppetto written by cloudsmith.com.

Geppetto understands Puppet manifest language and it provides syntax highlighting and warnings when making a spelling mistake or forgetting to add the closing curly bracket in the manifest of your own.

Whether you are new to Puppet or more experienced manifest developer Geppetto will improve the productivity and ensures the quality of your manifests. Installation instructions for Geppetto can be found on Puppetlabs website at http://puppetlabs.com/blog/geppetto-a-puppet-ide/