Sheepdog on CentOS 6

I am in the process of trying to solve a problem at $WORK. The main issue is we try to keep our servers in a hardware rotation schedule. Things like iSCSI disk arrays are hard to work into that rotation, so I am hoping that sheepdog will serve as a way to aggregate the storage that we already purchase with each server giving us a fault tolerant storage that we can easily upgrade as we grow, or replace servers.

What is Sheepdog:

From the sheepdog wiki: Sheepdog is a distributed storage system for QEMU. It provides highly available block level storage volumes that can be attached to QEMU-based virtual machines. Sheepdog scales to several hundreds nodes, and supports advanced volume management features such as snapshot, cloning, and thin provisioning.

Technology Stack

We will be using CentOS as the host. We will then compile Corosync, Qemu and Sheepdog from source to get a working test cluster. My test set up consists of two Dell R300 servers. One thing to note is that I could not get this working in a KVM virtual machine and ended up just using the spare hardware I had available, so be aware of that limitation.

Getting started

I am assuming a few things:

1. That you have SELinux set to disabled or permissive. Getting SELinux working with this set up is a future goal but for this simple proof of concept the added complexity/security is not needed.
2. You are starting with a minimal install. This post will list out the required packages to install to be able to compile the programs
3. You have opened up at least ports 5404-5405 udp (for corosync) and 7000 tcp (for sheepdog) in your firewall
4. Corosync from the repository is not installed. It is too old to be of use for us.

Install the required packages:

 yum install automake make gcc git nss-devel zlib-devel  

Compile and install userspace_rcu:

 wget http://lttng.org/files/urcu/userspace-rcu-0.7.3.tar.bz2  
 bunzip2 userspace-rcu-0.7.3.tar.bz2 && tar userspace-rcu-0.7.3.tar  
 cd userspace-rcu-0.7.3  
 ./configure  
 make install   

Compile and install corosync:

 git clone git://github.com/corosync/corosync.git  
 cd corosync  
 git checkout -b flatiron origin/flatiron  
 ./autogen.sh  
 ./configure --enable-nss  
 sudo make install  

Compile and install sheepdog:

 git clone git://github.com/collie/sheepdog.git  
 cd sheepdog  
 ./autogen.sh  
 ./configure  
 sudo make install  

Compile and install qemu:

 git clone git://git.sv.gnu.org/qemu.git  
 cd qemu  
 ./configure  
 sudo make install  

 Configure Corosync:

Copy the configuration file provided by the sheepdog wiki. Change the
bindnetaddr to your network ip address. See the man file for more information on this if you are confused.

Start Up and Test:

Start up corosync on each host and look at /var/log/cluster/corosync.log and make sure there are no errors reported. If you are using ext3 or ext4 you need to make sure that your disk is mounted with user_xttr support:

 mount -o remount,user_xattr /  

Next start up sheepdog on each host and format the cluster:

 collie cluster format --copies=2  

Then make sure that the nodes see each other by issuing a collie node list, finally you can create a disk using qemu-img:

 qemu-img create sheepdog:Demo 8G  

Conclustion

You now have a fault tolerant cluster that you can painless scale out your storage as you grow. Before you start running this in production you will want to run some tests to see how it responds to a node going down and other possible scenarios, also turning on and setting up SELinux would be a good thing.