SheepDog

I did a custom build of the Sheepdog package (the building of which I still need to document), but the installation went along the lines of:

apt install libzookeeper-mt2
apt install corosync  libcorosync-common-dev
dpkg -i /home/rburkholder/sheepdog_1.0+169.g65958e35-1_amd64.deb

Configuration for Sheepdog, is simple, and on a Debian Stretch machine, resides in /etc/default/sheepdog. Mine, I think, is pretty much default for now.

# start sheepdog at boot [yes|no]
START="yes"

# Arguments to run the daemon with
# Options:
#  -p, --port              specify the TCP port on which to listen
#  -l, --loglevel          specify the level of logging detail
#  -d, --debug             include debug messages in the log
#  -D, --directio          use direct IO when accessing the object store
#  -z, --zone              specify the zone id
#  -c, --cluster           specify the cluster driver
DAEMON_ARGS="-b 0.0.0.0 -c corosync  -l dir=/var/log/,level=debug,format=server"

# SHEEPDOG_PATH
#       Proper LSB systems will store sheepdog files in /var/lib/sheepdog.  The init script uses this directory by default.
#       The directory must be on a filesystem with xattr support.  In the case of ext3, user_xattr should be added  to  the
#       mount options.
#
#       mount -o remount,user_xattr /var/lib/shepdog
SHEEPDOG_PATH="/var/lib/sheepdog"

For use with libvirt, there is a fix to perform on a soft link:

rm /usr/sbin/collie
ln -s /usr/bin/dog /usr/sbin/collie

Once the corosync and sheepdog services are configured and running, sheepdog only needs one more command: format the cluster. I used the Erasure Code Support mechanism. The trick here, is based upon the directory (by default '/var/lib/sheepdog') as set in initialization scripts, is where the format command gets applied.

# dog cluster format -c 2:1

#   dog cluster info -v
Cluster status: running, auto-recovery enabled
Cluster store: plain with 2:1 redundancy policy
Cluster vnodes strategy: auto
Cluster vnode mode: node
Cluster created at Tue Oct 31 16:12:13 2017

Epoch Time           Version [Host:Port:V-Nodes,,,]
2017-10-31 16:12:13      1 [172.16.1.21:7000:128, 172.16.1.22:7000:128, 172.16.1.23:7000:128]

# dog node list
  Id   Host:Port         V-Nodes       Zone
   0   172.16.1.21:7000           128  352389386
   1   172.16.1.22:7000           128  369166602
   2   172.16.1.23:7000           128  385943818

This simple setup provides the block storage space to be used by libvirt and the virtualization guests under its control.

Following on from the previous article, this describes a corosync configuration for three appliances configured together in a 'triangle'. OSPF/BGP is running on each appliance. With this routing configuration, I am able to apply an ip address to the loopback interface, and make each of those addresses mutually reachable from each appliance.

I think most corosync examples make the assumption that all nodes are within the same segment. This then suggests a multicast solution. As I am using routing between each appliance, I need a unicast solution.

The following is an example configuration file for the second of three nodes / appliances. Notice the bind_addr is for the loopback address, and a complete list of all three nodes taking part in the quorum. There is a 'mcastport' listed, but because of 'transport: udpu', unicast is actually used from that port number. Continue reading "Corosync in a Three-Some" »

I have been looking at various distributed storage solutions, hoping to find something reliable in an open source style of solution. Some names I've encountered (open and closed source):

• Ceph: by some accounts, seems to be resource heavy, but at the same time, appears to be well used in the industry
• Open vStorage: could be a strong contender for me, but I have a bias against Java based applications.
• Lustre: I've been watching this for quite some time, but the features didn't quite mesh with my desires
• Zeta Systems: a mixture of proprietary and open solutions, which almost fits in with my perceptions, and uses ZFS as the underlying hardware format
• SheepDog: I keep coming back to looking at this. With a version 1 release a little while ago, the developers indicate is satisfies their 'single point of nothing' criteria, which overlap with some of my own criteria. In addition, it appears to be resource light, horizontally scaleable, and integrates with the tools I am trying to integrate: lxc, kvm, and libvirt.

As Debian doesn't have a very recent package built, I build from scratch. Since my test environment is small, I use corosync rather than zookeeper. Here are my build statements for a package build. I will need to add to this to show the build statement as well as the requisite packages:

apt install --no-install-recommends \
  build-essential \
  git \
  corosync corosync-dev \
  libsystemd-dev \
  autoconf \
  m4 \
  pkg-config \
  yasm \
  liburcu-dev \
  libcpg-dev \
  libcfg-dev \
  libfuse-dev \
  libcurl4-openssl-dev \
  libfcgi-dev \
  dh-make \
  devscripts \
  bash-completion \
  libzookeeper-mt-dev
git clone https://github.com/sheepdog/sheepdog.git
cd sheepdog/
git log > debian/changelog
./autogen.sh
./configure --sysconfdir=/etc/sheepdog --enable-corosync --enable-sheepfs --enable-http --enable-nfs --enable-systemd
make deb

From Which Format of QEMU Images Should I Run, there is this table:

| format          | snapshot/clone  | thin-provsion  |   DISCARD    |  encryption  |  compression
 raw over file    :      NO         |      YES       |      NO      |      NO      |       NO
 raw over sheepdog:      YES        |      YES       |      YES     |      NO      |       NO
qcow2 over sheepdog:     YES        |      YES       |      YES     |      YES     |       YES

Some links:

• Sheepdog is Ready: distributed block storage is turning from experiment to production use. has performance test scenarios and background on durability, scalability, manageability, and availability (can be run with multipath scsi targets).

On the Sheepdog mailing list, a mechanism, other than sheepfs, a way to present a file system:

You can do, through qemu-nbd, formatting it and mounting it.

sheepdog -> qemu-nbd -> /dev/nbd{x} -> xfs/ext3/ext4/.. -> mount

modprobe nbd
qemu-nbd sheepdog://localhost:7000/my_volume -c /dev/nbd1
# Optionally you can do the rest on a different machine using nbd-client on this step
mkfs.xfs /dev/nbd1
mount /dev/nbd1 /path/to/mount