I have been looking at various distributed storage solutions, hoping to find something reliable in an open source style of solution. Some names I've encountered (open and closed source):
- Ceph: by some accounts, seems to be resource heavy, but at the same time, appears to be well used in the industry
- Open vStorage: could be a strong contender for me, but I have a bias against Java based applications.
- Lustre: I've been watching this for quite some time, but the features didn't quite mesh with my desires
- Zeta Systems: a mixture of proprietary and open solutions, which almost fits in with my perceptions, and uses ZFS as the underlying hardware format
- SheepDog: I keep coming back to looking at this. With a version 1 release a little while ago, the developers indicate is satisfies their 'single point of nothing' criteria, which overlap with some of my own criteria. In addition, it appears to be resource light, horizontally scaleable, and integrates with the tools I am trying to integrate: lxc, kvm, and libvirt.
As Debian doesn't have a very recent package built, I build from scratch. Since my test environment is small, I use corosync rather than zookeeper. Here are my build statements for a package build. I will need to add to this to show the build statement as well as the requisite packages:
apt install --no-install-recommends \ build-essential \ git \ corosync corosync-dev \ libsystemd-dev \ autoconf \ m4 \ pkg-config \ yasm \ liburcu-dev \ libcpg-dev \ libcfg-dev \ libfuse-dev \ libcurl4-openssl-dev \ libfcgi-dev \ dh-make \ devscripts \ bash-completion \ libzookeeper-mt-dev git clone https://github.com/sheepdog/sheepdog.git cd sheepdog/ git log > debian/changelog ./autogen.sh ./configure --sysconfdir=/etc/sheepdog --enable-corosync --enable-sheepfs --enable-http --enable-nfs --enable-systemd make deb
From Which Format of QEMU Images Should I Run, there is this table:
| format | snapshot/clone | thin-provsion | DISCARD | encryption | compression raw over file : NO | YES | NO | NO | NO raw over sheepdog: YES | YES | YES | NO | NO qcow2 over sheepdog: YES | YES | YES | YES | YES
Some links:
- Sheepdog is Ready: distributed block storage is turning from experiment to production use. has performance test scenarios and background on durability, scalability, manageability, and availability (can be run with multipath scsi targets).
On the Sheepdog mailing list, a mechanism, other than sheepfs, a way to present a file system:
You can do, through qemu-nbd, formatting it and mounting it.
sheepdog -> qemu-nbd -> /dev/nbd{x} -> xfs/ext3/ext4/.. -> mount
modprobe nbd
qemu-nbd sheepdog://localhost:7000/my_volume -c /dev/nbd1
# Optionally you can do the rest on a different machine using nbd-client on this step
mkfs.xfs /dev/nbd1
mount /dev/nbd1 /path/to/mount
