Now that the networking side of things are complete with Quagga and OpenVSwitch, I can start working further up the technology layers. The next step is to work on virtualization technologies and their management. The focus of this article is to get Linux Containers (LXC) working within a Debian Wheezy 7.1 environment.
Monday, August 12. 2013
Linux Containers (LXC) on Debian Wheezy 7.1 With OpenVSwitch
Before getting into the installation steps of lxc, some background first. LXC is not a full virtualization solution with an independent kernal and bios, such as provided by Qemu/Kvm. Instead, it runs in a namespace of the host's kernel. As such, it has very fast startup characteristics. The draw backs are several: a) it doesn't offer the tighter security that something like OpenVZ offers (another container flavour), b) it is a more recent container solution and has much catching up to do, and c) and one of things it needs to catch up with is live migration of sessions from one host to another.
János Pásztor goes into some of the differences between LXC and OpenVZ. The disadvantage of OpenVZ is that it has not been welecomed in to recent versions of the Linux Kernel. It appears as though the game plan is the passing of the baton from OpenVZ to LXC.
From an LXC perspective, its usefulness is good when using it in environments where the administrator has control over its integration. It is not a viable candidate when trying to set hosting solutions for third parties. Tighter security for those scenarios can be supplied with a Qemu/Kvm style solution. Some additional reservations regarding OpenVZ and LXC.
The basics of installation for LXC are straight-forward:
aptitude install lxc libvirt-bin
The first package deals with lxc directly. If I wasn't using OpenVSWitch, lxc would be all that I require. But in order to use OpenVSwitch for the networking side of things, I need assistance from the libvirt system. As a side effect of the libvirt install, the Qemu/Kvm compontents get installed at the same time. However, since I'm focussing on lxc at the moment, Qemu/Kvm will be discussed in a different entry.
A manual entry into /etc/fstab is required followed by a reboot to resolve a "resource busy problem" as well as taking care of the mount:
cgroup /sys/fs/cgroup cgroup defaults 0 0
There is some explanation for control groups (cgroup) at Control Groups Resource Management.
Before performing the reboot, one other adjustment is required. The following needs to replace the 'GRUB_CMDLINE_LINUX=""' in /etc/default/grub:
GRUB_CMDLINE_LINUX="cgroup_enable=memory"
Then run the command 'update-grub'. Then perform the reboot. This is from the Debian Wiki on LXC.
After which, running 'lxc-checkconfig' should result in 'yes' for all items.
We can now move on to creating an lxc container. But even this isn't completely straightforward. Wheezy was released unstable in this regard, no functioning lxc template. Instead, based upon Horrors using Debian Wheezy, we need to obtain the template from elsewhere. LXC template lxc-debian-wheezy-template talks about the template.
The template uses debootstrap as part of the build process. The supplied script can be easily modified to install additional packages, or to choose different install configurations.
wget http://freedomboxblog.nl/wp-content/uploads/lxc-debian-wheezy.gz gzip -d lxc-debian-wheezy.gz mv lxc-debian-wheezy /usr/share/lxc/templates chmod +x /usr/share/lxc/templates/lxc-debian-wheezy
Because I am doing this on an amd64 flavour, some lines in /usr/share/lxc/templates/lxc-debian-wheezy needs to be modified:
rootfs=$1/rootfs-amd64 # the following are commented out, not proper #1:2345:respawn:/sbin/getty 38400 console #c1:12345:respawn:/sbin/getty 38400 tty1 linux #c2:12345:respawn:/sbin/getty 38400 tty2 linux #c3:12345:respawn:/sbin/getty 38400 tty3 linux #c4:12345:respawn:/sbin/getty 38400 tty4 linux # instead use these: 1:2345:respawn:/sbin/getty 38400 tty1 #2:23:respawn:/sbin/getty 38400 tty2 #3:23:respawn:/sbin/getty 38400 tty3 #4:23:respawn:/sbin/getty 38400 tty4 #5:23:respawn:/sbin/getty 38400 tty5 #6:23:respawn:/sbin/getty 38400 tty6
The lxc container can now be created:
lxc-create -n debianbase -t debian-wheezy
I am using btrfs as my file system for holding these containers. One interesting feature of the integration between btrfs and lxc is that the lxc-create command will automatically create a btrfs sub-volume for the new container. These allows the use of snapshots for each container. Therefore, the one caveat of which to be aware is that when removing a container, the btrfs subvolume delete command is required. For example, 'rmdir -rf /var/lib/lxc/debianbase' will complain that rootfs is on another device. The command 'btrfs subvolume delete /var/lib/lxc/debianbase/rootfs' will need to be run to remove the subvolme before the remaining elements can be removed with the rm command.
In any case, the creation process will take a few moments to transfer, extract, and configure.
Now that the container has been created, there will be a config file located in /var/lib/lxc/debianbase. This file is used by the lxc-start commands. But since we are using OpenVSwitch, errors will occur when trying to start with the lxc-start commands. Instead, we will ignore this file and use libvirt/virtsh to start the container. However, some additional configuration is first required. An .xml configuration file is required to define the lxc container to be run.
There are a couple of examples located within the libvirt LXC container driver documentation. There is another sample described as libvirt LXC container w/ bridged networking (and 2GB RAM).
The key point is that libvirt will only work with OpenVSwitch in bridged mode. How to Use Open vSwitch with Libvir discusses this a bit. Be sure to have the bridge device configured in OpenVSWitch with:
ovs-vsctl add-br br0
The key line in the libvirt file is to have '<virtualport type='openvswitch'/>' in place. I placed my debianbase.xml file in the /var/lib/lxc directory, to keep everything together. LibVirt Network XML format goes into the fine points of the configuration.
One other consideration is that I am running the amd64 64 bit version. The lxc-create command changed the root partition to be /var/lib/lxc/debianbase/rootfs/rootfs-amd64 instead of /var/lib/lxc/debianbase/rootfs.
libvirt will take care of the interface creation and integration between OpenVSwitch and the lxc container. But a configuration change will be required in the container to obtain an appropriate ip address. To use dhcp for addressing, a default dhcp stanza has been added for eth0. If a static ip address is to be assigned instead, use the normal addressing stanzas associated with that file.
Also, so you don't confuse yourself, set a different hostname in the file at: /var/lib/lxc/debianbase/rootfs/rootfs-amd64/etc/hostname.
LibVirt, the server, also needs to be told about OpenVSwitch. I copied a template from elsewhere in libvirt to use:
cp -p /etc/libvirt/qemu/networks/default.xml /etc/libvirt/lxc/network.xml
Here is an example network.xml file.
The network has to be started in libvirt:
virsh connect lxc:/// net-define /etc/libvirt/lxc/network.xml net-start host-bridge net-autostart host-bridge
The container can now be started with libvirt:
virsh connect lxc:/// define /var/lib/lxc/debianbase/debianbase.xml start debianbase console debianbase
For some reason, which has been noted by a number of authors, is that ssh is not configured properly, probably becuase the keys need to be regenerated, which the default debootstrap doesn't deal with. So, the ssh server has to be re-installed. From the console, a few house-keeping adjustments. The locale and ssh reinstall could be performed from a chroot environment:
cd /var/lib/lxc chroot debianbase/rootfs/rootfs-amd64/ pwd .... exit
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # fix some perl locale issues: echo "en_US.UTF-8 UTF-8" > /etc/locale.gen apt-get -y --force-yes install locales dpkg-reconfigure locales apt-get install --reinstall openssh-server # add some additional tools apt-get --no-install-recommends install -y adduser apt-utils iputils-ping rsyslog logrotate apt-get install --no-install-recommends openssh-blacklist openssh-blacklist-extra apt-get install wget less lsof screen nano inetutils-ping psmisc sockstat #apt-get install man apt-get clean apt-get autoremove
The perl locale issues are identified and remedied with Creating a LXC virtual machine template (from scratch).
LXC HOWTO has some interesting ideas regarding LXC containter initialization.
The ssh reinstall was mentioned at Setting up LXC containers in 30 minutes (Debian Wheezy).
Coruscating Lucubrations talks about Incremental backups with btrfs
Margaret Bierman talks about How I Use the Advanced Capabilities of Btrfs.
Dobrica Pavlinusic's Weblog / Blog has an entry discussing the use of private mac addresses:
You will notice that I'm using prefix AC:DE:48 prefix for my mac addresses, which, to best of my knowledge range for private use and used all over IEEE docs (something like private IP range, but for mac addresses). I also have habit of naming interfaces with last octet of IP adress for internal ones, and last two for external one and same notation for mac addresses. Our brain is good at spotting patterns, and if you can read hex this seems just natural...
A ctrl-d or the command exit will exit and close the container. To disconnect from the container, instead use 'ctrl-a q', or if using screen, use 'ctrl-a a q'. 'ctrl-a ctrl-a' can be used to perform a regular 'ctrl-a' in the session (ie, move to beginning of a line). These commands apparently won't work in foreground mode, only in background mode.
Once in the container, and an ip address has been obtained, apt-get install iputils-ping can be used to get the ping utility installed.
On a moderately related note, I was wondering how to get by without doing the virsh .xml file first. The last part of Libvirt 1.0.5 with Openvswitch 1.11.90 provides the clue. Use the command virt-install like:
# virt-install --connect qemu:///system --name DSL2 --ram 1024 --vcpus 1 \ --disk path=/tmp/dsl2,size=1,bus=virtio,cache=none \ --network=ovs-br0 \ <======= THIS LINE IS CHANGED. --vnc --os-type=linux --cdrom /dev/sr0
In a follow on article, I have to deal with one level of added complexity, working with vlans with libvirt. The question was brought up with [ovs-discuss] tag vlan communication inter LXC. The solution supposedly exists at Open vSwitch Frequently Asked Questions.
However, according to LibVirt Network XML format, libvirt handles VLAN oriented configurations starting with libvirt version 0.10.0. Debian Wheezy uses version 0.9.12. The solution to this is supplied via remarks at using VLANs with Open vSwitch Fake Bridges. This is actually quite useful as I think that there can be multiple networkxxx.xml files, each supporting a vlan, and as the guest file references a particular network name, the appropriate vlan can be selected. I haven't tested it yet, but this should also allow multiple interfaces, each connected to a particular vlan.
Some older info on LXC on Debian Squeeze.
The application 'virt-top -c lxc:///' can be used to view cpu/network/memory statistics of each guest.
When in virsh, the command 'console Look in /sys/fs/cgroup/libvirt/lxc for cgroup related settings and monitoring variables and related settings. For info on some of those settings:
LXC on Debian Squeeze.
More on cgroups.
docker: an open source project to pack, ship and run any application as a lightweight container making use of LXC.