Virtualization

Alioth: A Machine Learning Based Interference-Aware Performance Monitor for Multi-Tenancy Applications in Public Cloud

Multi-tenancy in public clouds may lead to co-location interference on shared resources, which possibly results in performance degradation of cloud applications. Cloud providers want to know when such events happen and how serious the degradation is, to perform interference-aware migrations and alleviate the problem. However, virtual machines (VM) in Infrastructure-as-a-Service public clouds are black-boxes to providers, where application-level performance information cannot be acquired. This makes performance monitoring intensely challenging as cloud providers can only rely on low-level metrics such as CPU usage and hardware counters.

We propose a novel machine learning framework, Alioth, to monitor the performance degradation of cloud applications. To feed the data-hungry models, we first elaborate interference generators and conduct comprehensive co-location experiments on a testbed to build Alioth-dataset which reflects the complexity and dynamicity in real-world scenarios. Then we construct Alioth by (1) augmenting features via recovering low-level metrics under no interference using denoising auto-encoders, (2) devising a transfer learning model based on domain adaptation neural network to make models generalize on test cases unseen in offline training, and (3) developing a SHAP explainer to automate feature selection and enhance model interpretability. Experiments show that Alioth achieves an average mean absolute error of 5.29% offline and 10.8% when testing on applications unseen in the training stage, outperforming the baseline methods. Alioth is also robust in signaling quality-of-service violation under dynamicity. Finally, we demonstrate a possible application of Alioth's interpretability, providing insights to benefit the decision-making of cloud operators. The dataset and code of Alioth have been released on GitHub.

I've repurposed the notes at Install Home Assistant OS with KVM on Ubuntu headless (CLI only) for a Debian Bullseye (v11) system.

Start by checking that virtualization is available (the second command counts the number of instances, and needs to be non-zero):

# lscpu|egrep '(Model|Virtualization)'
Model:                           158
Model name:                      Intel(R) Xeon(R) E-2176G CPU @ 3.70GHz
Virtualization:                  VT-x
# egrep -c '(vmx|svm)' /proc/cpuinfo
24

The virtualization packages can be installed, networking is not included as it is already configured:

apt install -y \
  qemu-kvm \
  qemu-utils  \
  libvirt-daemon-system \
  libvirt-clients \
  ovmf

Optional packages include:

• numad - User-level daemon that monitors NUMA topology and usage
• pm-utils - Utilities and scripts for power management

Confirm a successful installation:

 systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-04-26 00:10:10 MDT; 5min ago

I will have to return to this note at a later time, I decided to give domoticz a try. I was reading a reddit posting about the regular upgrade issues HASS has. So I think something a bit leaner might be more my style. Albeit, it may require more hands on.

And as it turns out, with Domoticz, there is a bit more hand-holding, and more manual installation, but I think it fits my style better. It has forced a deep examinatiohn of how Domoticz, ESPHome, the Arduino IDE, and the various Arduino boards work together.

I built an lxc container to hold the VMware Horizon client. My main display is used for rendering. Refer to GUI From an LXC Container on the Host for how to build the container and how to obtain the COOKIE.

apt install libatk1.0-0 libgdk-pixbuf2.0-0 libgtk-3-0 libxss1
apt install apulse libpulse0 libv4l-0 libgstreamer1.0-0 libgstreamer-plugins-base1.0-0 libsane
apt install python3
apt install wget
apt install --no-install-recommends   xserver-xorg-core   xauth   x11-apps x11-utils
export XAUTHORITY=/root/.Xauthority
touch ${XAUTHORITY}
xauth add ${DISPLAY} MIT-MAGIC-COOKIE-1 <cookie>
echo $LANG
dpkg-reconfigure locales
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export NO_AT_BRIDGE=1
wget https://download3.vmware.com/software/view/viewclients/CART21FQ4/VMware-Horizon-Client-2012-8.1.0-17349998.x64.bundle
sh ./VMware-Horizon-Client-2012-8.1.0-17349998.x64.bundle  --console
vmware-view

References:

• Fixing invalid locale settings in Debian 8 - fix a "invalid or unknown locale provided" status message when starting VMware Horizon.
• Couldn't connect to accessibility bus: Failed to connect to socket - source of the NO_AT_BRIDGE setting -

  WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-uyN7DFK2Jm: Connection refused

In a follow on post to my previous GUI From an LXC Container on the Host entry, here are some notes on linking sound in an LXC container to the host's sound system.

The primary reason for this was to get around Interactive Broker's broken sound requirements for really old libav libraries. Which is back to the version x53 libraries from the Debian Jessie / Ubuntu Precise 12.04 days. Long story short, I don't have this working yet. The basic issue I have is that, when running TWS in an LXC container, all the configuration menu options but one work. When I click on the 'Sound Manager' menu item, TWS locks up, and I don't see any log messages anywhere as to why this might happen. So there is something special in an LXC container environment which is locking up sound in TWS.

But I did get sound from FireFox / Youtube to work.

This on a Debian Bullseye system.

The basic solution is, in the LXC container configuration file, to mount the Pulse Audio socket from the user's account of the host into the container and set an environment for the user in the container. In this case, the user has UID of 1000:

lxc.mount.entry = /run/user/1000/pulse/native tmp/pulse/native none rw,bind,create=file 0 0
lxc.environment = PULSE_SERVER=unix:/tmp/pulse/native

For completeness, these can be added (helps with ALSA):

lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir
lxc.cgroup.devices.allow = c 116:* rwm

That is about all to make it all work. Some or all of the following package installs in the container might help:

sudo apt install \
  pulseaudio \
  pulseaudio-utils \
  pavucontrol \
  gstreamer1.0-libav gstreamer1.0-plugins-bad gstreamer1.0-pulseaudio \
  mpg123

• Proxmox Ceph Server
• Manage Ceph Services on Proxmox VE Nodes
• Manage Ceph Services on Proxmox VE Nodes - reformatted copy of above link
• Proxmox VE Ceph Benchmark 2018/02 with a referral to Benchmark Proxmox VE Ceph Cluster Performance
• Install Ceph Server on Proxmox VE - video tutorial
• Ceph PGs per Pool Calculator

• As Ceph handles data object redundancy and multiple parallel writes to disks (OSDs) on its own, using a RAID controller normally doesn’t improve performance or availability. On the contrary, Ceph is designed to handle whole disks on it’s own, without any abstraction in between. RAID controller are not designed for the Ceph use case and may complicate things and sometimes even reduce performance, as their write and caching algorithms may interfere with the ones from Ceph.
• Ceph Object Storage Daemons are storing objects for Ceph over the network. It is recommended to use one OSD per physical disk.
• We recommend a Ceph cluster size, starting with 12 OSDs, distributed evenly among your, at least three nodes (4 OSDs on each node).

As a continuation of a previous article on using KVM/Qemu from the command line, here are some simple instructions for installing Microsoft Windows 10 in a KVM/Qemu Virtualization environment.

To start, a Windows 10 installation ISO will be required from, for example: Windows 10 ISO. Also, to improve the response of Windows 10 in a KVM/Qemu environment, and to be able to read the local drives, the latest binary virtio drivers are required.

# qemu-img create -f raw images/win10.img 25g
# qemu-system-x86_64 \
  -drive id=disk0,format=raw,file=images/win10.img,if=virtio  \
  -drive file=/var/local/downloads/Win10_1803_English_x64.iso,index=1,media=cdrom  \
  -drive file=/var/local/downloads/virtio-win-0.1.171.iso,index=2,media=cdrom \
  -enable-kvm \
  -m 4096 \
  -cpu host \
  -smp cores=2,threads=4 \
  -name win10 \
  -net nic,model=virtio \
  -net tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-win10-v90 \
  -rtc base=localtime,clock=host \
  -usb -device usb-tablet \
  -vga virtio \
  -no-reboot \
  -boot d 

The above is used for the initial install. When Windows wants to reboot, remove the last two lines. For this example, this is a continuation of the previous environment where it uses an Open vSwitch script to attach the tap device to the OVS bridge via a vlan.

Network drivers will need to be installed from the virtio cd in directory NetKVM/w10/amd64.

Other interesting links:

• virt-installs/win10uefi.sh - found via Any secrets to installing Win10 onto QEMU+KVM virtual machine ? on reddit
• Windows 10 Virtualization with KVM - mentions the installation of the network card driver
• Installing Windows 10 on KVM (with Fedora Core) uses a virt-install mechanism but mentions Spice and QXL drivers for the display
• High KVM/QEMU CPU utilization when Windows 10 guest is idle talks about some performance tools for optimizing cpu utilization. When using the GUI managers, some extra options are enabled, which I have not enabled in my example above.
• Running Windows 10 on Linux using KVM with VGA Passthrough which is a very interesting article about how to configure pass-through devices in order to supply a guest with direct access to certain hardware devices.

Some of the perf commands from before:

perf kvm --host top -p `pidof qemu-system-x86_64`
perf stat -e 'kvm:*' -a -- sleep 1
perf kvm --host stat live

Some notes on manual operation of KVM/Qemu with open vswitch.

/etc/qemu-ifup:

#!/bin/bash
# logger shows that $1 is only supplied
logger "values = $1 $2 $3 $4 $5"
# a default bridge is set
BRIDGE=ovsbr0
# port name is similar to:  tap-deb1-v90 [last digits are a vlan id]
VLAN=`echo $1 | awk '/tap-[A-Za-z0-9]+-v[0-9]+/ {split($0,A,"-"); print substr(A[3],2)}'`
logger "vlan = $VLAN"
ovs-vsctl --may-exist add-br $BRIDGE
if [[ "$VLAN" != "" ]]; then
  ovs-vsctl --may-exist add-port $BRIDGE $1 tag=$VLAN
else
  ovs-vsctl --may-exist add-port $BRIDGE $1
  fi
ip link set dev $1 up

/etc/qemu-ifdown:

#!/bin/bash
# bridge needs to match that of the ifup script
BRIDGE=ovsbr0
ovs-vsctl --if-exists del-port $BRIDGE $1

Create a raw disk image of 8GB. QCow has compression/encryption overhead which isn't necessary in some instances. QEMU/Images references other formats, and describes how to use the loop interface to mount a raw image to view it.

qemu-img create -f raw images/deb1.img 8g

Initial build from a cd and prevent a reboot:

qemu-system-x86_64 \
  -drive format=raw,file=images/deb1.img  \
  -cdrom /home/rpb/Downloads/debian-testing-amd64-netinst.iso  \
  -no-reboot \
  -boot d \
  -m 1024 \
  -cpu kvm64 \
  -name deb1 \
  -no-reboot \
  -enable-kvm \
  -net nic,model=virtio \
  -net tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-deb1-v90

Subsequent operations can be run without the -cdrom and -no-reboot operations. The ifname will be the interface name as show in 'ip link' and the last two digits are the vlan to be used by the qemu ifup/ifdown scripts.

For disk io optimization in a virtual machine, it is suggested by Use elevator=noop For Linux Virtual Machines and KVM convert qcow2 disk images to raw disk images for performance to use raw disk images and to set the scheduler to none/noop:

# cat /sys/block/sda/queue/scheduler
[mq-deadline] none
# echo none > /sys/block/sda/queue/scheduler
# cat /sys/block/sda/queue/scheduler
[none] mq-deadline

Putting 'elevator=none' on the kernel command line via '/etc/default/grub' doesn't seem to work for me. Perhaps there is a newer flavour available.

A suggested command to test in qcow2, raw and host mode (generating 1.1GB of data):

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

'-ctrl-grab' changes to right-control instead of Ctrl-Alt.

Image file examination commands:

# qemu-img info images/buckeye1.img
image: images/buckeye1.img
file format: raw
virtual size: 8.0G (8589934592 bytes)
disk size: 1.9G

# file images/buckeye1.img
images/buckeye1.img: DOS/MBR boot sector

# fdisk -lu images/buckeye1.img
Disk images/buckeye1.img: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6e9340d7

Device               Boot    Start      End  Sectors  Size Id Type
images/buckeye1.img1 *        2048 14680063 14678016    7G 83 Linux
images/buckeye1.img2      14682110 16775167  2093058 1022M  5 Extended
images/buckeye1.img5      14682112 16775167  2093056 1022M 82 Linux swap / Solaris

Standard Performance Evaluation Corporation: buried in there somewhere should be suggested configurations for guests and hosts.

Block layer statistics in /sys/block//stat: a note for future monitoring

For some machines with LXC containers, they have been running for a number of years. I want to take the easy way out and move the containers from one physical machine to another. At another time, I will rebuild the containers.

Since I am running BTRFS subvolumes for each container, I could be using BTRFS snapshot/send/receive commands to migrate/copy/replicate subvolumes. But before attempting that, I wanted to give the 'copy' a try. To do this properly, at the source, use the following -- with numeric-owner being a required paramenter -- command to collect the files:

tar --numeric-owner -czvf mycontainer.tar.gz /var/lib/lxc/my_container

At the destination, expand that file out:

tar --numeric-owner -xzvf mycontainer.tar.gz -C /var/lib/lxc/

The lxc users mailing list and Stack OverFlow were helpful.

Other stuff to do:

• Read up on CGroups in the Linux Programmer's Manual

In migrating from a very old version of LXC to a much newer version of LXC, I was getting errors. I needed to run a some debug to get a handle on errors:

lxc-start -n container -F --logpriority=DEBUG --logfile log

I had errors along the lines of:

Activating lvm and md swap...done.
Checking file systems...Segmentation fault (core dumped)
failed (code 139).

ServerFault had the solution: put "vsyscall=emulate" into /etc/default/grub, run 'update-grub' and reboot. Looks like I need to modernize my containers so I can eliminate this workaround, which may have some security considerations. There is a Debian Bug for this.

einstein home has a blog with some kernel references to the issue, in effect saying: "vsyscall is now disabled on latest linux distros". A lengthier LWN article at On vsyscalls and the vDSO. This works with kernel 4.14, my current version, but I see somewhere else that the workaround is entirely removed in kernel 4.15, at least in the Arch world. At bug 847154: "This breaks (e)glibc 2.13 and earlier".

Low End Box: a web site which presents hosting specials by many different VPS providers: VPS under $10 per month, physical servers under $50 per month. Over-subscription must work well for the casual user. The archives go back to 2008.

Free Shell Accounts are listed at Red Pill.

prgmr.com: LINUX & BSD VIRTUAL PRIVATE SERVERS

201808/10 - listing of providers with bgp services: bgp.services

2018/08/13 - more listings: Server Hosting Providers: list maintained by DataPlane.

2018/10/04 - edgeLinux - provides the ideal platform to run all the Virtual Servers you need, easily and efficiently. By leveraging both virtualization (KVM) and bare-metal containerization (LXC), the user has maximum flexibility on how to get the most out of any hardware.

2018/11/10 - from cryptography maillist: Arp Networks, Vexx Host.

2018/11/11 - from the same list: http://www.tektonic.net/ - smallish provider in Pennsylvania

2018/11/12 - more from the same list: https://www.jtan.com/ - They have a real shell and ssh access (not a cpanel, etc.) . They know how, and can use pgp (gpg) if you need it. They accept bitcoin. I actually paid them with e-gold a long time ago. They are very knowledgeable (some of them are open-source developers) . You may talk to a real person (but email preferred, of course) .

2019/06/15 Server Hunter - "If you are or were subscribed to a VPS or dedicated server, we encourage you to rate your server provider on serverhunter.com."

Debian Stretch based packages used with libvirt in various capacities:

• qemu-kvm - main package
• libvirt-daemon-system - runs the daemon, and unfortunately, installs iptables, where I currently use nftables
• numad - multi-processor tools
• bridge-utils - to be used with Free Range Routing in an EVPN capacity
• lxc - containers
• ctop - container statistics

Optional extras:
• python-libvirt - python library
• qemu-utils - some image commands
• virtinst
• virt-top
• qemu-guest-agent - for install in a guest
• virt-manager - graphical interface
• virt-viewer
• libvirt-dev - for custom c code
• libvirt-sanlock - custom c code for locking library
• libvirt-wireshark - troubleshooting wire format

For related activities:
• packer
• snapper
• libguestfs-tools
• libguestfs-rescue

With Sheepdog installed, and the cluster formatted, a libvirt/qemu/kvm usable block store needs to be created. This means creating a pool. Here is the content of pool.xml:

<pool type="sheepdog">
  <name>pool1</name>
  <source>
    <name>pool1</name>
    <host name='127.0.0.1' port='7000'/>
  </source>
</pool>

Only the 'name' attribute seems to be used. Not sure what 'source:name' is used for. This command is run on each of the three nodes (which creates a non-persistent pool):

# virsh pool-create pool.xml

Use the following instead to create a persistent, auto-starting pool:

# virsh pool-define pool.xml
# virsh pool-start pool1
# virsh pool-autostart pool1

Which results with the following:

# virsh pool-list
 Name                 State      Autostart
-------------------------------------------
 pool1                active     yes

The pool size doesn't seem to use the sheepdog pool size, but seems to be a red herring. When the volume gets associated, then things look correct.

# virsh pool-info pool1
Name:           pool1
UUID:           4bf5a447-39c5-491e-9d05-9f4c4b68ff16
State:          running
Persistent:     yes
Autostart:      yes
Capacity:       43.96 GiB
Allocation:     13.50 KiB
Available:      43.96 GiB

Debugging for virsh commands can be turned on with:

export LIBVIRT_DEBUG=1
export LIBVIRT_LOG_OUTPUTS="1:file:virsh.log"

To create a volume, here is an example parameter file (vol1.xml) (as in before, it seems only the name attribute name has any importance):

<volume>
  <name>vol1a</name>
  <key>sheep/vol1</key>
  <source>
  </source>
  <capacity unit='bytes'>1000000000</capacity>
  <allocation unit='bytes'>1000000000</allocation>
  <target>
    <path>sheepdog:vol1</path>
    <format type='unknown'/>
    <permissions>
      <mode>00</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</volume>

This then creates the volume, and is associated with the previously defined pool:

 virsh vol-create pool1c vol1.xml

Which then provides us with:

# virsh vol-info vol1
Name:           vol1
Type:           network
Capacity:       953.67 MiB
Allocation:     0.00 B

Some documents:
• virsh command reference
• virtualization administration guide

I havn't needed to do it till today, but I effectively have a bastion-host/jump-host in to which I make an ssh connection before connecting to other devices. Most, if almost all, sessions are text based, so, ... not a problem. But now I require a web session and a gui session to a number of virtualized guests, which don't have exterior access, on the second hop.

The cheat would be to make a network connection to the guests in question, but, well, that would be cheating, and would protections already in place.

Fortunately, on a related problem, I came across Scott's Weblog entry defining the solution for the same problem I was having, that of Accessing VNC Consoles of KVM Guests via SSH.

I have used port forwarding already, but only for single hops. The trick for multiple hops is to use the -g option [Allows remote hosts to connect to local forwarded ports] for the second hop. So stealing his syntax, here is the first hop:

ssh <:username>@<:remote host IP address or DNS name>  \
      -L <local port>:<remote FWD address>:<remote port>

Then on the second hop, a -g is added:

ssh <:username>@<:remote host IP address or DNS name> \
      -L <local port>:<remote FWD address>:<remote port> -g

is typically 127.0.0.1. However that can be changed to 'off-box' hops when trying to get through proxy box protections. , for vnc, is typically 5900 for display :0. Choose 5901 for display :1, etc.

Scott provided another informative command to show listening ports on a KVM host:

netstat -tunelp | grep LISTEN

Matt Oswalt has an article called ESXi vSwitch Load Balancing Woes where he deep dives into VMware's ESXi virtual switch and, based upon personal experience, discusses port channels, hash algorithm selection, and path selection. I need to refer back to this to see if networks I've seen conform to what he discusses.

• mounting a qemu image: short tutorial uses the nbd module, along with qemu-nbd to mound and view a qcow2 image
• QEMU/Images discusses image types, creating an image, converting images between popular formats, copying an image to physical device for use, when to use multiple images, how to share images, additional info on mounting, as well as obtaining information about the image.
• Virtualization Administration Guide: the Fedora guide, but generic enough to be used on many distributions. Impressive. And covers libvirt tools. With examples from virt-maanger (the gui).

Link to a Planet Virt Tools which is a collection of bloggers talking about Linux Virtualisation.

To go along with that is an amazing collection of Debian Build Tools, which includes things like debootstrap, qemubuilder, debian-installer, virt-inst, debootstick, Kameleon and a bunch of others.

Buiding Debian Images talks about various ways of building debian based images. The slide show mentions bootstarp-vz as the way many providers build their images.

I don't think I've landed here before: QEMU Emulator User Documentation. It references 2.10 Direct Linux Boot for fast Linux Kernel testing. The documentation is referenced from QEMU: processor emulator.

libVirt is a virtualization API.

Daniel P. Berrangé is a prolific writer on all things KVM / QEMU / Virtualization / libvirt.

While using Planet Virt Tools as a jumping off point for 'blogging about open source virtualization", I came across Virtio 1.0 and Windows Guests.

Last but not least, we have the KVM - Kernel Virtual Machine.