Multi-tenancy in public clouds may lead to co-location interference on shared resources, which possibly results in performance degradation of cloud applications. Cloud providers want to know when such events happen and how serious the degradation is, to perform interference-aware migrations and alleviate the problem. However, virtual machines (VM) in Infrastructure-as-a-Service public clouds are black-boxes to providers, where application-level performance information cannot be acquired. This makes performance monitoring intensely challenging as cloud providers can only rely on low-level metrics such as CPU usage and hardware counters.
We propose a novel machine learning framework, Alioth, to monitor the performance degradation of cloud applications. To feed the data-hungry models, we first elaborate interference generators and conduct comprehensive co-location experiments on a testbed to build Alioth-dataset which reflects the complexity and dynamicity in real-world scenarios. Then we construct Alioth by (1) augmenting features via recovering low-level metrics under no interference using denoising auto-encoders, (2) devising a transfer learning model based on domain adaptation neural network to make models generalize on test cases unseen in offline training, and (3) developing a SHAP explainer to automate feature selection and enhance model interpretability. Experiments show that Alioth achieves an average mean absolute error of 5.29% offline and 10.8% when testing on applications unseen in the training stage, outperforming the baseline methods. Alioth is also robust in signaling quality-of-service violation under dynamicity. Finally, we demonstrate a possible application of Alioth's interpretability, providing insights to benefit the decision-making of cloud operators. The dataset and code of Alioth have been released on GitHub.
Wednesday, July 19. 2023
Alioth: A Machine Learning Based Interference-Aware Performance Monitor for Multi-Tenancy Applications in Public Cloud
Wireguard in a Debian LXC Container
There was a note on reddit/r/debian which states that Wireguard is fully integrated into the Linux Kernel as of kernel v5.10. I suppose I could have saved a bunch of drama with upgrading to Bookworm which has kernel v6.1 natively, by instead using Bullseye-Backports, but I decided to go all the way. Hindsight is 20/20. A few other machines were already running Bookworm so I thought I had no problems.
It is nice to see that wireguard-tools references nftables. And there are a number of examples as reference for various scenarios.
So, with Wireguard in the kernel, no dkms installation is required. Just the installation of the tools (assumes root or sudo). Use the --no-install-recommends, otherwise your kernel will be replaced with a real-time kernel.
# apt install --not-install-recommends wireguard-tools # cd /etc/wireguard
Create the keys for a peer to peer session:
# wg genkey | tee key_server_private | wg pubkey > key_server_public # wg genkey | tee key_client_private | wg pubkey > key_client_public # chmod -v 600 key* # ls -al /etc/wireguard/ total 20 drwx------ 1 root root 54 Jul 18 04:30 . drwxr-xr-x 1 root root 2348 Jul 19 01:29 .. -rw------- 1 root root 45 Jul 19 02:49 key_client_private -rw------- 1 root root 45 Jul 19 02:49 key_client_public -rw------- 1 root root 45 Jul 19 02:49 key_server_private -rw------- 1 root root 45 Jul 19 02:49 key_server_public
A sample edge interface for server side termination of VPN (file name: wg0.conf):
[Interface] Address = 10.20.10.1/24 #SaveConfig = true ListenPort = 51820 PrivateKey = <server private key> [Peer] PublicKey = <client public key> AllowedIPs = 10.20.10.0/24
A sample client interface, say on an Android for connection back to the server side (flie name: wg0-client.conf):
[Interface] Address = 10.20.10.11/24 PrivateKey = <client private key> DNS = 10.10.30.100 [Peer] PublicKey = <server public key> Endpoint = <server outside address>:51820 AllowedIPs = 10.20.10.0/24, 10.10.0.0/16 PersistentKeepalive = 21
If the allowed address is '0.0.0.0/0', then all traffic goes through the VPN. Use ipv6-test.com or ipleak.net to verify that traffic is going trough the VPN, or use something like WhatIsMyIpAddress.
Impressively, someone has created a QR generator which will generate a code to the terminal window (not a graphic file, but an ansii thingy in a terminal window). This can then be scanned by Android WireGuard to load the configuration.
$ qrencode -t ansiutf8 < wg-android.conf
I use a saltstack script to build a zone based firewall composed of nftable rules. Basically two rules are needed: a) burn a port through the firewall, and b) allow access to the interior network sections for one or all ports.
To turn on the interface and start it automatically:
# chmod -v 600 /etc/wireguard/wg0.conf # wg-quick up wg0 # systemctl enable wg-quick@wg0.service
To turn off the interface and keep it off:
# wg-quick down wg0 # systemctl disable wg-quick@wg0.service
To show connections and status:
# wg show interface: wg0 public key: <server public key> private key: (hidden) listening port: 51820 peer:endpoint: :4496 allowed ips: 10.20.10.0/24 latest handshake: 44 minutes, 26 seconds ago transfer: 2.50 MiB received, 33.47 MiB sent
With the SaveConfig enabled, more clients can be added and saved:
# wg genkey | tee key_mac_private | wg pubkey > key_mac_public # wg set wg0 peer <mac public key> allowed-ips 10.20.10.12/32
Stan's Blog mentioned terminating the server side VPN on UDP port 53. Many/Most networks allow this out, so would/could be a way out of a heavily protected network to the destination.
Note: this config was added to a privileged lxc container, nothing special was required for building the wireguard interface.
SaltStack on Debian Bookworm
I found out the hard way that SaltStack and Debian no longer place nice together. I had upgraded a Debian installation from Bullseye to Bookworm, along with the resident Salt Minion. When attempting to use the minion, it no longer starts up, due to various imports no longer working. Which was due to the salt-minion not being upgraded. The error message would started this odyssey:
salt ImportError: cannot import name 'Markup' from 'jinja2'
Taking a look at the Debian Developer Information for Salt, the last version started in 'unstable' was 3004.1 back in December of 2022. This is now almost 8 months later and little or no movement. There was some mention in a ticket somewhere that Salt release cycles don't cater to Debian stable release cycles. Not sure if that is a legitimate reason or not, but, well, for whatever reason, SaltStack management in Debian is no longer a simple no brainer.
However, after a little digging, there is a way to run SaltStack versions 3006 (current as of this writing). It is simple to install on Bullseye, but not easily done on Bookworm.
On Bullseye (as root, or implies sudo):
# cd ~ # apt remove salt-minion salt-master # apt install curl # curl -L https://bootstrap.saltstack.com -o install_salt.sh # sh install_salt.sh -M onedir
The '-M' installs the salt master at the same time (for machines running master). If you forget to do that, you'll need to diagnose and fix the systemctl mask error with the following:
# apt install file # file /etc/systemd/system/salt-master.service # rm /etc/systemd/system/salt-master.service # systemctl daemon-reload # sh install_salt.sh -M onedir
The 'sh install_salt.sh -M onedir' should show a symlink to /dev/nul, which the 'rm ...' will fix.
On Bookworm, the bootstrap isn't scheduled to work till beginning of 2024 sometime I think with Salt 3007 or 3008 -- more info in [FEATURE REQUEST] Add Salt support for Debian 12 #64223 .
In the meantime, I had to cheat a bit:
- in /etc/debian_version, change 12.0 to 11.0
- in /etc/apt/sources.list, change bookworm to bullseye
- rm /etc/apt/sources.list.d/salt.list
- run apt update
- run the commands listed above for installing the one or both the salt services
- restore /etc/debian_version and /etc/apt/sources.list to their original content
I'm sure there are more elegant ways of doing this, but this worked to fake the needed version 11 in the installation script and directory traversal requirements
Note, more info on the Salt Install/Bootstrap Process.