Raymond P. Burkholder - Things I Do

Alioth: A Machine Learning Based Interference-Aware Performance Monitor for Multi-Tenancy Applications in Public Cloud

Multi-tenancy in public clouds may lead to co-location interference on shared resources, which possibly results in performance degradation of cloud applications. Cloud providers want to know when such events happen and how serious the degradation is, to perform interference-aware migrations and alleviate the problem. However, virtual machines (VM) in Infrastructure-as-a-Service public clouds are black-boxes to providers, where application-level performance information cannot be acquired. This makes performance monitoring intensely challenging as cloud providers can only rely on low-level metrics such as CPU usage and hardware counters.

We propose a novel machine learning framework, Alioth, to monitor the performance degradation of cloud applications. To feed the data-hungry models, we first elaborate interference generators and conduct comprehensive co-location experiments on a testbed to build Alioth-dataset which reflects the complexity and dynamicity in real-world scenarios. Then we construct Alioth by (1) augmenting features via recovering low-level metrics under no interference using denoising auto-encoders, (2) devising a transfer learning model based on domain adaptation neural network to make models generalize on test cases unseen in offline training, and (3) developing a SHAP explainer to automate feature selection and enhance model interpretability. Experiments show that Alioth achieves an average mean absolute error of 5.29% offline and 10.8% when testing on applications unseen in the training stage, outperforming the baseline methods. Alioth is also robust in signaling quality-of-service violation under dynamicity. Finally, we demonstrate a possible application of Alioth's interpretability, providing insights to benefit the decision-making of cloud operators. The dataset and code of Alioth have been released on GitHub.

There was a note on reddit/r/debian which states that Wireguard is fully integrated into the Linux Kernel as of kernel v5.10. I suppose I could have saved a bunch of drama with upgrading to Bookworm which has kernel v6.1 natively, by instead using Bullseye-Backports, but I decided to go all the way. Hindsight is 20/20. A few other machines were already running Bookworm so I thought I had no problems.

It is nice to see that wireguard-tools references nftables. And there are a number of examples as reference for various scenarios.

So, with Wireguard in the kernel, no dkms installation is required. Just the installation of the tools (assumes root or sudo). Use the --no-install-recommends, otherwise your kernel will be replaced with a real-time kernel.

# apt install --not-install-recommends wireguard-tools
# cd /etc/wireguard

Create the keys for a peer to peer session:

# wg genkey | tee key_server_private | wg pubkey > key_server_public
# wg genkey | tee key_client_private | wg pubkey > key_client_public
# chmod -v 600 key*
# ls -al /etc/wireguard/
  total 20
  drwx------ 1 root root   54 Jul 18 04:30 .
  drwxr-xr-x 1 root root 2348 Jul 19 01:29 ..
  -rw------- 1 root root   45 Jul 19 02:49 key_client_private
  -rw------- 1 root root   45 Jul 19 02:49 key_client_public
  -rw------- 1 root root   45 Jul 19 02:49 key_server_private
  -rw------- 1 root root   45 Jul 19 02:49 key_server_public

A sample edge interface for server side termination of VPN (file name: wg0.conf):

[Interface]
Address = 10.20.10.1/24
#SaveConfig = true
ListenPort = 51820
PrivateKey = <server private key>

[Peer]
PublicKey = <client public key>
AllowedIPs = 10.20.10.0/24

A sample client interface, say on an Android for connection back to the server side (flie name: wg0-client.conf):

[Interface]
Address = 10.20.10.11/24
PrivateKey = <client private key>
DNS = 10.10.30.100

[Peer]
PublicKey = <server public key>
Endpoint = <server outside address>:51820
AllowedIPs = 10.20.10.0/24, 10.10.0.0/16
PersistentKeepalive = 21

If the allowed address is '0.0.0.0/0', then all traffic goes through the VPN. Use ipv6-test.com or ipleak.net to verify that traffic is going trough the VPN, or use something like WhatIsMyIpAddress.

Impressively, someone has created a QR generator which will generate a code to the terminal window (not a graphic file, but an ansii thingy in a terminal window). This can then be scanned by Android WireGuard to load the configuration.

$ qrencode -t ansiutf8 < wg-android.conf

I use a saltstack script to build a zone based firewall composed of nftable rules. Basically two rules are needed: a) burn a port through the firewall, and b) allow access to the interior network sections for one or all ports.

To turn on the interface and start it automatically:

# chmod -v 600 /etc/wireguard/wg0.conf
# wg-quick up wg0
# systemctl enable wg-quick@wg0.service

To turn off the interface and keep it off:

# wg-quick down wg0
# systemctl disable wg-quick@wg0.service

To show connections and status:

# wg show
interface: wg0
  public key: <server public key>
  private key: (hidden)
  listening port: 51820

peer: 
  endpoint: :4496
  allowed ips: 10.20.10.0/24
  latest handshake: 44 minutes, 26 seconds ago
  transfer: 2.50 MiB received, 33.47 MiB sent

With the SaveConfig enabled, more clients can be added and saved:

# wg genkey | tee key_mac_private | wg pubkey > key_mac_public 
# wg set wg0 peer <mac public key> allowed-ips 10.20.10.12/32

Stan's Blog mentioned terminating the server side VPN on UDP port 53. Many/Most networks allow this out, so would/could be a way out of a heavily protected network to the destination.

Note: this config was added to a privileged lxc container, nothing special was required for building the wireguard interface.

I found out the hard way that SaltStack and Debian no longer place nice together. I had upgraded a Debian installation from Bullseye to Bookworm, along with the resident Salt Minion. When attempting to use the minion, it no longer starts up, due to various imports no longer working. Which was due to the salt-minion not being upgraded. The error message would started this odyssey:

salt ImportError: cannot import name 'Markup' from 'jinja2'

Taking a look at the Debian Developer Information for Salt, the last version started in 'unstable' was 3004.1 back in December of 2022. This is now almost 8 months later and little or no movement. There was some mention in a ticket somewhere that Salt release cycles don't cater to Debian stable release cycles. Not sure if that is a legitimate reason or not, but, well, for whatever reason, SaltStack management in Debian is no longer a simple no brainer.

However, after a little digging, there is a way to run SaltStack versions 3006 (current as of this writing). It is simple to install on Bullseye, but not easily done on Bookworm.

On Bullseye (as root, or implies sudo):

# cd ~ # apt remove salt-minion salt-master # apt install curl # curl -L https://bootstrap.saltstack.com -o install_salt.sh # sh install_salt.sh -M onedir

The '-M' installs the salt master at the same time (for machines running master). If you forget to do that, you'll need to diagnose and fix the systemctl mask error with the following:

# apt install file # file /etc/systemd/system/salt-master.service # rm /etc/systemd/system/salt-master.service # systemctl daemon-reload # sh install_salt.sh -M onedir

The 'sh install_salt.sh -M onedir' should show a symlink to /dev/nul, which the 'rm ...' will fix.

On Bookworm, the bootstrap isn't scheduled to work till beginning of 2024 sometime I think with Salt 3007 or 3008 -- more info in [FEATURE REQUEST] Add Salt support for Debian 12 #64223 .

In the meantime, I had to cheat a bit:

• in /etc/debian_version, change 12.0 to 11.0
• in /etc/apt/sources.list, change bookworm to bullseye
• rm /etc/apt/sources.list.d/salt.list
• run apt update
• run the commands listed above for installing the one or both the salt services
• restore /etc/debian_version and /etc/apt/sources.list to their original content

I'm sure there are more elegant ways of doing this, but this worked to fake the needed version 11 in the installation script and directory traversal requirements

Note, more info on the Salt Install/Bootstrap Process.

Market Making with Deep Reinforcement Learning from Limit Order Books

Market making (MM) is an important research topic in quantitative finance, the agent needs to continuously optimize ask and bid quotes to provide liquidity and make profits. The limit order book (LOB) contains information on all active limit orders, which is an essential basis for decision-making. The modeling of evolving, high-dimensional and low signal-to-noise ratio LOB data is a critical challenge. Traditional MM strategy relied on strong assumptions such as price process, order arrival process, etc. Previous reinforcement learning (RL) works handcrafted market features, which is insufficient to represent the market. This paper proposes a RL agent for market making with LOB data. We leverage a neural network with convolutional filters and attention mechanism (Attn-LOB) for feature extraction from LOB. We design a new continuous action space and a hybrid reward function for the MM task. Finally, we conduct comprehensive experiments on latency and interpretability, showing that our agent has good applicability.

• make sure the MTA's forward and reverse DNS match
• set up an SPF record, probably "v=spf1 mx ~all"
• manage DNS MX, DKIM, DMARC and SPF records (SPF, DKIM, and DMARC are a recommended order of implementation)
• set up DKIM signing for each domain you host, make the DKIM domain match the From: domain
• start slow and look at any bounces
• maybe collect DMARC stats but for a small volume MTA, not very interesting
• instead of struggling with Postfix, OpenDKIM, Dovecot and friends (and losing out on quite a few features), try looking at maddy
• arrange for a backup MTA
• manage reverse lookup records, including managing the uncertain chain of authority between the instance and the nearest SOA
• manage certificates associated with TLS for SMTP and IMAP
• manage DKIM certificate
• manage one's upstream to address PBL issues
• keep the MTA secure and free from DOS attack
• Use a static IPv4 address for your email system [counter point: If DNS is fully functional and IP addresses don't change too quickly and TTL is configured properly on DNS records, ... then why is a static IP address strictly required? ]
• Do not share this IPv4 address with user machines
• Do not host your email system 'in the cloud'
• Make sure that your IP address is not listed in the PBL
• Provide an MX record
• Provide meaningful and consistent reverse DNS
• Your system should say HELO (or EHLO) with its hostname
• Keep your software completely up-to-date
• Ensure that only authorised users can send email through your system.
• Limit outgoing email volumes
• Accept reports of problems with your systems
• Review the mail system logs on a regular basis
• Be reliable (viz have at least 4 9s availability)
• Don't be an open relay
• Don't create backscatter
• Maintain a good reputation
• users can probably have acceptable success with all but the large email providers if they simply have their MTA hello as the name that the ISP assigns to the connection presuming that the ISP has forward and reverse DNS configured therefor.

Debian with Automated Snapper Rollbacks is a short tutorial about setting up a Debian linux system with automated BTRFS snapshots of the system and easy rollback to previous auto-generated snapshots. Once it's setup, it'll automatically take pre/post snapshots when you run `apt` and you can boot them from grub.

There is a company called the 'Wellness Company'. They suggest the following natural ingredients, rather than using Melatonin for assistance with sleep:

• Passion Flower to relax and promote an easy start to your sleep routine.
• Rafuma Leaf stabilizes the mind and prepares it for restful sleep.
• Kava mitigates the impacts of stress as you wind down.
• Chamomile for its time-tested properties of providing a gentle calming effect on the brain and body.
• Valerian Root to help reduce anxiety.
• Ashwagandha Root to help regulate the negative impacts of daily stress on your immune system.

They are saying that the hormone-based supplement Melatonin has risks such as: sleep-wake cycle disruption, hormonal effects, drug interactions, dependence & tolerance, as well as purity concerns.

Intelligent Traffic Control with Smart Speed Bumps

Traffic congestion and safety continue to pose significant challenges in urban environments. In this paper, we introduce the Smart Speed Bump (SSBump), a novel traffic calming solution that leverages the Internet of Things (IoT) and innovative non-Newtonian fluid materials to enhance road safety, optimize emergency response times, and improve the overall driving experience. The SSBump uses IoT sensors to detect and communicate with emergency vehicles, reducing response times by temporarily deflating. These sensors also analyze traffic patterns and inform data-driven decisions. Additionally, the SSBump uses an Oobleck mixture that adapts its behavior based on the velocity of approaching vehicles, resulting in a safer and more comfortable experience for drivers. This study commences with an overview of the prevalent traffic congestion, followed by a discussion on various available options in this domain. Subsequently, the paper explores the advantages of smart speed bumps and their operational mechanisms. Finally, it presents a comprehensive analysis of the results, its challenges, and the prospects of the work. The findings of this research demonstrate the potential of the SSBump system to revolutionize traffic control, emergency response time, and the driving experience in smart cities, making it a game-changing innovation for advanced transportation systems.

Graph Neural Network based Log Anomaly Detection and Explanation

Event logs are widely used to record the status of high-tech systems, making log anomaly detection important for monitoring those systems. Most existing log anomaly detection methods take a log event count matrix or log event sequences as input, exploiting quantitative and/or sequential relationships between log events to detect anomalies. Unfortunately, only considering quantitative or sequential relationships may result in many false positives and/or false negatives. To alleviate this problem, we propose a graph-based method for unsupervised log anomaly detection, dubbed Logs2Graphs, which first converts event logs into attributed, directed, and weighted graphs, and then leverages graph neural networks to perform graph-level anomaly detection. Specifically, we introduce One-Class Digraph Inception Convolutional Networks, abbreviated as OCDiGCN, a novel graph neural network model for detecting graph-level anomalies in a collection of attributed, directed, and weighted graphs. By coupling the graph representation and anomaly detection steps, OCDiGCN can learn a representation that is especially suited for anomaly detection, resulting in a high detection accuracy. Importantly, for each identified anomaly, we additionally provide a small subset of nodes that play a crucial role in OCDiGCN's prediction as explanations, which can offer valuable cues for subsequent root cause diagnosis. Experiments on five benchmark datasets show that Logs2Graphs performs at least on par state-of-the-art log anomaly detection methods on simple datasets while largely outperforming state-of-the-art log anomaly detection methods on complicated datasets.

Effects of Explanation Specificity on Passengers in Autonomous Driving

The nature of explanations provided by an explainable AI algorithm has been a topic of interest in the explainable AI and human-computer interaction community. In this paper, we investigate the effects of natural language explanations' specificity on passengers in autonomous driving. We extended an existing data-driven tree-based explainer algorithm by adding a rule-based option for explanation generation. We generated auditory natural language explanations with different levels of specificity (abstract and specific) and tested these explanations in a within-subject user study (N=39) using an immersive physical driving simulation setup. Our results showed that both abstract and specific explanations had similar positive effects on passengers' perceived safety and the feeling of anxiety. However, the specific explanations influenced the desire of passengers to takeover driving control from the autonomous vehicle (AV), while the abstract explanations did not. We conclude that natural language auditory explanations are useful for passengers in autonomous driving, and their specificity levels could influence how much in-vehicle participants would wish to be in control of the driving activity.

Machine learning for potion development at Hogwarts

Objective: To determine whether machine learning methods can generate useful potion recipes for research and teaching at Hogwarts School of Witchcraft and Wizardry. Design: Using deep neural networks to classify generated recipes into a standard drug classification system. Setting: Hogwarts School of Witchcraft and Wizardry. Data sources: 72 potion recipes from the Hogwarts curriculum, extracted from the Harry Potter Wiki. Results: Most generated recipes fall into the categories of psychoanaleptics and dermatologicals. The number of recipes predicted for each category reflected the number of training recipes. Predicted probabilities were often above 90% but some recipes were classified into 2 or more categories with similar probabilities which complicates anticipating the predicted effects. Conclusions: Machine learning powered methods are able to generate potentially useful potion recipes for teaching and research at Hogwarts. This corresponds to similar efforts in the non-magical world where such methods have been applied to identify potentially effective drug combinations.

I had an application running which had an open file it was actively using.

I accidentally performed an rm (remove) on the file rather than another I actually meant to remove.

Due to Linux's method of linking files, even though the directory entry link to the file was removed, the file is still open and has an additional link via the application process pseudo directory.

The process sub-directory is composed of the process id. The process id can be found with something like:

$ pidof BasketTrading
2663042

The deleted file can then be found with:

$ lsof -p 2663042 | grep deleted
BasketTra 2663042  rpb   18u      REG               0,46   2739200 29438986 /home/.../BasketTrading.db (deleted)

The 18u reflects the file descriptor used for the file. This can be used to perform a simple copy of the file to an alternate location. A link does not seem to fix it. You may want to complete any outstanding writes to the file first. But do not close it or the application. If you do, the file will be unrecoverable.

$ ls /proc/2663042/fd/18
/proc/2663042/fd/18

$ cp /proc/2663042/fd/18 /home/.../BasketTrading.db.rescue