Raymond P. Burkholder - Things I Do

The difference between reactive and proactive monitoring comes down to tracking the right network performance metrics. Important metrics that consistently predict issues before they impact users include:

• Round-trip time variations: Subtle increases in RTT often precede more serious network congestion. Set baseline thresholds for different times of day and get alerted when patterns deviate
• Buffer utilization patterns: Most admins ignore this until it's too late. Monitoring buffer usage across key network devices reveals impending bottlenecks before they trigger packet loss
• Error rates on interfaces: Even small increases in CRC errors or interface discards can signal hardware issues days before actual failures occur
• TCP retransmission rates: This metric reveals application performance issues that users might experience from data transmission rates before they report them
• Network traffic symmetry: Unusual asymmetric traffic patterns often indicate security issues or misconfigured applications

Reference: Stop Reactive Network Troubleshooting: Monitor These 5 Metrics to Prevent Downtime

Cloud environments introduce unique challenges for network performance monitoring. Some standard metrics like latency and packet loss remain important, but you'll also want to focus on metrics specific to your cloud connectivity. For hybrid environments, these are good metrics:

• Inter-region latency: Especially important for globally distributed applications
• Connection establishment time: Often overlooked but crucial for microservice architectures
• Throughput consistency: More important than raw bandwidth in many cloud scenarios
• DNS resolution time: Can be a hidden bottleneck in cloud environments

It has been a while since I last setup NordVPN on a Debian Linux using StrongSwan. StrongSwan is now using 'native' files rather than the now deprecated ipsec files. NordVPN Example: How to connect to NordVPN with IKEv2/IPSec on Linux refers to the old format. Here is a new format.

Here is my take on a successful installation.

apt install \
  --no-install-recommends \
    strongswan \
    libstrongswan-standard-plugins \
    libstrongswan-extra-plugins \
    libcharon-extra-plugins
wget https://downloads.nordcdn.com/certificates/root.pem -O /etc/swanctl/x509ca/NordVPN.pem
sed -i 's/load = yes/load = no/' /etc/strongswan.d/charon/constraints.conf

An example /etc/swanctl/swanctl.conf file:

connections {
  nordvpn {
    version = 2
    proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-ecp521,aes192-sha256-modp3072,default
    rekey_time = 0s
    fragmentation = yes
    dpd_delay = 300s
    local_addrs = %defaultroute
    remote_addrs = 
    vips=0.0.0.0,::
    local {
      auth = eap-mschapv2
      eap_id = "<username>"
    }
    remote {
      auth = pubkey
      cacerts = /etc/swanctl/x509ca/NordVPN.pem
      id = %any
    }
    children {
      nordvpn {
        remote_ts = 0.0.0.0/0,::/0
        rekey_time = 0s
        dpd_action = clear
        esp_proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-modp3072,aes192-sha256-ecp256-modp3072,default
      }
    }
  }
}

secrets {
  eap-nordvpn {
    id = "<username>"
    secret = "<password>"
  }
}

If you have a local network to which you need access when the vpn is up, StrongSwan using route table 220 for forwarding. Use the following command to see current settings:

# ip rule list
0:      from all lookup local
220:    from all lookup 220
32766:  from all lookup main
32767:  from all lookup default

# ip route list table 220
default via 192.168.1.10 dev eth0 proto static src 

To add your local network to the route table. Additional subnets are added in a similar way. Change the interface name to suit your local circumstances. Use the updown Plugin for better control of the local routing.

ip route add table 192.168.1.0/24 dev wlan0

This may be required for changes made:

# systemctl restart strongswan

Tunnel related state and status commands:

sudo swanctl --load-conns
sudo swanctl --list-conns
sudo swanctl --list-certs
sudo swanctl --list-sas
sudo swanctl --initiate --child nordvpn
sudo swanctl --terminate --child nordvpn
sudo swanctl --reload-settings

References:

• How to Setup an IKEv2 VPN Connection on Arch Linux (Example: NordVPN) - primary configuration template
• Allow Strongswan roadwarrior to access local LAN - keeping local traffic out of the vpn

In Proxmox, one can visually connect to an active virtual machine through one of two ways:

• noVNC - default mechanism - works with most default installations
• SPICE - enhanced mechanism - requires virtual machine support

With Windows based Virtual Machines, the SPICE client can be installed via the Redhat supplied VirtIO .iso.

If running Firefox from a Debian Linux machine, run 'sudo apt install virt-viewer' to install the viewer executable.

As a one time configuration in Firefox:

• go into about:config, and add the key 'network.protocol-handler.expose.virt-viewer' as boolean and set to true
• go into about:preferences, and set "What should Firefox do with other files" to "Ask whether to open or save files".
• in Proxmox, open a SPICE console for a virtual machine, which downloads a customized .vv file,
• Firefox will then request to open a Virt-Viewer file with Remove Viewer - at this point, you can set it as the default viewer, and it will show up in the application preferences

When installing a Microsoft Windows 11 Operating System in Proxmox, a few helpful hints here. This is with Proxmox v9.

Windows 11 can be downloaded from Microsoft Windows 11 Download. During the installation, you'll need a license key, so have that ready.

The RedHat Virtio drivers will be required during initial install.

When defining the configuration for the virtual machine in proxmox, attach the Win11 ISO, attach EUFI local, and provide a TPM area.

When starting from the ISO, hit a key to start. During the process of selecting and formatting the drive, you'll need to browse to the Virtio ISO, and browse into the \amd64\win11 sub-directory to load the initial storage sub-system drivers.

Later on in the process, the installation will request access to the network. Use shift-F10 to start a text console. In there, type:

oobe\bypassnro

The machine will reboot. This can be used to bypass making a Microsoft account and to create a local account instead.

Optionally:

• Once it returns, you need to disconnect networking. So you can do terminal again and ipconfig /release, or just set the NIC for disconnected.
• At some point it'll guilt you into creating an account but there will be an "I don't have internet" option, and that's your way out.

Inspiration from How to install Win11 in Proxmox | Quick guide | And fix problems of network search.

From Install Windows 11 22H2 , some other tricks:

• in the user box type "no@thankyou.com" and any password. It'll error out when trying to login and let you click next to create a local account. Dev bypass must be
• When connecting to the network, Press Win key + E to bring up an explorer window, install all the drivers needed off the driver ISO
• Windows 10 guest best practices
• Choose setup as work account + choose domain setup. The account it first creates is a local account. (I did this as of last week)
• oobe = out-of-box-experience

From Hacker News - %CPU utilization is a lie referencing Brendan Long's Blog

"a server at 60% utilization had zero room left" can be explained with queuing theory:

• Up to a hair over 60% utilization the queuing delays on any work queue remain essentially negligible. At 70 they become noticeable, and at 80% they've doubled. And then it just turns into a shitshow from there on.
• rule of thumb is 60% is zero, and 80% is the inflection point where delays go exponential.
• CPU utilization metrics are frequently averaged over a long period of time (like a minute), but if your SLO is 100 ms, what you care about is whether there's any ~100 ms period where CPU utilization is at 100%. Measuring p99 (or even p100) CPU utilization can make this a lot more visible.

Lifted from Installing Virtio Drivers for Windows on Proxmox to Enable Paravirtualization / Fix Incorrect Memory Monitoring, there is a section at the end which discusses settings for disk caching modes in Proxmox. The terms relate to how frequently data is flushed to the underlying storage system. There are trade-offs for performance and durability.

• Direct sync - highest data security but has poor performance - ensures that every time data is written to memory, it is immediately synchronized to the disk. This means each write operation introduces disk latency because the system must wait for the disk to confirm the write is complete before continuing with other operations. Although this method provides the highest data security, frequent disk operations lead to lower performance.
• Write through - balance between data consistency and performance - writing data to memory and immediately writes it to disk without waiting for disk confirmation. It offers good data consistency and high performance by reducing the time spent waiting for disk confirmation. However, disk write operations still occur, leading to some performance overhead.
• Write back - improves performance but comes with certain data risks - first writes data to the memory cache and writes it back to the disk at an appropriate time. This reduces the number of disk accesses and improves system performance. However, because data is cached in memory, there is a risk of data loss if the system crashes or experiences a power outage before writing back to the disk.
• Write back (unsafe) - highest performance (fastest among all modes) but compromises data security and consistency - similar to the regular write back method, but it makes no guarantees about data consistency. This means that in the event of a system failure or power outage, data not yet written back to the disk could be permanently lost. While this method provides higher performance, it sacrifices data security and consistency. RamFS would be a use case here.

Perform your own testing given your own physical infrastructure and weigh the pro's and con's.

Starting with version VE 8.4, proxmox host directories can be shared into a windows or linux virtual machine. I have a version 9 running here as an example:

# pveversion
pve-manager/9.0.6/49c767b70aeb6648 (running kernel: 6.14.8-2-pve)

Check to see that the appropriate package is installed, it should be by default:

# dpkg -l | grep virtiofsd
ii  virtiofsd         1.13.2-1      amd64     Virtio-fs vhost-user device daemon

Create a directory for sharing, use something like /mnt or /var/local, depending upon your preference:

# mkdir /var/local/shared

In the proxmox management site, navigate Datacenter -> Directory Mappings, then add a mapping with the directory just created. The name supplied will be the name seen in the virtual machine.

Also in the proxmox management site, navigate to the virtual machine settings -> Hardware and add a Virtiofs Filesystem Passthrough. Select the directory id you just created. Select the 'Direct IO' box (may be optional). Then restart the machine if already started to have the setting take effect.

In a Windows virtual machine, be sure you've installed the Windows VirtIO Drivers.

Open the Services cmdlet (services.msc). There should be a 'VirtIO-FS Service". Startup type can be set to automatic. If only temporary use is required, simply start the service. A new device should become visible in This PC in file explorer.

For Windows based virtual machines, WinFSP is required. It is mentioned at How to install virtiofs drivers on Windows.

Once installed, open services.msc, and start the VirtIO-FS Service

In explorer, you should see the shared data folder in 'This PC'.

Risk Manager for Intrusion Tolerant Systems: Enhancing HAL 9000 with New Scoring and Data Sources

Intrusion Tolerant Systems (ITSs) have become increasingly critical due to the rise of multi-domain adversaries exploiting diverse attack surfaces. ITS architectures aim to tolerate intrusions, ensuring system compromise is prevented or mitigated even with adversary presence. Existing ITS solutions often employ Risk Managers leveraging public security intelligence to adjust system defenses dynamically against emerging threats. However, these approaches rely heavily on databases like NVD and ExploitDB, which require manual analysis for newly discovered vulnerabilities. This dependency limits the system's responsiveness to rapidly evolving threats. HAL 9000, an ITS Risk Manager introduced in our prior work, addressed these challenges through machine learning. By analyzing descriptions of known vulnerabilities, HAL 9000 predicts and assesses new vulnerabilities automatically. To calculate the risk of a system, it also incorporates the Exploitability Probability Scoring system to estimate the likelihood of exploitation within 30 days, enhancing proactive defense capabilities.

Despite its success, HAL 9000's reliance on NVD and ExploitDB knowledge is a limitation, considering the availability of other sources of information. This extended work introduces a custom-built scraper that continuously mines diverse threat sources, including security advisories, research forums, and real-time exploit proofs-of-concept. This significantly expands HAL 9000's intelligence base, enabling earlier detection and assessment of unverified vulnerabilities. Our evaluation demonstrates that integrating scraper-derived intelligence with HAL 9000's risk management framework substantially improves its ability to address emerging threats. This paper details the scraper's integration into the architecture, its role in providing additional information on new threats, and the effects on HAL 9000's management.

This is an interesting take on security: toleration of intruders to systems. I suppose this is somewhat akin to limiting blast radius. One interesting nugget:

... system applies a proactive recovery mechanism to protect the system from undetected attacks. A central controller periodically rotates the state of each VM, transitioning between active, cleansing, ready, and active states.

This would be the pre-emption step of what file checksum utilities would do. However, this doesn't eliminate the types of intrusions where intruders make use of elevated account access.

AI agentic programming is an emerging paradigm in which large language models (LLMs) autonomously plan, execute, and interact with external tools like compilers, debuggers, and version control systems to iteratively perform complex software development tasks. Unlike conventional code generation tools, agentic systems are capable of decomposing high-level goals, coordinating multi-step processes, and adapting their behavior based on intermediate feedback. These capabilities are transforming the software development practice. As this emerging field evolves rapidly, there is a need to define its scope, consolidate its technical foundations, and identify open research challenges. This survey provides a comprehensive and timely review of AI agentic programming. We introduce a taxonomy of agent behaviors and system architectures, and examine core techniques including planning, memory and context management, tool integration, and execution monitoring. We also analyze existing benchmarks and evaluation methodologies used to assess coding agent performance. Our study identifies several key challenges, including limitations in handling long context, a lack of persistent memory across tasks, and concerns around safety, alignment with user intent, and collaboration with human developers. We discuss emerging opportunities to improve the reliability, adaptability, and transparency of agentic systems. By synthesizing recent advances and outlining future directions, this survey aims to provide a foundation for research and development in building the next generation of intelligent and trustworthy AI coding agents.

Open Questions about Time and Self-reference in Living Systems

Living systems exhibit a range of fundamental characteristics: they are active, self-referential, self-modifying systems. This paper explores how these characteristics create challenges for conventional scientific approaches and why they require new theoretical and formal frameworks. We introduce a distinction between 'natural time', the continuing present of physical processes, and 'representational time', with its framework of past, present and future that emerges with life itself. Representational time enables memory, learning and prediction, functions of living systems essential for their survival. Through examples from evolution, embryogenesis and metamorphosis we show how living systems navigate the apparent contradictions arising from self-reference as natural time unwinds self-referential loops into developmental spirals. Conventional mathematical and computational formalisms struggle to model self-referential and self-modifying systems without running into paradox. We identify promising new directions for modelling self-referential systems, including domain theory, co-algebra, genetic programming, and self-modifying algorithms. There are broad implications for biology, cognitive science and social sciences, because self-reference and self-modification are not problems to be avoided but core features of living systems that must be modelled to understand life's open-ended creativity.

I've done some pre-seeding to be able to auto-install Debian on bare metal and as virtual machines. Using classes is a useful addition for conditional execution of configuration management.

What you probably ought to be doing instead is specifying the option you want via the `classes` setting (an alias for `auto-install/classes`), and then implementing the behaviour you want by setting `preseed/run` in a minimal preseed.cfg, and having a script do whatever you want in response to the classes= setting you specified. So your preseed.cfg would just be:

d-i	preseed/run	string start.sh

and your start.sh would be something like:

#!/bin/sh
. /usr/share/debconf/confmodule

db_get auto-install/classes
db_set preseed/include "${RET:-default}".cfg;;

(you might want to check that the value of $RET makes sense before using it)

Then the real configs go in `default.cfg` and `alternative.cfg` (assuming you're planning on specifying `classes=alternative`)

You may be able to get inspiration from here: https://hands.com/d-i/

although the class handling there is quite deeply buried.

Here's a simpler example of using preseed/run to do odd things to D-I, that while not involving classes, is easier to follow: https://hands.com/d-i/bug/846002/

BTW `auto-install/classes` is reserved for users (it is not used by D-I internally), so it's there specifically for you to implement whatever scheme you like.

Additional information on pre-seeding can be found at B.2. Using preseeding.

A reference for my future self for some interesting troubleshooting capability

When triggering the issue please issue as well a Alt+SysRq+t to dump a list of current tasks and their information to the log. (on systems where you do not have the SysRq key, it might be the PrintScreen one). You need to enable as well debugging dumps, so have to enable it:

echo 8 > /proc/sys/kernel/sysrq

then once you have triggered the crash issue a Alt+SysRq+t and provide the log got over the netconsole.

More information on SysRq key hacks: https://docs.kernel.org/admin-guide/sysrq.html

Lastly, since you have a efi system, we might take as well advantage of logging via pstore. It seems active on your system, so check on next crash please as well /var/lib/systemd/pstore directory for fresh crash logs.

For pstore: https://docs.kernel.org/admin-guide/pstore-blk.html and systemd-pstore.service(8).

... so let's go the bisect route, I will replicate the excellent small howto from Uwe Kleine-Koenig here.

The whole will involve to compile multiple kernels. First we need to prepare a config for your system and to prepare the respective upstream versions, in your case we want to bisect the stable versions in one specific branch, so we can shortcut, and we know we want to bisect changes between 6.12.35 and 6.12.38 in your case.

    git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
    cd linux-stable
    git checkout v6.12.35
    cp /boot/config-$(uname -r) .config
    yes '' | make localmodconfig
    make savedefconfig
    mv defconfig arch/x86/configs/my_defconfig

As a fist step now compile a local 6.12.35 to ensure it is "good".

    # test 6.12.35 to ensure this is "good"
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm the problem is not present.

And now the same for the version you suspect is broken, is "bad" for git bisect speach.

    # test 6.12.38 to ensure this is "bad"
    git checkout v6.12.38
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm the problem with audio exists.

So we have now a range of from good to bad kernel and we can start the bisect:

    git bisect start v6.12.38 v6.12.35

    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install, try to boot and verify the state of the problem

and if the problem is hit run:

    git bisect bad

    git bisect good

Please pay attention to always select the just built kernel for booting, it won't always be the default kernel picked up by grub.

Then provide the output of

    git bisect log

In the course of the bisection you might have to uninstall previous kernels again to not exhaust the disk space in /boot. Also in the end uninstall all self-built kernels again.

Learned a new detail command for USB port details.

The below is a listing from an Asus X670e motherboard. Notice that the command provides port speeds (which implies a USB port type, which may be dependent upon capability/type of device connected). And yes, I still have a Microsoft 4000 keyboard. I wish they still made them. Extremely comfortable and has an excellent layout. Any good places to find them?

$ lsusb -vt
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 003.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/12p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 002: Dev 002, If 0, Class=Audio, Driver=snd-usb-audio, 480M
        ID 0424:3fb7 Microchip Technology, Inc. (formerly SMSC)
    |__ Port 002: Dev 002, If 1, Class=Audio, Driver=snd-usb-audio, 480M
        ID 0424:3fb7 Microchip Technology, Inc. (formerly SMSC)
    |__ Port 002: Dev 002, If 2, Class=Audio, Driver=snd-usb-audio, 480M
        ID 0424:3fb7 Microchip Technology, Inc. (formerly SMSC)
    |__ Port 002: Dev 002, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        ID 0424:3fb7 Microchip Technology, Inc. (formerly SMSC)
    |__ Port 006: Dev 003, If 0, Class=Wireless, Driver=btusb, 480M
        ID 0489:e0e2 Foxconn / Hon Hai
    |__ Port 006: Dev 003, If 1, Class=Wireless, Driver=btusb, 480M
        ID 0489:e0e2 Foxconn / Hon Hai
    |__ Port 006: Dev 003, If 2, Class=Wireless, Driver=btusb, 480M
        ID 0489:e0e2 Foxconn / Hon Hai
    |__ Port 007: Dev 004, If 0, Class=Vendor Specific Class, Driver=[none], 12M
        ID 0b05:19af ASUSTek Computer, Inc.
    |__ Port 007: Dev 004, If 2, Class=Human Interface Device, Driver=usbhid, 12M
        ID 0b05:19af ASUSTek Computer, Inc.
/:  Bus 004.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/5p, 20000M/x2
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 005.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/12p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 006.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/5p, 20000M/x2
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 007.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 008.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 009.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 002: Dev 003, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        ID 045e:00db Microsoft Corp. Natural Ergonomic Keyboard 4000 V1.0
    |__ Port 002: Dev 003, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M

I needed to step back a few commits to check where something went wrong. Or at what point something went wrong.

This works with a mixture of local commits and with primary commits sent to remote.

• git stash # whatever isn't committed yet
• git checkout X #go back in time - where X is the SHA256 hash of the commit
• git checkout . # throw away any changes
• git checkout master #go forward to current
• git stash pop # bring back any unsaved changes