Raymond P. Burkholder - Things I Do

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

Pip vs Pipenv: Which is better and which to learn first compares pipenv vs the python3-pip and virtualenv package combo.

After referring to that, I think I'll just stick with the standard pip/virtualenv combo for now.

To get started:

# install basic packages
apt-get install python3 python3-pip virtualenv  python3-venv git

# create a project directory - example ansible
python3 -m venv ansible

# activate the project
cd ansible
source bin/activate

# example installation of packages
pip install ansible
pip install argcomplete
activate-global-python-argcomplete
source ~/.bash_completion
ansible-config init --disabled > ansible.cfg

# to deactivate the project
deactivate

# upgrade
python3 -m pip install --upgrade ansible

At some point, integrate python3-ansible-runner, the github source and links to documentation.

pct_id=101
pct_name=test01
pct create $pct_id /var/lib/pve/local-btrfs/template/cache/trixie-13-3-template.tar.xz  \
  -hostname $pct_name \
  -description 'demo build' \
  -onboot 1 \
  -startup up=3 \
  -ostype debian \
  -arch amd64 \
  -cores 2 \
  -memory 1024 \
  -nameserver 10.10.10.10 -searchdomain 'example.com' \
  -net0 name=vlan30,bridge=vmbr1,ip=dhcp,tag=30,type=veth \
  -rootfs local-btrfs:8,mountoptions="noatime;discard" \
  -swap 512

There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:

• Build a linux based virtual machine, I use Debian's recent release
• Install LXC and its template package
• Construct and initialize an LXC container
• Shut it down and and zip it up
• Copy it over to the ProxMox template directory

The details:

# build the linux vm - details not relevant here
# ssh into the vm, or start a command line
# install basic packages

sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync

# basic container templates are in:
#   /usr/share/lxc/templates/ 
# for debian as well as other distributions

# create an lxc container, provide a list any additional packages

lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny

# start and attach to the container
lxc-start trixie-template
lxc-attach trixie-template

# prepare for generating template
apt clean
apt purge

# Remove SSH host keys to ensure unique keys for each clone:
rm /etc/ssh/ssh_host_*

# Empty the machine ID file:
truncate -s 0 /etc/machine-id

# clear history
unset HISTFILE
# truncate history
history -c
> ~/.bash_history
# the following has a space in front to prevent inclusion in the history
 shutdown -h now

# the shutdown returns to the virtual machine's prompt
# compress the directory structure

cd /var/lib/lxc/trixie-template/

# remove /dev files as they can't be created in an unprivileged container
# an example error message if not removed:
#   tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted
# construction of a new container will re-create the directory and files

rm ./rootfs/dev/ptmx
rm ./rootfs/dev/zero
rm ./rootfs/dev/tty3
rm ./rootfs/dev/urandom
rm ./rootfs/dev/null
rm ./rootfs/dev/tty
rm ./rootfs/dev/console
rm ./rootfs/dev/tty4
rm ./rootfs/dev/tty2
rm ./rootfs/dev/random
rm ./rootfs/dev/tty1
rm ./rootfs/dev/full

# cd into rootfs and zip the container

cd rootfs
tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./

# the xz file can be copied over to proxmox and placed into
# /var/lib/pve/local-btrfs/template/cache/
# for use as a template for container creation

During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.

If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:

lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less

An existing cache can be updated with something like:

sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64
apt-get update
apt-get dist-upgrade
apt-get clean
exit

courtesy of Updating lxc image/container caches

One other note, there are two package candidates for installing the ping utility:

• iputils-ping - native Linux ping, preferred for Debian/Linux
• inetutils-ping - general gnu version, used on a variety of posix sytstems, less preferred

Some fix-ups in the process:

• apt-get install less
• dpkg-reconfigure locales
• useradd user

Something from a Debian mailing list:

I found the root cause, when testing 6.12.57 I installed the image then the headers and the NVIDIA DKMS module was not rebuilt because the matching linux- headers package was not installed at the time the kernel image was configured.

If I install the headers first and then the linux-image package, DKMS correctly builds the NVIDIA module and 6.12.63 works fine, so it doesn't look like a kernel regression after all.

I don't know if I should manually run dkms autoinstall myself after a kernel update (I never had to before) or if there was a bug during the install process of this update.

Makes sense, I had NVidia compile fail in a similar. This makes it obvious what I should have observed.

The difference between reactive and proactive monitoring comes down to tracking the right network performance metrics. Important metrics that consistently predict issues before they impact users include:

• Round-trip time variations: Subtle increases in RTT often precede more serious network congestion. Set baseline thresholds for different times of day and get alerted when patterns deviate
• Buffer utilization patterns: Most admins ignore this until it's too late. Monitoring buffer usage across key network devices reveals impending bottlenecks before they trigger packet loss
• Error rates on interfaces: Even small increases in CRC errors or interface discards can signal hardware issues days before actual failures occur
• TCP retransmission rates: This metric reveals application performance issues that users might experience from data transmission rates before they report them
• Network traffic symmetry: Unusual asymmetric traffic patterns often indicate security issues or misconfigured applications

Reference: Stop Reactive Network Troubleshooting: Monitor These 5 Metrics to Prevent Downtime

Cloud environments introduce unique challenges for network performance monitoring. Some standard metrics like latency and packet loss remain important, but you'll also want to focus on metrics specific to your cloud connectivity. For hybrid environments, these are good metrics:

• Inter-region latency: Especially important for globally distributed applications
• Connection establishment time: Often overlooked but crucial for microservice architectures
• Throughput consistency: More important than raw bandwidth in many cloud scenarios
• DNS resolution time: Can be a hidden bottleneck in cloud environments

It has been a while since I last setup NordVPN on a Debian Linux using StrongSwan. StrongSwan is now using 'native' files rather than the now deprecated ipsec files. NordVPN Example: How to connect to NordVPN with IKEv2/IPSec on Linux refers to the old format. Here is a new format.

Here is my take on a successful installation.

apt install \
  --no-install-recommends \
    strongswan \
    libstrongswan-standard-plugins \
    libstrongswan-extra-plugins \
    libcharon-extra-plugins
wget https://downloads.nordcdn.com/certificates/root.pem -O /etc/swanctl/x509ca/NordVPN.pem
sed -i 's/load = yes/load = no/' /etc/strongswan.d/charon/constraints.conf

An example /etc/swanctl/swanctl.conf file:

connections {
  nordvpn {
    version = 2
    proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-ecp521,aes192-sha256-modp3072,default
    rekey_time = 0s
    fragmentation = yes
    dpd_delay = 300s
    local_addrs = %defaultroute
    remote_addrs = 
    vips=0.0.0.0,::
    local {
      auth = eap-mschapv2
      eap_id = "<username>"
    }
    remote {
      auth = pubkey
      cacerts = /etc/swanctl/x509ca/NordVPN.pem
      id = %any
    }
    children {
      nordvpn {
        remote_ts = 0.0.0.0/0,::/0
        rekey_time = 0s
        dpd_action = clear
        esp_proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-modp3072,aes192-sha256-ecp256-modp3072,default
      }
    }
  }
}

secrets {
  eap-nordvpn {
    id = "<username>"
    secret = "<password>"
  }
}

If you have a local network to which you need access when the vpn is up, StrongSwan using route table 220 for forwarding. Use the following command to see current settings:

# ip rule list
0:      from all lookup local
220:    from all lookup 220
32766:  from all lookup main
32767:  from all lookup default

# ip route list table 220
default via 192.168.1.10 dev eth0 proto static src 

To add your local network to the route table. Additional subnets are added in a similar way. Change the interface name to suit your local circumstances. Use the updown Plugin for better control of the local routing.

ip route add table 192.168.1.0/24 dev wlan0

This may be required for changes made:

# systemctl restart strongswan

Tunnel related state and status commands:

sudo swanctl --load-conns
sudo swanctl --list-conns
sudo swanctl --list-certs
sudo swanctl --list-sas
sudo swanctl --initiate --child nordvpn
sudo swanctl --terminate --child nordvpn
sudo swanctl --reload-settings

References:

• How to Setup an IKEv2 VPN Connection on Arch Linux (Example: NordVPN) - primary configuration template
• Allow Strongswan roadwarrior to access local LAN - keeping local traffic out of the vpn

In Proxmox, one can visually connect to an active virtual machine through one of two ways:

• noVNC - default mechanism - works with most default installations
• SPICE - enhanced mechanism - requires virtual machine support

With Windows based Virtual Machines, the SPICE client can be installed via the Redhat supplied VirtIO .iso.

If running Firefox from a Debian Linux machine, run 'sudo apt install virt-viewer' to install the viewer executable.

As a one time configuration in Firefox:

• go into about:config, and add the key 'network.protocol-handler.expose.virt-viewer' as boolean and set to true
• go into about:preferences, and set "What should Firefox do with other files" to "Ask whether to open or save files".
• in Proxmox, open a SPICE console for a virtual machine, which downloads a customized .vv file,
• Firefox will then request to open a Virt-Viewer file with Remove Viewer - at this point, you can set it as the default viewer, and it will show up in the application preferences

When installing a Microsoft Windows 11 Operating System in Proxmox, a few helpful hints here. This is with Proxmox v9.

Windows 11 can be downloaded from Microsoft Windows 11 Download. During the installation, you'll need a license key, so have that ready.

The RedHat Virtio drivers will be required during initial install.

When defining the configuration for the virtual machine in proxmox, attach the Win11 ISO, attach EUFI local, and provide a TPM area.

When starting from the ISO, hit a key to start. During the process of selecting and formatting the drive, you'll need to browse to the Virtio ISO, and browse into the \amd64\win11 sub-directory to load the initial storage sub-system drivers.

Later on in the process, the installation will request access to the network. Use shift-F10 to start a text console. In there, type:

oobe\bypassnro

The machine will reboot. This can be used to bypass making a Microsoft account and to create a local account instead.

Optionally:

• Once it returns, you need to disconnect networking. So you can do terminal again and ipconfig /release, or just set the NIC for disconnected.
• At some point it'll guilt you into creating an account but there will be an "I don't have internet" option, and that's your way out.

Inspiration from How to install Win11 in Proxmox | Quick guide | And fix problems of network search.

From Install Windows 11 22H2 , some other tricks:

• in the user box type "no@thankyou.com" and any password. It'll error out when trying to login and let you click next to create a local account. Dev bypass must be
• When connecting to the network, Press Win key + E to bring up an explorer window, install all the drivers needed off the driver ISO
• Windows 10 guest best practices
• Choose setup as work account + choose domain setup. The account it first creates is a local account. (I did this as of last week)
• oobe = out-of-box-experience

From Hacker News - %CPU utilization is a lie referencing Brendan Long's Blog

"a server at 60% utilization had zero room left" can be explained with queuing theory:

• Up to a hair over 60% utilization the queuing delays on any work queue remain essentially negligible. At 70 they become noticeable, and at 80% they've doubled. And then it just turns into a shitshow from there on.
• rule of thumb is 60% is zero, and 80% is the inflection point where delays go exponential.
• CPU utilization metrics are frequently averaged over a long period of time (like a minute), but if your SLO is 100 ms, what you care about is whether there's any ~100 ms period where CPU utilization is at 100%. Measuring p99 (or even p100) CPU utilization can make this a lot more visible.

Lifted from Installing Virtio Drivers for Windows on Proxmox to Enable Paravirtualization / Fix Incorrect Memory Monitoring, there is a section at the end which discusses settings for disk caching modes in Proxmox. The terms relate to how frequently data is flushed to the underlying storage system. There are trade-offs for performance and durability.

• Direct sync - highest data security but has poor performance - ensures that every time data is written to memory, it is immediately synchronized to the disk. This means each write operation introduces disk latency because the system must wait for the disk to confirm the write is complete before continuing with other operations. Although this method provides the highest data security, frequent disk operations lead to lower performance.
• Write through - balance between data consistency and performance - writing data to memory and immediately writes it to disk without waiting for disk confirmation. It offers good data consistency and high performance by reducing the time spent waiting for disk confirmation. However, disk write operations still occur, leading to some performance overhead.
• Write back - improves performance but comes with certain data risks - first writes data to the memory cache and writes it back to the disk at an appropriate time. This reduces the number of disk accesses and improves system performance. However, because data is cached in memory, there is a risk of data loss if the system crashes or experiences a power outage before writing back to the disk.
• Write back (unsafe) - highest performance (fastest among all modes) but compromises data security and consistency - similar to the regular write back method, but it makes no guarantees about data consistency. This means that in the event of a system failure or power outage, data not yet written back to the disk could be permanently lost. While this method provides higher performance, it sacrifices data security and consistency. RamFS would be a use case here.

Perform your own testing given your own physical infrastructure and weigh the pro's and con's.

Starting with version VE 8.4, proxmox host directories can be shared into a windows or linux virtual machine. I have a version 9 running here as an example:

# pveversion
pve-manager/9.0.6/49c767b70aeb6648 (running kernel: 6.14.8-2-pve)

Check to see that the appropriate package is installed, it should be by default:

# dpkg -l | grep virtiofsd
ii  virtiofsd         1.13.2-1      amd64     Virtio-fs vhost-user device daemon

Create a directory for sharing, use something like /mnt or /var/local, depending upon your preference:

# mkdir /var/local/shared

In the proxmox management site, navigate Datacenter -> Directory Mappings, then add a mapping with the directory just created. The name supplied will be the name seen in the virtual machine.

Also in the proxmox management site, navigate to the virtual machine settings -> Hardware and add a Virtiofs Filesystem Passthrough. Select the directory id you just created. Select the 'Direct IO' box (may be optional). Then restart the machine if already started to have the setting take effect.

In a Windows virtual machine, be sure you've installed the Windows VirtIO Drivers.

Open the Services cmdlet (services.msc). There should be a 'VirtIO-FS Service". Startup type can be set to automatic. If only temporary use is required, simply start the service. A new device should become visible in This PC in file explorer.

For Windows based virtual machines, WinFSP is required. It is mentioned at How to install virtiofs drivers on Windows.

Once installed, open services.msc, and start the VirtIO-FS Service

In explorer, you should see the shared data folder in 'This PC'.

Risk Manager for Intrusion Tolerant Systems: Enhancing HAL 9000 with New Scoring and Data Sources

Intrusion Tolerant Systems (ITSs) have become increasingly critical due to the rise of multi-domain adversaries exploiting diverse attack surfaces. ITS architectures aim to tolerate intrusions, ensuring system compromise is prevented or mitigated even with adversary presence. Existing ITS solutions often employ Risk Managers leveraging public security intelligence to adjust system defenses dynamically against emerging threats. However, these approaches rely heavily on databases like NVD and ExploitDB, which require manual analysis for newly discovered vulnerabilities. This dependency limits the system's responsiveness to rapidly evolving threats. HAL 9000, an ITS Risk Manager introduced in our prior work, addressed these challenges through machine learning. By analyzing descriptions of known vulnerabilities, HAL 9000 predicts and assesses new vulnerabilities automatically. To calculate the risk of a system, it also incorporates the Exploitability Probability Scoring system to estimate the likelihood of exploitation within 30 days, enhancing proactive defense capabilities.

Despite its success, HAL 9000's reliance on NVD and ExploitDB knowledge is a limitation, considering the availability of other sources of information. This extended work introduces a custom-built scraper that continuously mines diverse threat sources, including security advisories, research forums, and real-time exploit proofs-of-concept. This significantly expands HAL 9000's intelligence base, enabling earlier detection and assessment of unverified vulnerabilities. Our evaluation demonstrates that integrating scraper-derived intelligence with HAL 9000's risk management framework substantially improves its ability to address emerging threats. This paper details the scraper's integration into the architecture, its role in providing additional information on new threats, and the effects on HAL 9000's management.

This is an interesting take on security: toleration of intruders to systems. I suppose this is somewhat akin to limiting blast radius. One interesting nugget:

... system applies a proactive recovery mechanism to protect the system from undetected attacks. A central controller periodically rotates the state of each VM, transitioning between active, cleansing, ready, and active states.

This would be the pre-emption step of what file checksum utilities would do. However, this doesn't eliminate the types of intrusions where intruders make use of elevated account access.

AI agentic programming is an emerging paradigm in which large language models (LLMs) autonomously plan, execute, and interact with external tools like compilers, debuggers, and version control systems to iteratively perform complex software development tasks. Unlike conventional code generation tools, agentic systems are capable of decomposing high-level goals, coordinating multi-step processes, and adapting their behavior based on intermediate feedback. These capabilities are transforming the software development practice. As this emerging field evolves rapidly, there is a need to define its scope, consolidate its technical foundations, and identify open research challenges. This survey provides a comprehensive and timely review of AI agentic programming. We introduce a taxonomy of agent behaviors and system architectures, and examine core techniques including planning, memory and context management, tool integration, and execution monitoring. We also analyze existing benchmarks and evaluation methodologies used to assess coding agent performance. Our study identifies several key challenges, including limitations in handling long context, a lack of persistent memory across tasks, and concerns around safety, alignment with user intent, and collaboration with human developers. We discuss emerging opportunities to improve the reliability, adaptability, and transparency of agentic systems. By synthesizing recent advances and outlining future directions, this survey aims to provide a foundation for research and development in building the next generation of intelligent and trustworthy AI coding agents.

Open Questions about Time and Self-reference in Living Systems

Living systems exhibit a range of fundamental characteristics: they are active, self-referential, self-modifying systems. This paper explores how these characteristics create challenges for conventional scientific approaches and why they require new theoretical and formal frameworks. We introduce a distinction between 'natural time', the continuing present of physical processes, and 'representational time', with its framework of past, present and future that emerges with life itself. Representational time enables memory, learning and prediction, functions of living systems essential for their survival. Through examples from evolution, embryogenesis and metamorphosis we show how living systems navigate the apparent contradictions arising from self-reference as natural time unwinds self-referential loops into developmental spirals. Conventional mathematical and computational formalisms struggle to model self-referential and self-modifying systems without running into paradox. We identify promising new directions for modelling self-referential systems, including domain theory, co-algebra, genetic programming, and self-modifying algorithms. There are broad implications for biology, cognitive science and social sciences, because self-reference and self-modification are not problems to be avoided but core features of living systems that must be modelled to understand life's open-ended creativity.