Raymond P. Burkholder - Things I Do

Taking RTL_433 in unprivileged LXC container, with automatic device path change for inspiration, I was able to connect multiple APC and Eaton UPS USB cables to a server with the following mechanism. I forwarded the USB connections into an LXC container as I use my nut2mqtt - Communication between Network UPS Tools (NUT) and MQTT for monitoring.

The Proxmox host is configured in a standard way using the primary UPS USB connection (still yet to try).

As a USB cable is inserted, "journalctl -f" will log information similar to:

Apr 25 15:48:21 host02 kernel: usb 3-4: new full-speed USB device number 10 using xhci_hcd
Apr 25 15:48:21 host02 kernel: usb 3-4: New USB device found, idVendor=051d, idProduct=0002, bcdDevice= 0.90
Apr 25 15:48:21 host02 kernel: usb 3-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Apr 25 15:48:21 host02 kernel: usb 3-4: Product: Back-UPS XS 1300G FW:864.L6 .D USB FW:L6
Apr 25 15:48:21 host02 kernel: usb 3-4: Manufacturer: American Power Conversion
Apr 25 15:48:21 host02 kernel: usb 3-4: SerialNumber: 4B12P636
Apr 25 15:48:21 host02 kernel: hid-generic 0003:051D:0002.0007: hiddev3,hidraw5: ......

Use "lsusb" to determine the bus and device assignment based upon the 'idVendor' and 'idProduct' assignments above:

# lsusb -d 051d:0002
Bus 003 Device 010: ID 051d:0002 American Power Conversion Uninterruptible Power Supply

Thus you'll see an associated entry in /dev:

~# ls -altr /dev/bus/usb/003/010
crw-rw-rw- 1 root root 189, 265 Apr 25 15:48 /dev/bus/usb/003/010

Note the cgroup2 '189, 265' identifiers. Adding an entry to your /etc/pve/lxc/<lxc-id>.conf entry will be required:

lxc.cgroup2.devices.allow: c 189:* rwm

Whenever the USB cable is plugged in, a different device may be constructed. To pass that information to Proxmox for when it starts the container, use something like:

# cat /etc/udev/rules.d/50-ups-usb.rules
SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="051d", ENV{ID_MODEL_ID}=="0002", ENV{ID_SERIAL_SHORT}=="4B12P636", MODE="0666", RUN="/usr/sbin/pct set <lxc-id> --dev0 mode=0666,path=$devnode"

If the file is changed, it can be processed with:

# reload rules and rerun
udevadm control --reload-rules
udevadm trigger
# optional cable test simulation
udevadm test /dev/bus/usb/003/010

The ENV{ID_VENDOR_ID}, ENV{ID_MODEL_ID} & ENV{ID_SERIAL_SHORT} variable names and content can be confirmed with:

# udevadm info  /dev/bus/usb/003/010
P: /devices/pci0000:80/0000:80:14.0/usb3/3-4
M: 3-4
R: 4
J: c189:265
U: usb
T: usb_device
D: c 189:265
N: bus/usb/003/010
L: 0
V: usb
E: DEVPATH=/devices/pci0000:80/0000:80:14.0/usb3/3-4
E: DEVNAME=/dev/bus/usb/003/010
E: DEVTYPE=usb_device
E: DRIVER=usb
E: PRODUCT=51d/2/90
E: TYPE=0/0/0
E: BUSNUM=003
E: DEVNUM=010
E: MAJOR=189
E: MINOR=265
E: SUBSYSTEM=usb
E: USEC_INITIALIZED=2435707957355
E: ID_BUS=usb
E: ID_MODEL=Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6
E: ID_MODEL_ENC=Back-UPS\x20XS\x201300G\x20FW:864.L6\x20.D\x20USB\x20FW:L6\x20
E: ID_MODEL_ID=0002
E: ID_SERIAL=American_Power_Conversion_Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6_4B1233P63346
E: ID_SERIAL_SHORT=4B12P636
E: ID_VENDOR=American_Power_Conversion
E: ID_VENDOR_ENC=American\x20Power\x20Conversion
E: ID_VENDOR_ID=051d
E: ID_REVISION=0090
E: ID_USB_MODEL=Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6
E: ID_USB_MODEL_ENC=Back-UPS\x20XS\x201300G\x20FW:864.L6\x20.D\x20USB\x20FW:L6\x20
E: ID_USB_MODEL_ID=0002
E: ID_USB_SERIAL=American_Power_Conversion_Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6_4B1233P63346
E: ID_USB_SERIAL_SHORT=4B12P636
E: ID_USB_VENDOR=American_Power_Conversion
E: ID_USB_VENDOR_ENC=American\x20Power\x20Conversion
E: ID_USB_VENDOR_ID=051d
E: ID_USB_REVISION=0090
E: ID_USB_INTERFACES=:030000:
E: ID_VENDOR_FROM_DATABASE=American Power Conversion
E: ID_MODEL_FROM_DATABASE=Uninterruptible Power Supply
E: ID_PATH_WITH_USB_REVISION=pci-0000:80:14.0-usbv2-0:4
E: ID_PATH=pci-0000:80:14.0-usb-0:4
E: ID_PATH_TAG=pci-0000_80_14_0-usb-0_4
E: ID_FOR_SEAT=usb-pci-0000_80_14_0-usb-0_4
E: TAGS=:seat:
E: CURRENT_TAGS=:seat:

Resources:

• udev(7) — Linux manual page - Dynamic device management
• Persistent names for USB-serial devices in Linux (/dev/ttyUSBx -> /dev/custom-name) - I first looked at doing symlinks, but realized it is not required if I can perform a 'pct set' directly on the container configuration.

Note to self: come back to this and redo some non-functional audio logic located in some code which needs to be uprooted and re-juiced.

JUCE - open source, cross-platform, software development framework provided as C++ source code, that can be used to create standalone software on Windows, macOS, Linux, iOS and Android, as well as VST, VST3, AU, AUv3, AAX and LV2 plug-ins.

The Git Commands I Run Before Reading Any Code - some interesting simple queries on a git repository to gain some understanding on development patterns

Most changed files in the last year:

git log --format=format: --name-only --since="1 year ago" | sort | uniq -c | sort -nr | head -20

Hacker News - referenced

APT based distributions like Debian can be containerized with a tool called debootstrap. It is part of the image build process of lxc-create. It is also referenced in Docker Base Images for building an image from scratch.

When looking at the build scripts included in the package installation, repositories for the following distributions can be found in /usr/share/debootstrap/scripts:

• debian - universal operating system
• trisquel - a distribution of the GNU operating system, with the kernel GNU Linux-libre
• ubuntu - modern enterprise open source
• pardus - Turkish
• kali - open-source, Debian-based Linux distribution geared towards various information security tasks, such as Penetration Testing, Security Research, Computer Forensics and Reverse Engineering
• elxr - Enterprise-Grade Linux for Edge-to-Cloud Deployments
• pureos - fully-convergent, user friendly, secure and freedom respecting OS for your daily usage

The wiki shows a simple two liner to get the basics of the distribution in place (as root):

mkdir trixie-chroot
debootstrap stable trixie-chroot http://deb.debian.org/debian/

Enter the chroot and note new root:

root@test:~# pwd
/root
root@test:~# chroot trixie/
root@test:/# pwd
/
root@test:/# exit
exit
root@test:~# pwd
/root

After the debootstrap, create the base for docker, and give it a try:

tar -C trixie-chroot -c . | docker import - trixie
docker run trixie cat /etc/debian_version
docker run --rm -i -t trixie /bin/bash

Although debootstrap can be used to build an image for a version subsequent, it is generally recommended to use debootstrap from at least the desired version to ensure it has the proper updates and dependencies.

Command line to summarize the referenced repositories:

grep -h default_mirror /usr/share/debootstrap/scripts/* \
  | sed 's/default_mirror//' \
  | sed 's/[ \t]//g' \
  | sort \
  | uniq

Images vs Containers

• Docker Image: blueprint with app code and dependencies - static, read-only
• Docker Container: running instance of an image - dynamic, executable

Each instruction in the Dockerfile adds an extra layer to the Docker image. Minimize the number of layers by consolidating the instructions to increase the build’s performance and time.

Avoid using multiple RUN commands as it creates multiple cacheable layers which will affect the efficiency of the build process.

Use a single process per container: Each container should run a single process. This makes it easier to manage and monitor containers and helps to keep containers lightweight.

Images can exist without containers, whereas a container needs an image to run. We can create multiple containers from the same image, each with its own unique data and state

Docker commands

• Docker Run: It used for launching the containers from images, with specifying the runtime options and commands.
• Docker Pull: It fetches the container images from the container registry like Docker Hub to the local machine.
• Docker ps: It helps in displaying the running containers along with their important information like container ID, image used and status.
• Docker Stop: It helps in halting the running containers gracefully shutting down the processes within them.
• Docker Start: It helps in restarting the stopped containers, resuming their operations from the previous state.
• Docker Login: It helps to login in to the docker registry enabling the access to private repositories.

Docker network commands:

• docker network ls
• docker network inspect

Documentation

• Dockerfile reference - examples
• Dockerfile overview - with example
• Command Line Completion
• Base images - includes building from scratch
• CLI Cheat Sheet
• tutorial

First of all, the obligatory caveat from 2023: where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.

I do totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.

However, the combination of Docker in LXC is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application, particularly when an application is packaged as a Docker container, without recourse for building a native LXC container of the application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.

The key benefit is that devices such as one or more GPUs can be passed through to multiple LXC containers plus any nested docker containers. Otherwise, in the scenario where the GPU or PCIe device is passed through to a VM, as far as I know, it has to be dedicated to the VM. I've read that the devices can not be shared between a VM and LXC containers due to configuration differences between VM pass-through and LXC pass-through.

Given the caveat, I'll see if I can make this work. Not so easy. Trying to run

docker run --rm hello-world

Yields an error:

docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: 
mount source: "overlay", target: "/tmp/containerd-mount2030888385", 
fstype: overlay, flags: 0, 
data: "
  workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work,
  upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs,
  lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", 
  err: permission denied

With an associated apparmor error in Proxmox:

audit: type=1400 audit(1774803476.655:145): 
  apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 
  profile="lxc-131_" 
  name="/tmp/containerd-mount2030888385/" 
  pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"

The simple solution is to set nesting=1 in the proxmox lxc options.

The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:

> ps aux
root      41  0.0  0.0   2680  1808 ?    Ss   20:09   0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh

If so, this can be disabled:

systemctl disable ifupdown-wait-online.service

In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging

systemctl edit systemd-networkd-wait-online.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

In my case, I then saw something like:

root@frigate01:~# systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured

Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored
Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.

I have used a non-standard interface name. I resolved this by updating the edit with the following:

> systemctl edit systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60
#Environment=SYSTEMD_LOG_LEVEL=debug

The empty ExecStart line clears the original command parameters.

• How to Fix systemd-networkd-wait-online Service Timing Out During Boot
• systemd-networkd-wait-online.service(8) — Linux manual page

Some Docker commands:

docker run --rm -it hello-world bash

Docker installation is easy enough:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

This installs the latest stable release of:

• Docker CLI,
• Docker Engine,
• Docker Buildx,
• Docker Compose,
• containerd, and
• runc.

To get an idea of usage:

sh -c docker version

• ollama - framework for running models
• PyTorch Installation
• Hugging Face - models
• Open WebUI - extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It is built around universal standards, supporting Ollama and OpenAI-compatible Protocols (specifically Chat Completions).

How to Enable GPU Passthrough to LXC Containers in Proxmox indicates that the process of providing passthrough of a GPU to both an LXC container as well as a Virtual Machine is not possible as the two types of configurations conflict with each other.

As my own preference is to run whatever possible in LXC containers, I'll summarize the configuration I used, which is an amalgamation of configurations from several sites.

My current installation is ProxMox v9.1.6 with:

• ProArt Z890-CREATOR WIFI
• Intel(R) Core(TM) Ultra 9 285K
• Corsair CMP64GX5M2X6600C32 (128G 4400 MT/s) - ECC would have been nice
• NVIDIA Corporation AD103 [GeForce RTX 4070] (rev a1)

In BIOS/UEFI, enable these:

• VT-d / IOMMU
• Above 4G Decoding
• PCIe Native Power Management (if available)

Proxmox kernel parameters:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

update-grub
reboot

VFIO Binding - optional but recommended:

# /etc/modprobe.d/vfio.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Obtain Linux drivers from NVidia. The CUDA toolkit is not required. Only the drivers are required in ProxMox. Toolkits and add-ons are added within the container.

Install the drivers:

apt install build-essential
apt install pve-headers-$(uname -r)
sh NVIDIA-Linux-x86_64-595.58.03.run
# note, use the open kernel, rather than proprietary

Blacklist nouveau:

cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF

Test that the card is accessible:

nvidia-smi

Enable Data Persistence to prevent the GPU from re-initializing with each use.

nvidia-persistenced --persistence-mode
systemctl enable nvidia-persistenced

Then restart:

reboot

Identify the nvidia devices requiring passthrough:

root@host02:~# ls -al /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Mar 28 11:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Mar 28 11:58 /dev/nvidiactl
crw-rw-rw- 1 root root 505,   0 Mar 28 11:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 505,   1 Mar 28 11:58 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Mar 28 11:58 .
drwxr-xr-x 21 root root   5060 Mar 28 11:58 ..
cr--------  1 root root 508, 1 Mar 28 11:58 nvidia-cap1
cr--r--r--  1 root root 508, 2 Mar 28 11:58 nvidia-cap2

Note the numbers 195, 505 and 508 in this list (yours may be different).

Construct a container, and prior to starting, place the following into /etc/pve/lxc/<vmid>.conf (based upon the device listing above):

dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-modeset
dev3: /dev/nvidia-uvm
dev4: /dev/nvidia-uvm-tools
dev5: /dev/nvidia-caps/nvidia-cap1
dev6: /dev/nvidia-caps/nvidia-cap2

These lines are optional in the config file, one site talks about them by my container seems to work without them (they may be an old style cgroup2 style passthrough rather than the device oriented passthrough above):

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 505:* rwm
lxc.cgroup2.devices.allow: c 508:* rwm

Start the container and push the driver file into the container:

pct push <vmid> downloads/NVIDIA-Linux-x86_64-595.58.03.run /root/NVIDIA-Linux-x86_64-595.58.03.run

In the container, install the driver, minus the kernel module:

apt install kmod
sh NVIDIA-Linux-x86_64-595.58.03.run --no-kernel-modules

Run nvidia-smi in the container to confirm the card is reachable.

Add nvtop at the host or the container level to chart live GPU utllization:

apt install nvtop

Additional resources:

• Complete GPU passthrough guide for AI workloads, avoid the mistakes I made so you don't have to
• [TUTORIAL] NVIDIA drivers instalation Proxmox and CT
• How to Enable GPU Passthrough to LXC Containers in Proxmox - contains ollama startup examples with OpenWebUI

Another type of test to run when pytorch is installed:

python -c "import torch; print(torch.cuda.is_available())"

Some say this happens due to Wayland usage. My usage is Wayland/X11, not that it may make a difference. It is that way because I like that fact that the windows of an application get placed back to where they were when the application is re-opened.

The solution appears to be:

Tools -> Options -> General -> Enable 'Use LibreOffice dialogs'

In the Nanog email list, the following was posted as a summary of current tooling use for network management in Debian:

Linux has a bunch of different possible ways to administer all of this stuff.

• The most comprehensive CLI mechanism is iproute2 (the “ip” command and some related constructs).
• The most comprehensive and capable persistent configuration database mechanism is systemd-networkd.

Other persistent mechanisms include:

• Netplan (YAML based configurations that now days mostly get parsed into systemd-networkd files and then executed).
• Debian Traditional (the /etc/network/interfaces file and/or interfaces.d directory, ifup/ifdown/etc.). Lacks many features, but most can be worked around with iproute2 shell commands added to triggers in the file.
• Debian Traditional can be supplemented with ifupdown2 - ifupdown replacement from Cumulus Networks
• NetworkManager (semi-capable, but any capabilities it lacks are just hard to cope with).

My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done.

Overall, it’s a pretty comprehensive interface and provides good logs for troubleshooting in most circumstances.

When creating a Debian testing/forky LXC container on a Debian trixie machine, the following error may be encountered in the output:

I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://deb.debian.org/debian...
E: Couldn't find these debs: isc-dhcp-client
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian

This is a result of bug #1125011 in the Debian bug tracker.

There are several possible solutions:

• Manually apply the patches supplied by the Debian LXC team
• Probably might be solved by running lxc-create on a testing/forky machine, where the solution may have already been applied - I have not confirmed this
• Or it may work on a sid machine

If running Firefox on a Debian Linux machine, install virt-viewer:

sudo apt install virt-viewer

Ensure the VirtIO drivers and such have been installed in the virtual machine (if Windows) in order to provide SPICE services.

Then, in Firefox on your workstation:

• go into about:config, and add the key 'network.protocol-handler.expose.virt-viewer' as boolean and set to true
• go into about:preferences, and set "What should Firefox do with other files" to "Ask whether to open or save files".
• in Proxmox, open a SPICE based console for a virtual machine, which attempts a download or a run of a customized .vv file,
• Firefox will then request to open a Virt-Viewer file with Remote Viewer - at this point, you can set it as the default viewer, and it will show up in the application preferences

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

I used the following for a trixie v13.3 version of a container:

lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,

Pip vs Pipenv: Which is better and which to learn first compares pipenv vs the python3-pip and virtualenv package combo.

After referring to that, I think I'll just stick with the standard pip/virtualenv combo for now.

To get started:

# install basic packages
apt-get install python3 python3-pip virtualenv  python3-venv git

# create a project directory - example ansible
python3 -m venv ansible

# activate the project
cd ansible
source bin/activate

# example installation of packages
pip install ansible
pip install argcomplete
activate-global-python-argcomplete
source ~/.bash_completion
ansible-config init --disabled > ansible.cfg

# to deactivate the project
deactivate

# upgrade
python3 -m pip install --upgrade ansible

At some point, integrate python3-ansible-runner, the github source and links to documentation.