Raymond P. Burkholder - Things I Do

From Plex GPU transcoding in Docker on LXC on Proxmox v2:

# make sure that all nvidia devices are loaded upon boot
cat >/etc/systemd/system/nvidia-pre-lxc-init.service <<'EOF'
[Unit]
Description=Initialize NVIDIA devices early (before Proxmox guests)
After=systemd-modules-load.service
Before=pve-guests.service
Wants=pve-guests.service

[Service]
Type=oneshot
RemainAfterExit=yes

ExecStartPre=-/sbin/modprobe nvidia
ExecStartPre=-/sbin/modprobe nvidia_uvm
ExecStart=-/usr/bin/nvidia-smi -L
ExecStart=-/usr/bin/nvidia-modprobe -u -c=0

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable nvidia-pre-lxc-init.service

reboot then try nvidia-smi to confirm function.

• Qwen 3.5 9B - general purpose locally run LLM
• OctoPrint - web-based 3D printer control software that allows you to remotely control and monitor your 3D printer from a web interface. It was designed to be compatible with a wide range of 3D printers.
• Trilium Notes - self-hosted note-taking and personal knowledge management application. It enables users to organize information in a hierarchical tree structure and supports rich text editing, internal linking, images, attachments, and powerful scripting capabilities. This version reflects the most current development efforts under the TriliumNext organization and replaces all prior forks or legacy variants. Trilium is ideal for building personal wikis, structured documentation, and long-term knowledge archives, giving users full local control and privacy.
• Vaultwarden - self-hosted password manager which provides secure and encrypted password storage. It uses client-side encryption and provides access to passwords through a web interface and mobile apps.
• NetBox - the source of truth for everything on your network, from physical components like power systems and cabling to virtual assets like IP addresses and VLANs. Network automation and observability tools depend on NetBox’s authoritative data to roll out configurations, monitor changes, and accelerate operations across the enterprise

Not having the desire to put frigate on a standalone server of some sort, I wanted it to run as a container in Proxmox. Since Frigate is supplied as a Docker container, and I don't want to run Docker directly on the Proxmox host, there are two choices open to me:

• run Frigate in a virtual machine - prevents sharing an Nvidia card with other machines or containers, or
• run Frigate in an LXC or an OCI container - an Nvidia card can be shared with multiple containers, and resource requirements are reduced, at the risk of security issues (which are reduced if the host is a private server)

I'm running on Proxmox 9.1.9 with kernel SMP PREEMPT_DYNAMIC PMX 6.17.13-4 (2026-04-21T22:03Z) x86_64 GNU/Linux

This installation is a privileged container as I had troubles with non-privileged. I'll have to come back to this based upon: Running in rootless LXC, and maybe require the user id mapping from Unprivileged LXC containers.

Some installation commands, based upon connecting Frigate to an Nvidia GPU. Some instructions come from Installing the NVIDIA Container Toolkit, which is required for connecting to the Nvidia GPU (commands assume sudo or running as root, which is default in a basic LXC container). The link contains installation, testing and troubleshooting instructions. Continue reading "Frigate Running in Docker inside LXC on Proxmox" »

Taking RTL_433 in unprivileged LXC container, with automatic device path change for inspiration, I was able to connect multiple APC and Eaton UPS USB cables to a server with the following mechanism. I forwarded the USB connections into an LXC container as I use my nut2mqtt - Communication between Network UPS Tools (NUT) and MQTT for monitoring.

The Proxmox host is configured in a standard way using the primary UPS USB connection (still yet to try).

As a USB cable is inserted, "journalctl -f" will log information similar to:

Apr 25 15:48:21 host02 kernel: usb 3-4: new full-speed USB device number 10 using xhci_hcd
Apr 25 15:48:21 host02 kernel: usb 3-4: New USB device found, idVendor=051d, idProduct=0002, bcdDevice= 0.90
Apr 25 15:48:21 host02 kernel: usb 3-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Apr 25 15:48:21 host02 kernel: usb 3-4: Product: Back-UPS XS 1300G FW:864.L6 .D USB FW:L6
Apr 25 15:48:21 host02 kernel: usb 3-4: Manufacturer: American Power Conversion
Apr 25 15:48:21 host02 kernel: usb 3-4: SerialNumber: 4B12P636
Apr 25 15:48:21 host02 kernel: hid-generic 0003:051D:0002.0007: hiddev3,hidraw5: ......

Use "lsusb" to determine the bus and device assignment based upon the 'idVendor' and 'idProduct' assignments above:

# lsusb -d 051d:0002
Bus 003 Device 010: ID 051d:0002 American Power Conversion Uninterruptible Power Supply

Thus you'll see an associated entry in /dev:

~# ls -altr /dev/bus/usb/003/010
crw-rw-rw- 1 root root 189, 265 Apr 25 15:48 /dev/bus/usb/003/010

Note the cgroup2 '189, 265' identifiers. Adding an entry to your /etc/pve/lxc/<lxc-id>.conf entry will be required:

lxc.cgroup2.devices.allow: c 189:* rwm

Whenever the USB cable is plugged in, a different device may be constructed. To pass that information to Proxmox for when it starts the container, use something like:

# cat /etc/udev/rules.d/50-ups-usb.rules
SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="051d", ENV{ID_MODEL_ID}=="0002", ENV{ID_SERIAL_SHORT}=="4B12P636", MODE="0666", RUN="/usr/sbin/pct set <lxc-id> --dev0 mode=0666,path=$devnode"

If the file is changed, it can be processed with:

# reload rules and rerun
udevadm control --reload-rules
udevadm trigger
# optional cable test simulation
udevadm test /dev/bus/usb/003/010

The ENV{ID_VENDOR_ID}, ENV{ID_MODEL_ID} & ENV{ID_SERIAL_SHORT} variable names and content can be confirmed with:

# udevadm info  /dev/bus/usb/003/010
P: /devices/pci0000:80/0000:80:14.0/usb3/3-4
M: 3-4
R: 4
J: c189:265
U: usb
T: usb_device
D: c 189:265
N: bus/usb/003/010
L: 0
V: usb
E: DEVPATH=/devices/pci0000:80/0000:80:14.0/usb3/3-4
E: DEVNAME=/dev/bus/usb/003/010
E: DEVTYPE=usb_device
E: DRIVER=usb
E: PRODUCT=51d/2/90
E: TYPE=0/0/0
E: BUSNUM=003
E: DEVNUM=010
E: MAJOR=189
E: MINOR=265
E: SUBSYSTEM=usb
E: USEC_INITIALIZED=2435707957355
E: ID_BUS=usb
E: ID_MODEL=Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6
E: ID_MODEL_ENC=Back-UPS\x20XS\x201300G\x20FW:864.L6\x20.D\x20USB\x20FW:L6\x20
E: ID_MODEL_ID=0002
E: ID_SERIAL=American_Power_Conversion_Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6_4B1233P63346
E: ID_SERIAL_SHORT=4B12P636
E: ID_VENDOR=American_Power_Conversion
E: ID_VENDOR_ENC=American\x20Power\x20Conversion
E: ID_VENDOR_ID=051d
E: ID_REVISION=0090
E: ID_USB_MODEL=Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6
E: ID_USB_MODEL_ENC=Back-UPS\x20XS\x201300G\x20FW:864.L6\x20.D\x20USB\x20FW:L6\x20
E: ID_USB_MODEL_ID=0002
E: ID_USB_SERIAL=American_Power_Conversion_Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6_4B1233P63346
E: ID_USB_SERIAL_SHORT=4B12P636
E: ID_USB_VENDOR=American_Power_Conversion
E: ID_USB_VENDOR_ENC=American\x20Power\x20Conversion
E: ID_USB_VENDOR_ID=051d
E: ID_USB_REVISION=0090
E: ID_USB_INTERFACES=:030000:
E: ID_VENDOR_FROM_DATABASE=American Power Conversion
E: ID_MODEL_FROM_DATABASE=Uninterruptible Power Supply
E: ID_PATH_WITH_USB_REVISION=pci-0000:80:14.0-usbv2-0:4
E: ID_PATH=pci-0000:80:14.0-usb-0:4
E: ID_PATH_TAG=pci-0000_80_14_0-usb-0_4
E: ID_FOR_SEAT=usb-pci-0000_80_14_0-usb-0_4
E: TAGS=:seat:
E: CURRENT_TAGS=:seat:

Resources:

• udev(7) — Linux manual page - Dynamic device management
• Persistent names for USB-serial devices in Linux (/dev/ttyUSBx -> /dev/custom-name) - I first looked at doing symlinks, but realized it is not required if I can perform a 'pct set' directly on the container configuration.

Note to self: come back to this and redo some non-functional audio logic located in some code which needs to be uprooted and re-juiced.

JUCE - open source, cross-platform, software development framework provided as C++ source code, that can be used to create standalone software on Windows, macOS, Linux, iOS and Android, as well as VST, VST3, AU, AUv3, AAX and LV2 plug-ins.

The Git Commands I Run Before Reading Any Code - some interesting simple queries on a git repository to gain some understanding on development patterns

Most changed files in the last year:

git log --format=format: --name-only --since="1 year ago" | sort | uniq -c | sort -nr | head -20

List of contributors:

git shortlog -sn --no-merges

Example of looking for hot words in comment messages:

git log -i -E --grep="fix|bug|broken" --name-only --format='' | sort | uniq -c | sort -nr | head -20

Time line of commits:

git log --format='%ad' --date=format:'%Y-%m' | sort | uniq -c

Another example of comment keyword searches:

git log --oneline --since="1 year ago" | grep -iE 'revert|hotfix|emergency|rollback'

Hacker News - referenced

APT based distributions like Debian can be containerized with a tool called debootstrap. It is part of the image build process of lxc-create. It is also referenced in Docker Base Images for building an image from scratch.

When looking at the build scripts included in the package installation, repositories for the following distributions can be found in /usr/share/debootstrap/scripts:

• debian - universal operating system
• trisquel - a distribution of the GNU operating system, with the kernel GNU Linux-libre
• ubuntu - modern enterprise open source
• pardus - Turkish
• kali - open-source, Debian-based Linux distribution geared towards various information security tasks, such as Penetration Testing, Security Research, Computer Forensics and Reverse Engineering
• elxr - Enterprise-Grade Linux for Edge-to-Cloud Deployments
• pureos - fully-convergent, user friendly, secure and freedom respecting OS for your daily usage

The wiki shows a simple two liner to get the basics of the distribution in place (as root):

mkdir trixie-chroot
debootstrap stable trixie-chroot http://deb.debian.org/debian/

Enter the chroot and note new root:

root@test:~# pwd
/root
root@test:~# chroot trixie/
root@test:/# pwd
/
root@test:/# exit
exit
root@test:~# pwd
/root

After the debootstrap, create the base for docker, and give it a try:

tar -C trixie-chroot -c . | docker import - trixie
docker run trixie cat /etc/debian_version
docker run --rm -i -t trixie /bin/bash

Although debootstrap can be used to build an image for a version subsequent, it is generally recommended to use debootstrap from at least the desired version to ensure it has the proper updates and dependencies.

Command line to summarize the referenced repositories:

grep -h default_mirror /usr/share/debootstrap/scripts/* \
  | sed 's/default_mirror//' \
  | sed 's/[ \t]//g' \
  | sort \
  | uniq

Images vs Containers

• Docker Image: blueprint with app code and dependencies - static, read-only
• Docker Container: running instance of an image - dynamic, executable

Each instruction in the Dockerfile adds an extra layer to the Docker image. Minimize the number of layers by consolidating the instructions to increase the build’s performance and time.

Avoid using multiple RUN commands as it creates multiple cacheable layers which will affect the efficiency of the build process.

Use a single process per container: Each container should run a single process. This makes it easier to manage and monitor containers and helps to keep containers lightweight.

Images can exist without containers, whereas a container needs an image to run. We can create multiple containers from the same image, each with its own unique data and state

Docker commands

• Docker Run: It used for launching the containers from images, with specifying the runtime options and commands.
• Docker Pull: It fetches the container images from the container registry like Docker Hub to the local machine.
• Docker ps: It helps in displaying the running containers along with their important information like container ID, image used and status.
• Docker Stop: It helps in halting the running containers gracefully shutting down the processes within them.
• Docker Start: It helps in restarting the stopped containers, resuming their operations from the previous state.
• Docker Login: It helps to login in to the docker registry enabling the access to private repositories.

Docker network commands:

• docker network ls
• docker network inspect

Documentation

• Dockerfile reference - examples
• Dockerfile overview - with example
• Command Line Completion
• Base images - includes building from scratch
• CLI Cheat Sheet
• tutorial

First of all, the obligatory caveat from 2023: where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.

I do totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.

However, the combination of Docker in LXC is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application, particularly when an application is packaged as a Docker container, without recourse for building a native LXC container of the application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.

The key benefit is that devices such as one or more GPUs can be passed through to multiple LXC containers plus any nested docker containers. Otherwise, in the scenario where the GPU or PCIe device is passed through to a VM, as far as I know, it has to be dedicated to the VM. I've read that the devices can not be shared between a VM and LXC containers due to configuration differences between VM pass-through and LXC pass-through.

Given the caveat, I'll see if I can make this work. Not so easy. Trying to run

docker run --rm hello-world

Yields an error:

docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: 
mount source: "overlay", target: "/tmp/containerd-mount2030888385", 
fstype: overlay, flags: 0, 
data: "
  workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work,
  upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs,
  lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", 
  err: permission denied

With an associated apparmor error in Proxmox:

audit: type=1400 audit(1774803476.655:145): 
  apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 
  profile="lxc-131_" 
  name="/tmp/containerd-mount2030888385/" 
  pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"

The simple solution is to set nesting=1 in the proxmox lxc options.

The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:

> ps aux
root      41  0.0  0.0   2680  1808 ?    Ss   20:09   0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh

If so, this can be disabled:

systemctl disable ifupdown-wait-online.service

In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging

systemctl edit systemd-networkd-wait-online.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

In my case, I then saw something like:

root@frigate01:~# systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured

Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored
Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.

I have used a non-standard interface name. I resolved this by updating the edit with the following:

> systemctl edit systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60
#Environment=SYSTEMD_LOG_LEVEL=debug

The empty ExecStart line clears the original command parameters.

• How to Fix systemd-networkd-wait-online Service Timing Out During Boot
• systemd-networkd-wait-online.service(8) — Linux manual page

Some Docker commands:

docker run --rm -it hello-world bash

Docker installation is easy enough:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

This installs the latest stable release of:

• Docker CLI,
• Docker Engine,
• Docker Buildx,
• Docker Compose,
• containerd, and
• runc.

To get an idea of usage:

sh -c docker version

• ollama - framework for running models
• PyTorch Installation
• Hugging Face - models
• Open WebUI - extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It is built around universal standards, supporting Ollama and OpenAI-compatible Protocols (specifically Chat Completions).

How to Enable GPU Passthrough to LXC Containers in Proxmox indicates that the process of providing passthrough of a GPU to both an LXC container as well as a Virtual Machine is not possible as the two types of configurations conflict with each other.

As my own preference is to run whatever possible in LXC containers, I'll summarize the configuration I used, which is an amalgamation of configurations from several sites.

My current installation is ProxMox v9.1.6 with:

• ProArt Z890-CREATOR WIFI
• Intel(R) Core(TM) Ultra 9 285K
• Corsair CMP64GX5M2X6600C32 (128G 4400 MT/s) - ECC would have been nice
• NVIDIA Corporation AD103 [GeForce RTX 4070] (rev a1)

In BIOS/UEFI, enable these:

• VT-d / IOMMU
• Above 4G Decoding
• PCIe Native Power Management (if available)

Proxmox kernel parameters:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

update-grub
reboot

VFIO Binding - optional but recommended:

# /etc/modprobe.d/vfio.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Obtain Linux drivers from NVidia. The CUDA toolkit is not required. Only the drivers are required in ProxMox. Toolkits and add-ons are added within the container.

Install the drivers:

apt install build-essential
apt install pve-headers-$(uname -r)
sh NVIDIA-Linux-x86_64-595.58.03.run
# note, use the open kernel, rather than proprietary

Blacklist nouveau:

cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF

Test that the card is accessible:

nvidia-smi

Enable Data Persistence to prevent the GPU from re-initializing with each use.

nvidia-persistenced --persistence-mode
systemctl enable nvidia-persistenced

Then restart:

reboot

Identify the nvidia devices requiring passthrough:

root@host02:~# ls -al /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Mar 28 11:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Mar 28 11:58 /dev/nvidiactl
crw-rw-rw- 1 root root 505,   0 Mar 28 11:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 505,   1 Mar 28 11:58 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Mar 28 11:58 .
drwxr-xr-x 21 root root   5060 Mar 28 11:58 ..
cr--------  1 root root 508, 1 Mar 28 11:58 nvidia-cap1
cr--r--r--  1 root root 508, 2 Mar 28 11:58 nvidia-cap2

Note the numbers 195, 505 and 508 in this list (yours may be different).

Construct a container, and prior to starting, place the following into /etc/pve/lxc/<vmid>.conf (based upon the device listing above):

dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-modeset
dev3: /dev/nvidia-uvm
dev4: /dev/nvidia-uvm-tools
dev5: /dev/nvidia-caps/nvidia-cap1
dev6: /dev/nvidia-caps/nvidia-cap2

These lines are optional in the config file, one site talks about them by my container seems to work without them (they may be an old style cgroup2 style passthrough rather than the device oriented passthrough above):

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 505:* rwm
lxc.cgroup2.devices.allow: c 508:* rwm

Start the container and push the driver file into the container:

pct push <vmid> downloads/NVIDIA-Linux-x86_64-595.58.03.run /root/NVIDIA-Linux-x86_64-595.58.03.run

In the container, install the driver, minus the kernel module:

apt install kmod
sh NVIDIA-Linux-x86_64-595.58.03.run --no-kernel-modules

Run nvidia-smi in the container to confirm the card is reachable.

Add nvtop at the host or the container level to chart live GPU utllization:

apt install nvtop

Additional resources:

• Complete GPU passthrough guide for AI workloads, avoid the mistakes I made so you don't have to
• [TUTORIAL] NVIDIA drivers instalation Proxmox and CT
• How to Enable GPU Passthrough to LXC Containers in Proxmox - contains ollama startup examples with OpenWebUI

Another type of test to run when pytorch is installed:

python -c "import torch; print(torch.cuda.is_available())"

NOTE: for running in unprivileged container, some ideas in Plex GPU transcoding in Docker on LXC on Proxmox v2:

# if you're running in unprivileged mode, you also need to add permissions
# you either add the lines above, or the lines below -- not both
# gid/uid might need to be changed to suit your lxc-setup
dev0: /dev/nvidia0,gid=1000,uid=1000
dev1: /dev/nvidiactl,gid=1000,uid=1000
dev2: /dev/nvidia-modeset,gid=1000,uid=1000
dev3: /dev/nvidia-uvm,gid=1000,uid=1000
dev4: /dev/nvidia-uvm-tools,gid=1000,uid=1000
dev5: /dev/nvidia-caps/nvidia-cap1,gid=1000,uid=1000
dev6: /dev/nvidia-caps/nvidia-cap2,gid=1000,uid=1000
dev7: /dev/dri/card0,gid=1000,uid=1000
dev8: /dev/dri/renderD128,gid=1000,uid=1000

Some say this happens due to Wayland usage. My usage is Wayland/X11, not that it may make a difference. It is that way because I like that fact that the windows of an application get placed back to where they were when the application is re-opened.

The solution appears to be:

Tools -> Options -> General -> Enable 'Use LibreOffice dialogs'

In the Nanog email list, the following was posted as a summary of current tooling use for network management in Debian:

Linux has a bunch of different possible ways to administer all of this stuff.

• The most comprehensive CLI mechanism is iproute2 (the “ip” command and some related constructs).
• The most comprehensive and capable persistent configuration database mechanism is systemd-networkd.

Other persistent mechanisms include:

• Netplan (YAML based configurations that now days mostly get parsed into systemd-networkd files and then executed).
• Debian Traditional (the /etc/network/interfaces file and/or interfaces.d directory, ifup/ifdown/etc.). Lacks many features, but most can be worked around with iproute2 shell commands added to triggers in the file.
• Debian Traditional can be supplemented with ifupdown2 - ifupdown replacement from Cumulus Networks
• NetworkManager (semi-capable, but any capabilities it lacks are just hard to cope with).

My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done.

Overall, it’s a pretty comprehensive interface and provides good logs for troubleshooting in most circumstances.

When creating a Debian testing/forky LXC container on a Debian trixie machine, the following error may be encountered in the output:

I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://deb.debian.org/debian...
E: Couldn't find these debs: isc-dhcp-client
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian

This is a result of bug #1125011 in the Debian bug tracker.

There are several possible solutions:

• Manually apply the patches supplied by the Debian LXC team
• Probably might be solved by running lxc-create on a testing/forky machine, where the solution may have already been applied - I have not confirmed this
• Or it may work on a sid machine