Raymond P. Burkholder - Things I Do

I have an APC Smart UPS 1500 connected via USB cable to a BeagleBone Green running NUT 2.8.1-5 on Linux 6.18.36-bone40. Even on earlier Linux kernels and versions of NUT, during some sort of UPS transition, I would start to get a log full of the these error messages:

usbhid-ups: nut_libusb_get_report: No such device (it may have been disconnected)

In detail:

kernel: musb-hdrc musb-hdrc.1: Babble
kernel: usb 1-1: USB disconnect, device number 15
kernel: usb 1-1.4: USB disconnect, device number 16
kernel: usb 1-1: new high-speed USB device number 17 using musb-hdrc
kernel: usb 1-1: New USB device found, idVendor=1a40, idProduct=0201, bcdDevice= 1.00
kernel: usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
kernel: usb 1-1: Product: USB 2.0 Hub [MTT]
kernel: hub 1-1:1.0: USB hub found
kernel: hub 1-1:1.0: 7 ports detected
kernel: usb 1-1.4: new full-speed USB device number 18 using musb-hdrc
kernel: usb 1-1.4: New USB device found, idVendor=051d, idProduct=0003, bcdDevice= 0.01
kernel: usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
kernel: usb 1-1.4: Product: Smart-UPS_1500 FW:UPS 06.0 / ID=1028
kernel: usb 1-1.4: Manufacturer: American Power Conversion
kernel: usb 1-1.4: SerialNumber: unique_number
kernel: hid-generic 0003:051D:0003.0008: hiddev0,hidraw0: USB HID v1.11 Device [American Power Conversion  Smart-UPS_1500 FW:UPS 06.0 / ID=1028] on usb-musb-hdrc.1-1.4/input0
usbhid-ups[31894]: nut_libusb_get_report: No such device (it may have been disconnected)
usbhid-ups[31894]: nut_libusb_get_report: No such device (it may have been disconnected)

For some reason, USB devices 'disconnect' and reconnect. Then the device reconnects, a new DeviceID is assigned to the USB device. This breaks the usbhid-ups drivers supplied by NUT the packages. That is, usbhid-ups no longer sees the APC USB device.

One forum article suggested that there is not enough power in the BeagleBone system to propery power a USB connection. The suggested solution is to add a powered USB hub. I gave that a try with no success in eliminating the problem.

To resolve this, I'm not sure if recent versions of the driver fix this, but I used Debian's udev system to sense the device id change and to restart the NUT USB driver.

In /home/debian, which is the default home directory, I created two files and ran 'chmod +x' on each to make them executable. When a USB 'ADD' state is detected, nut.target is restarted to initiate the detection and connection process. When a removal is detected, the service is stopped. The stop does not seem to be reliable, but the restart appears to be reliable enough to get things started again.

$ cat usb_add.sh
#!/usr/bin/env bash

logger "USB state change detected: $1 $2 $3"
echo "$(date) USB state change detected: $1 $2 $3" >> /home/debian/usb-log.txt
systemctl restart nut.target
exit 0

$ cat usb_remove.sh
#!/usr/bin/env bash

logger "USB state change detected: $1 $2"
echo "$(date) USB state change detected: $1 $2" >> /home/debian/usb-log.txt
#systemctl stop nut.target
exit 0

To determine useful attributes I used the following (the first is a shorter result than the second):

udevadm info --name=/dev/bus/usb/001/017 
udevadm info --name=/dev/bus/usb/001/017 --attribute-walk

The rule which invokes these scripts is:

# cat  /etc/udev/rules.d/89-usb-ups-change.rules
SUBSYSTEM=="usb", MODE="0660", ACTION=="add",    ENV{ID_VENDOR_ID}=="051d", ENV{ID_MODEL_ID}=="0003", RUN+="/home/debian/usb_add.sh add '$env{DEVNAME}' '$env{ID_USB_SERIAL}'"
SUBSYSTEM=="usb", MODE="0660", ACTION=="remove", RUN+="/home/debian/usb_remove.sh remove $env{DEVNAME}"

These command variations were suggested to simulate a cable re-connect but did not reliably work. I resorted to physically disconnecting and reconnecting the cable to perform a proper test for tuning and production (using the major:minor usb device manufacturer code):

# major:minor usb device manufacturer code
usbreset 051d:0003
# dev bus id
udevadm test /dev/bus/usb/001/007

Each rule change test required the following to reload the udev rules:

udevadm control --reload-rules

To see what Nut sees:

# nut-scanner -U
Scanning USB bus.
[nutdev1]
        driver = "apc_modbus"
        port = "auto"
        vendorid = "051D"
        productid = "0003"
        product = "Smart-UPS_1500 FW:UPS 06.0 / ID=1028"
        serial = "unique id"
        vendor = "American Power Conversion"
        bus = "001"
        device = "017"
        busport = "002"
        ###NOTMATCHED-YET###bcdDevice = "0001"

A linux command provides a concise device and manufacturer id:

# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 016: ID 050d:0234 Belkin Components F5U234 USB 2.0 4-Port Hub
Bus 001 Device 017: ID 051d:0003 American Power Conversion UPS

Other troubleshooting commands include:

udevadm monitor

This solution for those who might encounter Update 2.7.4 to 2.8.4 CP UPS driver disconnect (nut_libusb_get_report: No such device)

• Scaling Akvorado BMP RIB with sharding
• Command line Norse God of Wind Hræsvelg move the clouds
• Fabric is an open-source framework for augmenting humans using AI. It provides a modular system for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere.
• Secure Boot and Microsoft CA Rollover - a heads-up for distributions
• Containers Are a Security Boundary (some assembly required)
• Armadillo - C++ library for linear algebra & scientific computing

Install tool:

sudo apt install xclip

Perform the copy:

cat filename.base64 | xclip -rmlastnl -selection clipboard

I had someone Create an offline installation of SQL Server Management Studio for me. It was a fun install. Sarcasm intended. The installer could be so much smaller I think if it didn't have so many language packs in one bundle. Why not, if building an offline installer, why not allow a language selection right there.

The first step was to learn to start the installer from the command line with:

SSMS_22.2.1\vs_SSMS_22.2.1.exe --noWeb

At this point, it would unpack files and then silently quit. No messages at the command line.

The logs indicate an invalid certificate:

Certificate is invalid: C:\temp\SSMS_22.2.1\vs_installer.opc
Error 0x80131509: Signature verification failed. 
Error: Unable to verify the integrity of the installation files: the certificate could not be verified.

Three of the certificates can be installed with:

certutil.exe -addstore -f "Root" "SSMS_22.2.1\certificates\manifestCounterSignRootCertificate.cer"
certutil.exe -addstore -f "Root" "SSMS_22.2.1\certificates\manifestRootCertificate.cer.cer"
certutil.exe -addstore -f "Root" "SSMS_22.2.1\certificates\vs_installer_opc.RootCertificate.cer"

By looking at the logs, there is a fourth certificate, but missing, so needs to be copied and installed in a similar manner from Microsoft Windows Code Signing PCA 2024.crt (github) or Microsoft Windows Code Signing PCA 2024.crt (microsoft.com)

The final hurdle has to Microsoft ODBC Driver installation. It installs Microsoft ODBC Driver 18 for SQL Server ok. But it fails while installing Microsoft OLEDB Driver 19 for SQL Server with a message along the lines of:

This application requires 
Microsoft Visual C++ Redistributable for Visual Studio 2022 
(x64/x86, version 14.38 at minimum)

It fails even when installing Latest supported redistributable version x64.

The secret here is that the x86 version also has to be downloaded and installed. So you should have both installers:

VC_redist.x64.exe
VC_redist.x86.exe

As a final note, the installer logs are in C:\Users\<UserName>\AppData\Local\Temp and are named dd_setup_yyyymmddhhmm_errors.log.

On installation, the Citrix Workspace installer will ask for libwebkit2gtk-4.0-37. However, in the latest Debian, libwebkit2gtk-4.1-0 is installed. That package brings in many other packages so there is no practical way to install an earlier version.

The solution to this give the older Debian Bookworm a try where the package is available for installation.

> apt-cache search libwebkit2gtk
libwebkit2gtk-4.0-37 - Web content engine library for GTK
libwebkit2gtk-4.0-dev - Web content engine library for GTK - development files
libwebkit2gtk-4.0-doc - Web content engine library for GTK - documentation
libwebkit2gtk-4.1-0 - Web content engine library for GTK
libwebkit2gtk-4.1-dev - Web content engine library for GTK - development files

To prep for Citrix installation, install the following:

# apt install libwebkit2gtk-4.0-37 libcurl4 libspeexdsp1

Once all the GPU and Docker prerequisites are in place, installing frigate-nve is easy: a) build a yaml file, b) docker compose the file.

In the LXC container, create a frigate directory and move into it:

mkdir frigate
cd frigate

In that directory, here is the content of my docker-compose-yaml file:

services:
  frigate:
    container_name: frigate
    #privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    stop_grace_period: 30s # allow enough time to shut down the various services
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "1512mb" # updated for my camera list based upon error messages in the log
#    devices:
#      - /dev/dri/renderD128:/dev/dri/renderD128 # AMD / Intel GPU, needs to be updated for your hardware
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              #device_ids: ['0'] # this is only needed when using multiple GPUs
              count: 1 # number of GPUs
              capabilities: [gpu]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /root/frigate/config:/config
      - /root/frigate/storage:/media/frigate
      - type: tmpfs # 1GB In-memory filesystem for recording segment storage
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      # - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "xxxxxxx"

Get it started:

# in a foreground process:
docker compose up
# or, as a background process:
docker compose up -d frigate

The login should be seen at something like (or your container's address):

https://127.0.0.1:8971/

Official installation documentation:

• Frigage NVR Installation

From Plex GPU transcoding in Docker on LXC on Proxmox v2:

# make sure that all nvidia devices are loaded upon boot
cat >/etc/systemd/system/nvidia-pre-lxc-init.service <<'EOF'
[Unit]
Description=Initialize NVIDIA devices early (before Proxmox guests)
After=systemd-modules-load.service
Before=pve-guests.service
Wants=pve-guests.service

[Service]
Type=oneshot
RemainAfterExit=yes

ExecStartPre=-/sbin/modprobe nvidia
ExecStartPre=-/sbin/modprobe nvidia_uvm
ExecStart=-/usr/bin/nvidia-smi -L
ExecStart=-/usr/bin/nvidia-modprobe -u -c=0

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable nvidia-pre-lxc-init.service

reboot then try nvidia-smi to confirm function.

• Qwen 3.5 9B - general purpose locally run LLM
• OctoPrint - web-based 3D printer control software that allows you to remotely control and monitor your 3D printer from a web interface. It was designed to be compatible with a wide range of 3D printers.
• Trilium Notes - self-hosted note-taking and personal knowledge management application. It enables users to organize information in a hierarchical tree structure and supports rich text editing, internal linking, images, attachments, and powerful scripting capabilities. This version reflects the most current development efforts under the TriliumNext organization and replaces all prior forks or legacy variants. Trilium is ideal for building personal wikis, structured documentation, and long-term knowledge archives, giving users full local control and privacy.
• Vaultwarden - self-hosted password manager which provides secure and encrypted password storage. It uses client-side encryption and provides access to passwords through a web interface and mobile apps.
• NetBox - the source of truth for everything on your network, from physical components like power systems and cabling to virtual assets like IP addresses and VLANs. Network automation and observability tools depend on NetBox’s authoritative data to roll out configurations, monitor changes, and accelerate operations across the enterprise

Not having the desire to put frigate on a standalone server of some sort, I wanted it to run as a container in Proxmox. Since Frigate is supplied as a Docker container, and I don't want to run Docker directly on the Proxmox host, there are two choices open to me:

• run Frigate in a virtual machine - prevents sharing an Nvidia card with other machines or containers, or
• run Frigate in an LXC or an OCI container - an Nvidia card can be shared with multiple containers, and resource requirements are reduced, at the risk of security issues (which are reduced if the host is a private server)

I'm running on Proxmox 9.1.9 with kernel SMP PREEMPT_DYNAMIC PMX 6.17.13-4 (2026-04-21T22:03Z) x86_64 GNU/Linux

This installation is a privileged container as I had troubles with non-privileged. I'll have to come back to this based upon: Running in rootless LXC, and maybe require the user id mapping from Unprivileged LXC containers.

Some installation commands, based upon connecting Frigate to an Nvidia GPU. Some instructions come from Installing the NVIDIA Container Toolkit, which is required for connecting to the Nvidia GPU (commands assume sudo or running as root, which is default in a basic LXC container). The link contains installation, testing and troubleshooting instructions. Continue reading "Prepping nvidia drivers for Frigate Running in..." »

Taking RTL_433 in unprivileged LXC container, with automatic device path change for inspiration, I was able to connect multiple APC and Eaton UPS USB cables to a server with the following mechanism. I forwarded the USB connections into an LXC container as I use my nut2mqtt - Communication between Network UPS Tools (NUT) and MQTT for monitoring.

The Proxmox host is configured in a standard way using the primary UPS USB connection (still yet to try).

As a USB cable is inserted, "journalctl -f" will log information similar to:

Apr 25 15:48:21 host02 kernel: usb 3-4: new full-speed USB device number 10 using xhci_hcd
Apr 25 15:48:21 host02 kernel: usb 3-4: New USB device found, idVendor=051d, idProduct=0002, bcdDevice= 0.90
Apr 25 15:48:21 host02 kernel: usb 3-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Apr 25 15:48:21 host02 kernel: usb 3-4: Product: Back-UPS XS 1300G FW:864.L6 .D USB FW:L6
Apr 25 15:48:21 host02 kernel: usb 3-4: Manufacturer: American Power Conversion
Apr 25 15:48:21 host02 kernel: usb 3-4: SerialNumber: 4B12P636
Apr 25 15:48:21 host02 kernel: hid-generic 0003:051D:0002.0007: hiddev3,hidraw5: ......

Use "lsusb" to determine the bus and device assignment based upon the 'idVendor' and 'idProduct' assignments above:

# lsusb -d 051d:0002
Bus 003 Device 010: ID 051d:0002 American Power Conversion Uninterruptible Power Supply

Thus you'll see an associated entry in /dev:

~# ls -altr /dev/bus/usb/003/010
crw-rw-rw- 1 root root 189, 265 Apr 25 15:48 /dev/bus/usb/003/010

Note the cgroup2 '189, 265' identifiers. Adding an entry to your /etc/pve/lxc/<lxc-id>.conf entry will be required:

lxc.cgroup2.devices.allow: c 189:* rwm

Whenever the USB cable is plugged in, a different device may be constructed. To pass that information to Proxmox for when it starts the container, use something like:

# cat /etc/udev/rules.d/50-ups-usb.rules
SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="051d", ENV{ID_MODEL_ID}=="0002", ENV{ID_SERIAL_SHORT}=="4B12P636", MODE="0666", RUN="/usr/sbin/pct set <lxc-id> --dev0 mode=0666,path=$devnode"

If the file is changed, it can be processed with:

# reload rules and rerun
udevadm control --reload-rules
udevadm trigger
# optional cable test simulation
udevadm test /dev/bus/usb/003/010

The ENV{ID_VENDOR_ID}, ENV{ID_MODEL_ID} & ENV{ID_SERIAL_SHORT} variable names and content can be confirmed with:

# udevadm info  /dev/bus/usb/003/010
P: /devices/pci0000:80/0000:80:14.0/usb3/3-4
M: 3-4
R: 4
J: c189:265
U: usb
T: usb_device
D: c 189:265
N: bus/usb/003/010
L: 0
V: usb
E: DEVPATH=/devices/pci0000:80/0000:80:14.0/usb3/3-4
E: DEVNAME=/dev/bus/usb/003/010
E: DEVTYPE=usb_device
E: DRIVER=usb
E: PRODUCT=51d/2/90
E: TYPE=0/0/0
E: BUSNUM=003
E: DEVNUM=010
E: MAJOR=189
E: MINOR=265
E: SUBSYSTEM=usb
E: USEC_INITIALIZED=2435707957355
E: ID_BUS=usb
E: ID_MODEL=Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6
E: ID_MODEL_ENC=Back-UPS\x20XS\x201300G\x20FW:864.L6\x20.D\x20USB\x20FW:L6\x20
E: ID_MODEL_ID=0002
E: ID_SERIAL=American_Power_Conversion_Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6_4B1233P63346
E: ID_SERIAL_SHORT=4B12P636
E: ID_VENDOR=American_Power_Conversion
E: ID_VENDOR_ENC=American\x20Power\x20Conversion
E: ID_VENDOR_ID=051d
E: ID_REVISION=0090
E: ID_USB_MODEL=Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6
E: ID_USB_MODEL_ENC=Back-UPS\x20XS\x201300G\x20FW:864.L6\x20.D\x20USB\x20FW:L6\x20
E: ID_USB_MODEL_ID=0002
E: ID_USB_SERIAL=American_Power_Conversion_Back-UPS_XS_1300G_FW:864.L6_.D_USB_FW:L6_4B1233P63346
E: ID_USB_SERIAL_SHORT=4B12P636
E: ID_USB_VENDOR=American_Power_Conversion
E: ID_USB_VENDOR_ENC=American\x20Power\x20Conversion
E: ID_USB_VENDOR_ID=051d
E: ID_USB_REVISION=0090
E: ID_USB_INTERFACES=:030000:
E: ID_VENDOR_FROM_DATABASE=American Power Conversion
E: ID_MODEL_FROM_DATABASE=Uninterruptible Power Supply
E: ID_PATH_WITH_USB_REVISION=pci-0000:80:14.0-usbv2-0:4
E: ID_PATH=pci-0000:80:14.0-usb-0:4
E: ID_PATH_TAG=pci-0000_80_14_0-usb-0_4
E: ID_FOR_SEAT=usb-pci-0000_80_14_0-usb-0_4
E: TAGS=:seat:
E: CURRENT_TAGS=:seat:

Resources:

• udev(7) — Linux manual page - Dynamic device management
• Persistent names for USB-serial devices in Linux (/dev/ttyUSBx -> /dev/custom-name) - I first looked at doing symlinks, but realized it is not required if I can perform a 'pct set' directly on the container configuration.

Note to self: come back to this and redo some non-functional audio logic located in some code which needs to be uprooted and re-juiced.

JUCE - open source, cross-platform, software development framework provided as C++ source code, that can be used to create standalone software on Windows, macOS, Linux, iOS and Android, as well as VST, VST3, AU, AUv3, AAX and LV2 plug-ins.

The Git Commands I Run Before Reading Any Code - some interesting simple queries on a git repository to gain some understanding on development patterns

Most changed files in the last year:

git log --format=format: --name-only --since="1 year ago" | sort | uniq -c | sort -nr | head -20

List of contributors:

git shortlog -sn --no-merges

Example of looking for hot words in comment messages:

git log -i -E --grep="fix|bug|broken" --name-only --format='' | sort | uniq -c | sort -nr | head -20

Time line of commits:

git log --format='%ad' --date=format:'%Y-%m' | sort | uniq -c

Another example of comment keyword searches:

git log --oneline --since="1 year ago" | grep -iE 'revert|hotfix|emergency|rollback'

Hacker News - referenced

APT based distributions like Debian can be containerized with a tool called debootstrap. It is part of the image build process of lxc-create. It is also referenced in Docker Base Images for building an image from scratch.

When looking at the build scripts included in the package installation, repositories for the following distributions can be found in /usr/share/debootstrap/scripts:

• debian - universal operating system
• trisquel - a distribution of the GNU operating system, with the kernel GNU Linux-libre
• ubuntu - modern enterprise open source
• pardus - Turkish
• kali - open-source, Debian-based Linux distribution geared towards various information security tasks, such as Penetration Testing, Security Research, Computer Forensics and Reverse Engineering
• elxr - Enterprise-Grade Linux for Edge-to-Cloud Deployments
• pureos - fully-convergent, user friendly, secure and freedom respecting OS for your daily usage

The wiki shows a simple two liner to get the basics of the distribution in place (as root):

mkdir trixie-chroot
debootstrap stable trixie-chroot http://deb.debian.org/debian/

Enter the chroot and note new root:

root@test:~# pwd
/root
root@test:~# chroot trixie/
root@test:/# pwd
/
root@test:/# exit
exit
root@test:~# pwd
/root

After the debootstrap, create the base for docker, and give it a try:

tar -C trixie-chroot -c . | docker import - trixie
docker run trixie cat /etc/debian_version
docker run --rm -i -t trixie /bin/bash

Although debootstrap can be used to build an image for a version subsequent, it is generally recommended to use debootstrap from at least the desired version to ensure it has the proper updates and dependencies.

Command line to summarize the referenced repositories:

grep -h default_mirror /usr/share/debootstrap/scripts/* \
  | sed 's/default_mirror//' \
  | sed 's/[ \t]//g' \
  | sort \
  | uniq

Images vs Containers

• Docker Image: blueprint with app code and dependencies - static, read-only
• Docker Container: running instance of an image - dynamic, executable

Each instruction in the Dockerfile adds an extra layer to the Docker image. Minimize the number of layers by consolidating the instructions to increase the build’s performance and time.

Avoid using multiple RUN commands as it creates multiple cacheable layers which will affect the efficiency of the build process.

Use a single process per container: Each container should run a single process. This makes it easier to manage and monitor containers and helps to keep containers lightweight.

Images can exist without containers, whereas a container needs an image to run. We can create multiple containers from the same image, each with its own unique data and state

Docker commands

• Docker Run: It used for launching the containers from images, with specifying the runtime options and commands.
• Docker Pull: It fetches the container images from the container registry like Docker Hub to the local machine.
• Docker ps: It helps in displaying the running containers along with their important information like container ID, image used and status.
• Docker Stop: It helps in halting the running containers gracefully shutting down the processes within them.
• Docker Start: It helps in restarting the stopped containers, resuming their operations from the previous state.
• Docker Login: It helps to login in to the docker registry enabling the access to private repositories.

Docker network commands:

• docker network ls
• docker network inspect

Docker image commands:

• docker image ls -a
• docker container ls -a

Documentation

• Dockerfile reference - examples
• Dockerfile overview - with example
• Command Line Completion
• Base images - includes building from scratch
• CLI Cheat Sheet
• tutorial

First of all, the obligatory caveat from 2023: where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.

I do totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.

However, the combination of Docker in LXC is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application, particularly when an application is packaged as a Docker container, without recourse for building a native LXC container of the application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.

The key benefit is that devices such as one or more GPUs can be passed through to multiple LXC containers plus any nested docker containers. Otherwise, in the scenario where the GPU or PCIe device is passed through to a VM, as far as I know, it has to be dedicated to the VM. I've read that the devices can not be shared between a VM and LXC containers due to configuration differences between VM pass-through and LXC pass-through.

Given the caveat, I'll see if I can make this work. Not so easy. Trying to run

docker run --rm hello-world

Yields an error:

docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: 
mount source: "overlay", target: "/tmp/containerd-mount2030888385", 
fstype: overlay, flags: 0, 
data: "
  workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work,
  upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs,
  lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", 
  err: permission denied

With an associated apparmor error in Proxmox:

audit: type=1400 audit(1774803476.655:145): 
  apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 
  profile="lxc-131_" 
  name="/tmp/containerd-mount2030888385/" 
  pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"

The simple solution is to set nesting=1 in the proxmox lxc options.

The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:

> ps aux
root      41  0.0  0.0   2680  1808 ?    Ss   20:09   0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh

If so, this can be disabled:

systemctl disable ifupdown-wait-online.service

In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging

systemctl edit systemd-networkd-wait-online.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

In my case, I then saw something like:

root@frigate01:~# systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured

Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored
Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.

I have used a non-standard interface name. I resolved this by updating the edit with the following:

> systemctl edit systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60
#Environment=SYSTEMD_LOG_LEVEL=debug

The empty ExecStart line clears the original command parameters.

• How to Fix systemd-networkd-wait-online Service Timing Out During Boot
• systemd-networkd-wait-online.service(8) — Linux manual page

Some Docker commands:

docker run --rm -it hello-world bash