Raymond P. Burkholder - Things I Do - Proxmox

LXC Fresh Container Construction From Scratch for Proxmox

nospam@example.com (Raymond P. Burkholder) — Fri, 27 Feb 2026 21:03:19 +0000

There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:

Build a linux based virtual machine, I use Debian's recent release
Install LXC and its template package
Construct and initialize an LXC container
Shut it down and and zip it up
Copy it over to the ProxMox template directory

The details:

# build the linux vm - details not relevant here
# ssh into the vm, or start a command line
# install basic packages

sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync

# basic container templates are in:
#   /usr/share/lxc/templates/ 
# for debian as well as other distributions

# create an lxc container, provide a list any additional packages

lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny

# start and attach to the container
lxc-start trixie-template
lxc-attach trixie-template

# prepare for generating template
apt clean
apt purge

# Remove SSH host keys to ensure unique keys for each clone:
rm /etc/ssh/ssh_host_*

# Empty the machine ID file:
truncate -s 0 /etc/machine-id

# clear history
unset HISTFILE
# truncate history
history -c
> ~/.bash_history
# the following has a space in front to prevent inclusion in the history
 shutdown -h now

# the shutdown returns to the virtual machine's prompt
# compress the directory structure

cd /var/lib/lxc/trixie-template/

# remove /dev files as they can't be created in an unprivileged container
# an example error message if not removed:
#   tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted
# construction of a new container will re-create the directory and files

rm ./rootfs/dev/ptmx
rm ./rootfs/dev/zero
rm ./rootfs/dev/tty3
rm ./rootfs/dev/urandom
rm ./rootfs/dev/null
rm ./rootfs/dev/tty
rm ./rootfs/dev/console
rm ./rootfs/dev/tty4
rm ./rootfs/dev/tty2
rm ./rootfs/dev/random
rm ./rootfs/dev/tty1
rm ./rootfs/dev/full

# cd into rootfs and zip the container

cd rootfs
tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./

# the xz file can be copied over to proxmox and placed into
# /var/lib/pve/local-btrfs/template/cache/
# for use as a template for container creation

During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.

If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:

lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less,python-minimal

An existing cache can be updated with something like:

sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64
apt-get update
apt-get dist-upgrade
apt-get clean
exit

courtesy of Updating lxc image/container caches

One other note, there are two package candidates for installing the ping utility:

iputils-ping - native Linux ping, preferred for Debian/Linux
inetutils-ping - general gnu version, used on a variety of posix sytstems, less preferred

Some fix-ups in the process:

apt-get install less
dpkg-reconfigure locales
useradd user

Docker Installation In LXC on ProxMox

nospam@example.com (Raymond P. Burkholder) — Sun, 29 Mar 2026 17:26:11 +0000

First of all, the obligatory caveat from 2023: where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.

I do totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.

However, the combination of Docker in LXC is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application, particularly when an application is packaged as a Docker container, without recourse for building a native LXC container of the application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.

The key benefit is that devices such as one or more GPUs can be passed through to multiple LXC containers plus any nested docker containers. Otherwise, in the scenario where the GPU or PCIe device is passed through to a VM, as far as I know, it has to be dedicated to the VM. I've read that the devices can not be shared between a VM and LXC containers due to configuration differences between VM pass-through and LXC pass-through.

Given the caveat, I'll see if I can make this work. Not so easy. Trying to run

docker run --rm hello-world

Yields an error:

docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: 
mount source: "overlay", target: "/tmp/containerd-mount2030888385", 
fstype: overlay, flags: 0, 
data: "
  workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work,
  upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs,
  lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", 
  err: permission denied

With an associated apparmor error in Proxmox:

audit: type=1400 audit(1774803476.655:145): 
  apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 
  profile="lxc-131_" 
  name="/tmp/containerd-mount2030888385/" 
  pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"

The simple solution is to set nesting=1 in the proxmox lxc options.

The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:

> ps aux
root      41  0.0  0.0   2680  1808 ?    Ss   20:09   0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh

If so, this can be disabled:

systemctl disable ifupdown-wait-online.service

In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging

systemctl edit systemd-networkd-wait-online.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

In my case, I then saw something like:

root@frigate01:~# systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured

Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored
Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.

I have used a non-standard interface name. I resolved this by updating the edit with the following:

> systemctl edit systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60
#Environment=SYSTEMD_LOG_LEVEL=debug

The empty ExecStart line clears the original command parameters.

Some Docker commands:

docker run --rm -it hello-world bash

NVidia GPU Passthrough to ProxMox LXC Container

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Mar 2026 17:26:45 +0000

How to Enable GPU Passthrough to LXC Containers in Proxmox indicates that the process of providing passthrough of a GPU to both an LXC container as well as a Virtual Machine is not possible as the two types of configurations conflict with each other.

As my own preference is to run whatever possible in LXC containers, I'll summarize the configuration I used, which is an amalgamation of configurations from several sites.

My current installation is ProxMox v9.1.6 with:

ProArt Z890-CREATOR WIFI
Intel(R) Core(TM) Ultra 9 285K
Corsair CMP64GX5M2X6600C32 (128G 4400 MT/s) - ECC would have been nice
NVIDIA Corporation AD103 [GeForce RTX 4070] (rev a1)

In BIOS/UEFI, enable these:

VT-d / IOMMU
Above 4G Decoding
PCIe Native Power Management (if available)

Proxmox kernel parameters:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

update-grub
reboot

VFIO Binding - optional but recommended:

# /etc/modprobe.d/vfio.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Obtain Linux drivers from NVidia. The CUDA toolkit is not required. Only the drivers are required in ProxMox. Toolkits and add-ons are added within the container.

Install the drivers:

apt install build-essential
apt install pve-headers-$(uname -r)
sh NVIDIA-Linux-x86_64-595.58.03.run
# note, use the open kernel, rather than proprietary

Blacklist nouveau:

cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF

Test that the card is accessible:

nvidia-smi

Enable Data Persistence to prevent the GPU from re-initializing with each use.

nvidia-persistenced --persistence-mode
systemctl enable nvidia-persistenced

Then restart:

reboot

Identify the nvidia devices requiring passthrough:

root@host02:~# ls -al /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Mar 28 11:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Mar 28 11:58 /dev/nvidiactl
crw-rw-rw- 1 root root 505,   0 Mar 28 11:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 505,   1 Mar 28 11:58 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Mar 28 11:58 .
drwxr-xr-x 21 root root   5060 Mar 28 11:58 ..
cr--------  1 root root 508, 1 Mar 28 11:58 nvidia-cap1
cr--r--r--  1 root root 508, 2 Mar 28 11:58 nvidia-cap2

Note the numbers 195, 505 and 508 in this list (yours may be different).

Construct a container, and prior to starting, place the following into /etc/pve/lxc/<vmid>.conf (based upon the device listing above):

dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-modeset
dev3: /dev/nvidia-uvm
dev4: /dev/nvidia-uvm-tools
dev5: /dev/nvidia-caps/nvidia-cap1
dev6: /dev/nvidia-caps/nvidia-cap2

These lines are optional in the config file, one site talks about them by my container seems to work without them (they may be an old style cgroup2 style passthrough rather than the device oriented passthrough above):

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 505:* rwm
lxc.cgroup2.devices.allow: c 508:* rwm

Start the container and push the driver file into the container:

pct push <vmid> downloads/NVIDIA-Linux-x86_64-595.58.03.run /root/NVIDIA-Linux-x86_64-595.58.03.run

In the container, install the driver, minus the kernel module:

apt install kmod
sh NVIDIA-Linux-x86_64-595.58.03.run --no-kernel-modules

Run nvidia-smi in the container to confirm the card is reachable.

Add nvtop at the host or the container level to chart live GPU utllization:

apt install nvtop

Additional resources:

Complete GPU passthrough guide for AI workloads, avoid the mistakes I made so you don't have to
[TUTORIAL] NVIDIA drivers instalation Proxmox and CT
How to Enable GPU Passthrough to LXC Containers in Proxmox - contains ollama startup examples with OpenWebUI

Another type of test to run when pytorch is installed:

python -c "import torch; print(torch.cuda.is_available())"

Opening ProxMox .vv files with virt-viewer (Debian & Firefox)

nospam@example.com (Raymond P. Burkholder) — Sat, 14 Mar 2026 22:54:52 +0000

If running Firefox on a Debian Linux machine, install virt-viewer:

sudo apt install virt-viewer

Ensure the VirtIO drivers and such have been installed in the virtual machine (if Windows) in order to provide SPICE services.

Then, in Firefox on your workstation:

go into about:config, and add the key 'network.protocol-handler.expose.virt-viewer' as boolean and set to true
go into about:preferences, and set "What should Firefox do with other files" to "Ask whether to open or save files".
in Proxmox, open a SPICE based console for a virtual machine, which attempts a download or a run of a customized .vv file,
Firefox will then request to open a Virt-Viewer file with Remote Viewer - at this point, you can set it as the default viewer, and it will show up in the application preferences

apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Feb 2026 23:51:54 +0000

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

I used the following for a trixie v13.3 version of a container:

lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,

Sample Proxmox command to build LXC container from Template

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Feb 2026 04:40:48 +0000

pct_id=101
pct_name=test01
pct create $pct_id /var/lib/pve/local-btrfs/template/cache/trixie-13-3-template.tar.xz  \
  -hostname $pct_name \
  -description 'demo build' \
  -onboot 1 \
  -startup up=3 \
  -ostype debian \
  -arch amd64 \
  -cores 2 \
  -memory 1024 \
  -nameserver 10.10.10.10 -searchdomain 'example.com' \
  -net0 name=vlan30,bridge=vmbr1,ip=dhcp,tag=30,type=veth \
  -rootfs local-btrfs:8,mountoptions="noatime;discard" \
  -swap 512