Raymond P. Burkholder - Things I Do

apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Feb 2026 23:51:54 +0000

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

I used the following for a trixie v13.3 version of a container:

lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,

Python Virtual Environment

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Feb 2026 19:27:47 +0000

Pip vs Pipenv: Which is better and which to learn first compares pipenv vs the python3-pip and virtualenv package combo.

After referring to that, I think I'll just stick with the standard pip/virtualenv combo for now.

To get started:

# install basic packages
apt-get install python3 python3-pip virtualenv  python3-venv git

# create a project directory - example ansible
python3 -m venv ansible

# activate the project
cd ansible
source bin/activate

# example installation of packages
pip install ansible
pip install argcomplete
activate-global-python-argcomplete
source ~/.bash_completion
ansible-config init --disabled > ansible.cfg

# to deactivate the project
deactivate

# upgrade
python3 -m pip install --upgrade ansible

At some point, integrate python3-ansible-runner, the github source and links to documentation.

LXC Fresh Container Construction From Scratch for Proxmox

nospam@example.com (Raymond P. Burkholder) — Fri, 27 Feb 2026 21:03:19 +0000

There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:

Build a linux based virtual machine, I use Debian's recent release
Install LXC and its template package
Construct and initialize an LXC container
Shut it down and and zip it up
Copy it over to the ProxMox template directory

The details:

# build the linux vm - details not relevant here
# ssh into the vm, or start a command line
# install basic packages

sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync

# basic container templates are in:
#   /usr/share/lxc/templates/ 
# for debian as well as other distributions

# create an lxc container, provide a list any additional packages

lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny

# start and attach to the container
lxc-start trixie-template
lxc-attach trixie-template

# prepare for generating template
apt clean
apt purge

# Remove SSH host keys to ensure unique keys for each clone:
rm /etc/ssh/ssh_host_*

# Empty the machine ID file:
truncate -s 0 /etc/machine-id

# clear history
unset HISTFILE
# truncate history
history -c
> ~/.bash_history
# the following has a space in front to prevent inclusion in the history
 shutdown -h now

# the shutdown returns to the virtual machine's prompt
# compress the directory structure

cd /var/lib/lxc/trixie-template/

# remove /dev files as they can't be created in an unprivileged container
# an example error message if not removed:
#   tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted
# construction of a new container will re-create the directory and files

rm ./rootfs/dev/ptmx
rm ./rootfs/dev/zero
rm ./rootfs/dev/tty3
rm ./rootfs/dev/urandom
rm ./rootfs/dev/null
rm ./rootfs/dev/tty
rm ./rootfs/dev/console
rm ./rootfs/dev/tty4
rm ./rootfs/dev/tty2
rm ./rootfs/dev/random
rm ./rootfs/dev/tty1
rm ./rootfs/dev/full

# cd into rootfs and zip the container

cd rootfs
tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./

# the xz file can be copied over to proxmox and placed into
# /var/lib/pve/local-btrfs/template/cache/
# for use as a template for container creation

During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.

If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:

lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less

An existing cache can be updated with something like:

sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64
apt-get update
apt-get dist-upgrade
apt-get clean
exit

courtesy of Updating lxc image/container caches

One other note, there are two package candidates for installing the ping utility:

iputils-ping - native Linux ping, preferred for Debian/Linux
inetutils-ping - general gnu version, used on a variety of posix sytstems, less preferred

Some fix-ups in the process:

apt-get install less
dpkg-reconfigure locales
useradd user

Sample Proxmox command to build LXC container from Template

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Feb 2026 04:40:48 +0000

pct_id=101
pct_name=test01
pct create $pct_id /var/lib/pve/local-btrfs/template/cache/trixie-13-3-template.tar.xz  \
  -hostname $pct_name \
  -description 'demo build' \
  -onboot 1 \
  -startup up=3 \
  -ostype debian \
  -arch amd64 \
  -cores 2 \
  -memory 1024 \
  -nameserver 10.10.10.10 -searchdomain 'example.com' \
  -net0 name=vlan30,bridge=vmbr1,ip=dhcp,tag=30,type=veth \
  -rootfs local-btrfs:8,mountoptions="noatime;discard" \
  -swap 512

Migrating LXC Containers From One Machine To Another

nospam@example.com (Raymond P. Burkholder) — Wed, 04 Apr 2018 15:39:24 +0000

For some machines with LXC containers, they have been running for a number of years. I want to take the easy way out and move the containers from one physical machine to another. At another time, I will rebuild the containers.

Since I am running BTRFS subvolumes for each container, I could be using BTRFS snapshot/send/receive commands to migrate/copy/replicate subvolumes. But before attempting that, I wanted to give the 'copy' a try. To do this properly, at the source, use the following -- with numeric-owner being a required paramenter -- command to collect the files:

tar --numeric-owner -czvf mycontainer.tar.gz /var/lib/lxc/my_container

At the destination, expand that file out:

tar --numeric-owner -xzvf mycontainer.tar.gz -C /var/lib/lxc/

The lxc users mailing list and Stack OverFlow were helpful.

Other stuff to do:

Read up on CGroups in the Linux Programmer's Manual

In migrating from a very old version of LXC to a much newer version of LXC, I was getting errors. I needed to run a some debug to get a handle on errors:

lxc-start -n container -F --logpriority=DEBUG --logfile log

I had errors along the lines of:

Activating lvm and md swap...done.
Checking file systems...Segmentation fault (core dumped)
failed (code 139).

ServerFault had the solution: put "vsyscall=emulate" into /etc/default/grub, run 'update-grub' and reboot. Looks like I need to modernize my containers so I can eliminate this workaround, which may have some security considerations. There is a Debian Bug for this.

einstein home has a blog with some kernel references to the issue, in effect saying: "vsyscall is now disabled on latest linux distros". A lengthier LWN article at On vsyscalls and the vDSO. This works with kernel 4.14, my current version, but I see somewhere else that the workaround is entirely removed in kernel 4.15, at least in the Arch world. At bug 847154: "This breaks (e)glibc 2.13 and earlier".

Note, see newer notes at LXC Fresh Container Construction From Scratch for Proxmox.

Debian Headers first then Kernel for DKMS rebuilds

nospam@example.com (Raymond P. Burkholder) — Sat, 17 Jan 2026 03:39:34 +0000

Something from a Debian mailing list:

I found the root cause, when testing 6.12.57 I installed the image then the headers and the NVIDIA DKMS module was not rebuilt because the matching linux- headers package was not installed at the time the kernel image was configured.
If I install the headers first and then the linux-image package, DKMS correctly builds the NVIDIA module and 6.12.63 works fine, so it doesn't look like a kernel regression after all.
I don't know if I should manually run dkms autoinstall myself after a kernel update (I never had to before) or if there was a bug during the install process of this update.

Makes sense, I had NVidia compile fail in a similar. This makes it obvious what I should have observed.