Raymond P. Burkholder - Things I Do - Linux

LXC Fresh Container Construction From Scratch for Proxmox

nospam@example.com (Raymond P. Burkholder) — Fri, 27 Feb 2026 21:03:19 +0000

There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:

Build a linux based virtual machine, I use Debian's recent release
Install LXC and its template package
Construct and initialize an LXC container
Shut it down and and zip it up
Copy it over to the ProxMox template directory

The details:

# build the linux vm - details not relevant here
# ssh into the vm, or start a command line
# install basic packages

sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync

# basic container templates are in:
#   /usr/share/lxc/templates/ 
# for debian as well as other distributions

# create an lxc container, provide a list any additional packages

lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny

# start and attach to the container
lxc-start trixie-template
lxc-attach trixie-template

# prepare for generating template
apt clean
apt purge

# Remove SSH host keys to ensure unique keys for each clone:
rm /etc/ssh/ssh_host_*

# Empty the machine ID file:
truncate -s 0 /etc/machine-id

# clear history
unset HISTFILE
# truncate history
history -c
> ~/.bash_history
# the following has a space in front to prevent inclusion in the history
 shutdown -h now

# the shutdown returns to the virtual machine's prompt
# compress the directory structure

cd /var/lib/lxc/trixie-template/

# remove /dev files as they can't be created in an unprivileged container
# an example error message if not removed:
#   tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted
# construction of a new container will re-create the directory and files

rm ./rootfs/dev/ptmx
rm ./rootfs/dev/zero
rm ./rootfs/dev/tty3
rm ./rootfs/dev/urandom
rm ./rootfs/dev/null
rm ./rootfs/dev/tty
rm ./rootfs/dev/console
rm ./rootfs/dev/tty4
rm ./rootfs/dev/tty2
rm ./rootfs/dev/random
rm ./rootfs/dev/tty1
rm ./rootfs/dev/full

# cd into rootfs and zip the container

cd rootfs
tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./

# the xz file can be copied over to proxmox and placed into
# /var/lib/pve/local-btrfs/template/cache/
# for use as a template for container creation

During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.

If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:

lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less,python-minimal

An existing cache can be updated with something like:

sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64
apt-get update
apt-get dist-upgrade
apt-get clean
exit

courtesy of Updating lxc image/container caches

One other note, there are two package candidates for installing the ping utility:

iputils-ping - native Linux ping, preferred for Debian/Linux
inetutils-ping - general gnu version, used on a variety of posix sytstems, less preferred

Some fix-ups in the process:

apt-get install less
dpkg-reconfigure locales
useradd user

Image Building Notes - debootstrap

nospam@example.com (Raymond P. Burkholder) — Sun, 05 Apr 2026 17:31:32 +0000

APT based distributions like Debian can be containerized with a tool called debootstrap. It is part of the image build process of lxc-create. It is also referenced in Docker Base Images for building an image from scratch.

When looking at the build scripts included in the package installation, repositories for the following distributions can be found in /usr/share/debootstrap/scripts:

debian - universal operating system
trisquel - a distribution of the GNU operating system, with the kernel GNU Linux-libre
ubuntu - modern enterprise open source
pardus - Turkish
kali - open-source, Debian-based Linux distribution geared towards various information security tasks, such as Penetration Testing, Security Research, Computer Forensics and Reverse Engineering
elxr - Enterprise-Grade Linux for Edge-to-Cloud Deployments
pureos - fully-convergent, user friendly, secure and freedom respecting OS for your daily usage

The wiki shows a simple two liner to get the basics of the distribution in place (as root):

mkdir trixie-chroot
debootstrap stable trixie-chroot http://deb.debian.org/debian/

Enter the chroot and note new root:

root@test:~# pwd
/root
root@test:~# chroot trixie/
root@test:/# pwd
/
root@test:/# exit
exit
root@test:~# pwd
/root

After the debootstrap, create the base for docker, and give it a try:

tar -C trixie-chroot -c . | docker import - trixie
docker run trixie cat /etc/debian_version
docker run --rm -i -t trixie /bin/bash

Although debootstrap can be used to build an image for a version subsequent, it is generally recommended to use debootstrap from at least the desired version to ensure it has the proper updates and dependencies.

Command line to summarize the referenced repositories:

grep -h default_mirror /usr/share/debootstrap/scripts/* \
  | sed 's/default_mirror//' \
  | sed 's/[ \t]//g' \
  | sort \
  | uniq

Debian Linux Network Configuration Tools

nospam@example.com (Raymond P. Burkholder) — Sun, 22 Mar 2026 15:06:04 +0000

In the Nanog email list, the following was posted as a summary of current tooling use for network management in Debian:

Linux has a bunch of different possible ways to administer all of this stuff.

The most comprehensive CLI mechanism is iproute2 (the “ip” command and some related constructs).
The most comprehensive and capable persistent configuration database mechanism is systemd-networkd.

Other persistent mechanisms include:

Netplan (YAML based configurations that now days mostly get parsed into systemd-networkd files and then executed).
Debian Traditional (the /etc/network/interfaces file and/or interfaces.d directory, ifup/ifdown/etc.). Lacks many features, but most can be worked around with iproute2 shell commands added to triggers in the file.
Debian Traditional can be supplemented with ifupdown2 - ifupdown replacement from Cumulus Networks
NetworkManager (semi-capable, but any capabilities it lacks are just hard to cope with).

My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done.
Overall, it’s a pretty comprehensive interface and provides good logs for troubleshooting in most circumstances.

lxc-create of debian testing on debian trixie

nospam@example.com (Raymond P. Burkholder) — Tue, 17 Mar 2026 03:10:34 +0000

When creating a Debian testing/forky LXC container on a Debian trixie machine, the following error may be encountered in the output:

I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://deb.debian.org/debian...
E: Couldn't find these debs: isc-dhcp-client
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian

This is a result of bug #1125011 in the Debian bug tracker.

There are several possible solutions:

Manually apply the patches supplied by the Debian LXC team
Probably might be solved by running lxc-create on a testing/forky machine, where the solution may have already been applied - I have not confirmed this
Or it may work on a sid machine

apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13

nospam@example.com (Raymond P. Burkholder) — Sat, 28 Feb 2026 23:51:54 +0000

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

I used the following for a trixie v13.3 version of a container:

lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,

Debian Headers first then Kernel for DKMS rebuilds

nospam@example.com (Raymond P. Burkholder) — Sat, 17 Jan 2026 03:39:34 +0000

Something from a Debian mailing list:

I found the root cause, when testing 6.12.57 I installed the image then the headers and the NVIDIA DKMS module was not rebuilt because the matching linux- headers package was not installed at the time the kernel image was configured.
If I install the headers first and then the linux-image package, DKMS correctly builds the NVIDIA module and 6.12.63 works fine, so it doesn't look like a kernel regression after all.
I don't know if I should manually run dkms autoinstall myself after a kernel update (I never had to before) or if there was a bug during the install process of this update.

Makes sense, I had NVidia compile fail in a similar. This makes it obvious what I should have observed.