There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:
The details:
# build the linux vm - details not relevant here # ssh into the vm, or start a command line # install basic packages sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync # basic container templates are in: # /usr/share/lxc/templates/ # for debian as well as other distributions # create an lxc container, provide a list any additional packages lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny # start and attach to the container lxc-start trixie-template lxc-attach trixie-template # prepare for generating template apt clean apt purge # Remove SSH host keys to ensure unique keys for each clone: rm /etc/ssh/ssh_host_* # Empty the machine ID file: truncate -s 0 /etc/machine-id # clear history unset HISTFILE # truncate history history -c > ~/.bash_history # the following has a space in front to prevent inclusion in the history shutdown -h now # the shutdown returns to the virtual machine's prompt # compress the directory structure cd /var/lib/lxc/trixie-template/ # remove /dev files as they can't be created in an unprivileged container # an example error message if not removed: # tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted # construction of a new container will re-create the directory and files rm ./rootfs/dev/ptmx rm ./rootfs/dev/zero rm ./rootfs/dev/tty3 rm ./rootfs/dev/urandom rm ./rootfs/dev/null rm ./rootfs/dev/tty rm ./rootfs/dev/console rm ./rootfs/dev/tty4 rm ./rootfs/dev/tty2 rm ./rootfs/dev/random rm ./rootfs/dev/tty1 rm ./rootfs/dev/full # cd into rootfs and zip the container cd rootfs tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./ # the xz file can be copied over to proxmox and placed into # /var/lib/pve/local-btrfs/template/cache/ # for use as a template for container creation
During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.
If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:
lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less,python-minimal
An existing cache can be updated with something like:
sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64 apt-get update apt-get dist-upgrade apt-get clean exit
courtesy of Updating lxc image/container caches
One other note, there are two package candidates for installing the ping utility:
Some fix-ups in the process:
First of all, the obligatory caveat from 2023: where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.
I do totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.
However, the combination of Docker in LXC is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application, particularly when an application is packaged as a Docker container, without recourse for building a native LXC container of the application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.
The key benefit is that devices such as one or more GPUs can be passed through to multiple LXC containers plus any nested docker containers. Otherwise, in the scenario where the GPU or PCIe device is passed through to a VM, as far as I know, it has to be dedicated to the VM. I've read that the devices can not be shared between a VM and LXC containers due to configuration differences between VM pass-through and LXC pass-through.
Given the caveat, I'll see if I can make this work. Not so easy. Trying to run
docker run --rm hello-world
Yields an error:
docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: mount source: "overlay", target: "/tmp/containerd-mount2030888385", fstype: overlay, flags: 0, data: " workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work, upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs, lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", err: permission denied
With an associated apparmor error in Proxmox:
audit: type=1400 audit(1774803476.655:145): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-131_" name="/tmp/containerd-mount2030888385/" pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"
The simple solution is to set nesting=1 in the proxmox lxc options.
The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:
> ps aux root 41 0.0 0.0 2680 1808 ? Ss 20:09 0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh
If so, this can be disabled:
systemctl disable ifupdown-wait-online.service
In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging
systemctl edit systemd-networkd-wait-online.service [Service] Environment=SYSTEMD_LOG_LEVEL=debug
In my case, I then saw something like:
root@frigate01:~# systemctl status systemd-networkd-wait-online.service ● systemd-networkd-wait-online.service - Wait for Network to be Configured Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.
I have used a non-standard interface name. I resolved this by updating the edit with the following:
> systemctl edit systemd-networkd-wait-online.service [Service] ExecStart= ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60 #Environment=SYSTEMD_LOG_LEVEL=debug
The empty ExecStart line clears the original command parameters.
Some Docker commands:
docker run --rm -it hello-world bash
Note to self: come back to this and redo some non-functional audio logic located in some code which needs to be uprooted and re-juiced.
JUCE - open source, cross-platform, software development framework provided as C++ source code, that can be used to create standalone software on Windows, macOS, Linux, iOS and Android, as well as VST, VST3, AU, AUv3, AAX and LV2 plug-ins.
The Git Commands I Run Before Reading Any Code - some interesting simple queries on a git repository to gain some understanding on development patterns
Most changed files in the last year:
List of contributors:
Example of looking for hot words in comment messages:
Time line of commits:
Another example of comment keyword searches:
Hacker News - referenced
git log --format=format: --name-only --since="1 year ago" | sort | uniq -c | sort -nr | head -20
git shortlog -sn --no-merges
git log -i -E --grep="fix|bug|broken" --name-only --format='' | sort | uniq -c | sort -nr | head -20
git log --format='%ad' --date=format:'%Y-%m' | sort | uniq -c
git log --oneline --since="1 year ago" | grep -iE 'revert|hotfix|emergency|rollback'
Images vs Containers
Each instruction in the Dockerfile adds an extra layer to the Docker image. Minimize the number of layers by consolidating the instructions to increase the build’s performance and time.
Avoid using multiple RUN commands as it creates multiple cacheable layers which will affect the efficiency of the build process.
Use a single process per container: Each container should run a single process. This makes it easier to manage and monitor containers and helps to keep containers lightweight.
Images can exist without containers, whereas a container needs an image to run. We can create multiple containers from the same image, each with its own unique data and state
Docker commands
Docker network commands:
Documentation
APT based distributions like Debian can be containerized with a tool called debootstrap. It is part of the image build process of lxc-create. It is also referenced in Docker Base Images for building an image from scratch.
When looking at the build scripts included in the package installation, repositories for the following distributions can be found in /usr/share/debootstrap/scripts:
The wiki shows a simple two liner to get the basics of the distribution in place (as root):
mkdir trixie-chroot debootstrap stable trixie-chroot http://deb.debian.org/debian/
Enter the chroot and note new root:
root@test:~# pwd /root root@test:~# chroot trixie/ root@test:/# pwd / root@test:/# exit exit root@test:~# pwd /root
After the debootstrap, create the base for docker, and give it a try:
tar -C trixie-chroot -c . | docker import - trixie docker run trixie cat /etc/debian_version docker run --rm -i -t trixie /bin/bash
Although debootstrap can be used to build an image for a version subsequent, it is generally recommended to use debootstrap from at least the desired version to ensure it has the proper updates and dependencies.
Command line to summarize the referenced repositories:
grep -h default_mirror /usr/share/debootstrap/scripts/* \ | sed 's/default_mirror//' \ | sed 's/[ \t]//g' \ | sort \ | uniq
Scrolling keys:
Go:
Select:
Search:
Change:
Delete:
Jumps:
Tabs:
Column Insertion (full vim needs to be installed, won't work with vim.tiny):
Scrolling and line numbers in .vimrc file:
" http://vim.wikia.com/wiki/Keep_your_cursor_centered_vertically_on_the_screen set scrolloff=5 " http://jeffkreeftmeijer.com/2013/vims-new-hybrid-line-number-mode/ set relativenumber set number " http://unix.stackexchange.com/questions/5310/how-can-i-disable-bells-visualbells-in-vim set visualbell set t_vb=
Kind of related, but using 'screen' to split windows (from StackExchange:
Lots of tips, tricks, and techniques at vim wikia.
How to Enable GPU Passthrough to LXC Containers in Proxmox indicates that the process of providing passthrough of a GPU to both an LXC container as well as a Virtual Machine is not possible as the two types of configurations conflict with each other.
As my own preference is to run whatever possible in LXC containers, I'll summarize the configuration I used, which is an amalgamation of configurations from several sites.
My current installation is ProxMox v9.1.6 with:
In BIOS/UEFI, enable these:
Proxmox kernel parameters:
# /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction" update-grub reboot
VFIO Binding - optional but recommended:
# /etc/modprobe.d/vfio.conf options vfio_iommu_type1 allow_unsafe_interrupts=1
Obtain Linux drivers from NVidia. The CUDA toolkit is not required. Only the drivers are required in ProxMox. Toolkits and add-ons are added within the container.
Install the drivers:
apt install build-essential apt install pve-headers-$(uname -r) sh NVIDIA-Linux-x86_64-595.58.03.run # note, use the open kernel, rather than proprietary
Blacklist nouveau:
cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF blacklist nouveau options nouveau modeset=0 EOF
Test that the card is accessible:
nvidia-smi
Enable Data Persistence to prevent the GPU from re-initializing with each use.
nvidia-persistenced --persistence-mode systemctl enable nvidia-persistenced
Then restart:
reboot
Identify the nvidia devices requiring passthrough:
root@host02:~# ls -al /dev/nvidia* crw-rw-rw- 1 root root 195, 0 Mar 28 11:58 /dev/nvidia0 crw-rw-rw- 1 root root 195, 255 Mar 28 11:58 /dev/nvidiactl crw-rw-rw- 1 root root 505, 0 Mar 28 11:58 /dev/nvidia-uvm crw-rw-rw- 1 root root 505, 1 Mar 28 11:58 /dev/nvidia-uvm-tools /dev/nvidia-caps: total 0 drwxr-xr-x 2 root root 80 Mar 28 11:58 . drwxr-xr-x 21 root root 5060 Mar 28 11:58 .. cr-------- 1 root root 508, 1 Mar 28 11:58 nvidia-cap1 cr--r--r-- 1 root root 508, 2 Mar 28 11:58 nvidia-cap2
Note the numbers 195, 505 and 508 in this list (yours may be different).
Construct a container, and prior to starting, place the following into /etc/pve/lxc/<vmid>.conf (based upon the device listing above):
dev0: /dev/nvidia0 dev1: /dev/nvidiactl dev2: /dev/nvidia-modeset dev3: /dev/nvidia-uvm dev4: /dev/nvidia-uvm-tools dev5: /dev/nvidia-caps/nvidia-cap1 dev6: /dev/nvidia-caps/nvidia-cap2
These lines are optional in the config file, one site talks about them by my container seems to work without them (they may be an old style cgroup2 style passthrough rather than the device oriented passthrough above):
lxc.cgroup2.devices.allow: c 195:* rwm lxc.cgroup2.devices.allow: c 505:* rwm lxc.cgroup2.devices.allow: c 508:* rwm
Start the container and push the driver file into the container:
pct push <vmid> downloads/NVIDIA-Linux-x86_64-595.58.03.run /root/NVIDIA-Linux-x86_64-595.58.03.run
In the container, install the driver, minus the kernel module:
apt install kmod sh NVIDIA-Linux-x86_64-595.58.03.run --no-kernel-modules
Run nvidia-smi in the container to confirm the card is reachable.
Add nvtop at the host or the container level to chart live GPU utllization:
apt install nvtop
Additional resources:
Another type of test to run when pytorch is installed:
python -c "import torch; print(torch.cuda.is_available())"
Docker installation is easy enough:
curl -fsSL https://get.docker.com -o get-docker.sh sh get-docker.sh
This installs the latest stable release of:
To get an idea of usage:
sh -c docker version
Some say this happens due to Wayland usage. My usage is Wayland/X11, not that it may make a difference. It is that way because I like that fact that the windows of an application get placed back to where they were when the application is re-opened.
The solution appears to be:
Tools -> Options -> General -> Enable 'Use LibreOffice dialogs'
In the Nanog email list, the following was posted as a summary of current tooling use for network management in Debian:
Linux has a bunch of different possible ways to administer all of this stuff.
- The most comprehensive CLI mechanism is iproute2 (the “ip” command and some related constructs).
- The most comprehensive and capable persistent configuration database mechanism is systemd-networkd.
Other persistent mechanisms include:
- Netplan (YAML based configurations that now days mostly get parsed into systemd-networkd files and then executed).
- Debian Traditional (the /etc/network/interfaces file and/or interfaces.d directory, ifup/ifdown/etc.). Lacks many features, but most can be worked around with iproute2 shell commands added to triggers in the file.
- Debian Traditional can be supplemented with ifupdown2 - ifupdown replacement from Cumulus Networks
- NetworkManager (semi-capable, but any capabilities it lacks are just hard to cope with).
My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done.
Overall, it’s a pretty comprehensive interface and provides good logs for troubleshooting in most circumstances.
When creating a Debian testing/forky LXC container on a Debian trixie machine, the following error may be encountered in the output:
I: Resolving dependencies of required packages... I: Resolving dependencies of base packages... I: Checking component main on http://deb.debian.org/debian... E: Couldn't find these debs: isc-dhcp-client Failed to download the rootfs, aborting. Failed to download 'debian base' failed to install debian
This is a result of bug #1125011 in the Debian bug tracker.
There are several possible solutions:
Pip vs Pipenv: Which is better and which to learn first compares pipenv vs the python3-pip and virtualenv package combo.
After referring to that, I think I'll just stick with the standard pip/virtualenv combo for now.
To get started:
# install basic packages apt-get install python3 python3-pip virtualenv python3-venv git # create a project directory - example ansible python3 -m venv ansible # activate the project cd ansible source bin/activate # example installation of packages pip install ansible pip install argcomplete activate-global-python-argcomplete source ~/.bash_completion ansible-config init --disabled > ansible.cfg # to deactivate the project deactivate # upgrade python3 -m pip install --upgrade ansible
At some point, integrate python3-ansible-runner, the github source and links to documentation.
If running Firefox on a Debian Linux machine, install virt-viewer:
sudo apt install virt-viewer
Ensure the VirtIO drivers and such have been installed in the virtual machine (if Windows) in order to provide SPICE services.
Then, in Firefox on your workstation:
After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:
apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"
My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.
Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):
lxc.apparmor.profile: unconfined features: keyctl=1,nesting=1
I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.
After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:
lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,
Restart the container, and the errors should no longer occur.
Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.
I used the following for a trixie v13.3 version of a container:
lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,
pct_id=101 pct_name=test01 pct create $pct_id /var/lib/pve/local-btrfs/template/cache/trixie-13-3-template.tar.xz \ -hostname $pct_name \ -description 'demo build' \ -onboot 1 \ -startup up=3 \ -ostype debian \ -arch amd64 \ -cores 2 \ -memory 1024 \ -nameserver 10.10.10.10 -searchdomain 'example.com' \ -net0 name=vlan30,bridge=vmbr1,ip=dhcp,tag=30,type=veth \ -rootfs local-btrfs:8,mountoptions="noatime;discard" \ -swap 512
For some machines with LXC containers, they have been running for a number of years. I want to take the easy way out and move the containers from one physical machine to another. At another time, I will rebuild the containers.
Since I am running BTRFS subvolumes for each container, I could be using BTRFS snapshot/send/receive commands to migrate/copy/replicate subvolumes. But before attempting that, I wanted to give the 'copy' a try. To do this properly, at the source, use the following -- with numeric-owner being a required paramenter -- command to collect the files:
tar --numeric-owner -czvf mycontainer.tar.gz /var/lib/lxc/my_container
At the destination, expand that file out:
tar --numeric-owner -xzvf mycontainer.tar.gz -C /var/lib/lxc/
The lxc users mailing list and Stack OverFlow were helpful.
Other stuff to do:
In migrating from a very old version of LXC to a much newer version of LXC, I was getting errors. I needed to run a some debug to get a handle on errors:
lxc-start -n container -F --logpriority=DEBUG --logfile log
I had errors along the lines of:
Activating lvm and md swap...done. Checking file systems...Segmentation fault (core dumped) failed (code 139).
ServerFault had the solution: put "vsyscall=emulate" into /etc/default/grub, run 'update-grub' and reboot. Looks like I need to modernize my containers so I can eliminate this workaround, which may have some security considerations. There is a Debian Bug for this.
einstein home has a blog with some kernel references to the issue, in effect saying: "vsyscall is now disabled on latest linux distros". A lengthier LWN article at On vsyscalls and the vDSO. This works with kernel 4.14, my current version, but I see somewhere else that the workaround is entirely removed in kernel 4.15, at least in the Arch world. At bug 847154: "This breaks (e)glibc 2.13 and earlier".
Note, see newer notes at LXC Fresh Container Construction From Scratch for Proxmox.
Something from a Debian mailing list:
I found the root cause, when testing 6.12.57 I installed the image then the headers and the NVIDIA DKMS module was not rebuilt because the matching linux- headers package was not installed at the time the kernel image was configured.
If I install the headers first and then the linux-image package, DKMS correctly builds the NVIDIA module and 6.12.63 works fine, so it doesn't look like a kernel regression after all.
I don't know if I should manually run dkms autoinstall myself after a kernel update (I never had to before) or if there was a bug during the install process of this update.
Makes sense, I had NVidia compile fail in a similar. This makes it obvious what I should have observed.