LXC

• Qwen 3.5 9B - general purpose locally run LLM
• OctoPrint - web-based 3D printer control software that allows you to remotely control and monitor your 3D printer from a web interface. It was designed to be compatible with a wide range of 3D printers.
• Trilium Notes - self-hosted note-taking and personal knowledge management application. It enables users to organize information in a hierarchical tree structure and supports rich text editing, internal linking, images, attachments, and powerful scripting capabilities. This version reflects the most current development efforts under the TriliumNext organization and replaces all prior forks or legacy variants. Trilium is ideal for building personal wikis, structured documentation, and long-term knowledge archives, giving users full local control and privacy.
• Vaultwarden - self-hosted password manager which provides secure and encrypted password storage. It uses client-side encryption and provides access to passwords through a web interface and mobile apps.
• NetBox - the source of truth for everything on your network, from physical components like power systems and cabling to virtual assets like IP addresses and VLANs. Network automation and observability tools depend on NetBox’s authoritative data to roll out configurations, monitor changes, and accelerate operations across the enterprise

First of all, the obligatory caveat from 2023: where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.

I do totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.

However, the combination of Docker in LXC is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application, particularly when an application is packaged as a Docker container, without recourse for building a native LXC container of the application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.

The key benefit is that devices such as one or more GPUs can be passed through to multiple LXC containers plus any nested docker containers. Otherwise, in the scenario where the GPU or PCIe device is passed through to a VM, as far as I know, it has to be dedicated to the VM. I've read that the devices can not be shared between a VM and LXC containers due to configuration differences between VM pass-through and LXC pass-through.

Given the caveat, I'll see if I can make this work. Not so easy. Trying to run

docker run --rm hello-world

Yields an error:

docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: 
mount source: "overlay", target: "/tmp/containerd-mount2030888385", 
fstype: overlay, flags: 0, 
data: "
  workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work,
  upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs,
  lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", 
  err: permission denied

With an associated apparmor error in Proxmox:

audit: type=1400 audit(1774803476.655:145): 
  apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 
  profile="lxc-131_" 
  name="/tmp/containerd-mount2030888385/" 
  pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"

The simple solution is to set nesting=1 in the proxmox lxc options.

The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:

> ps aux
root      41  0.0  0.0   2680  1808 ?    Ss   20:09   0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh

If so, this can be disabled:

systemctl disable ifupdown-wait-online.service

In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging

systemctl edit systemd-networkd-wait-online.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

In my case, I then saw something like:

root@frigate01:~# systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured

Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored
Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.

I have used a non-standard interface name. I resolved this by updating the edit with the following:

> systemctl edit systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60
#Environment=SYSTEMD_LOG_LEVEL=debug

The empty ExecStart line clears the original command parameters.

• How to Fix systemd-networkd-wait-online Service Timing Out During Boot
• systemd-networkd-wait-online.service(8) — Linux manual page

Some Docker commands:

docker run --rm -it hello-world bash

Docker installation is easy enough:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

This installs the latest stable release of:

• Docker CLI,
• Docker Engine,
• Docker Buildx,
• Docker Compose,
• containerd, and
• runc.

To get an idea of usage:

sh -c docker version

When creating a Debian testing/forky LXC container on a Debian trixie machine, the following error may be encountered in the output:

I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://deb.debian.org/debian...
E: Couldn't find these debs: isc-dhcp-client
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian

This is a result of bug #1125011 in the Debian bug tracker.

There are several possible solutions:

• Manually apply the patches supplied by the Debian LXC team
• Probably might be solved by running lxc-create on a testing/forky machine, where the solution may have already been applied - I have not confirmed this
• Or it may work on a sid machine

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

I used the following for a trixie v13.3 version of a container:

lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,

There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:

• Build a linux based virtual machine, I use Debian's recent release
• Install LXC and its template package
• Construct and initialize an LXC container
• Shut it down and and zip it up
• Copy it over to the ProxMox template directory

The details:

# build the linux vm - details not relevant here
# ssh into the vm, or start a command line
# install basic packages

sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync

# basic container templates are in:
#   /usr/share/lxc/templates/ 
# for debian as well as other distributions

# create an lxc container, provide a list any additional packages

lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny

# start and attach to the container
lxc-start trixie-template
lxc-attach trixie-template

# prepare for generating template
apt clean
apt purge

# Remove SSH host keys to ensure unique keys for each clone:
rm /etc/ssh/ssh_host_*

# Empty the machine ID file:
truncate -s 0 /etc/machine-id

# clear history
unset HISTFILE
# truncate history
history -c
> ~/.bash_history
# the following has a space in front to prevent inclusion in the history
 shutdown -h now

# the shutdown returns to the virtual machine's prompt
# compress the directory structure

cd /var/lib/lxc/trixie-template/

# remove /dev files as they can't be created in an unprivileged container
# an example error message if not removed:
#   tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted
# construction of a new container will re-create the directory and files

rm ./rootfs/dev/ptmx
rm ./rootfs/dev/zero
rm ./rootfs/dev/tty3
rm ./rootfs/dev/urandom
rm ./rootfs/dev/null
rm ./rootfs/dev/tty
rm ./rootfs/dev/console
rm ./rootfs/dev/tty4
rm ./rootfs/dev/tty2
rm ./rootfs/dev/random
rm ./rootfs/dev/tty1
rm ./rootfs/dev/full

# cd into rootfs and zip the container

cd rootfs
tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./

# the xz file can be copied over to proxmox and placed into
# /var/lib/pve/local-btrfs/template/cache/
# for use as a template for container creation

During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.

If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:

lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less,python-minimal

An existing cache can be updated with something like:

sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64
apt-get update
apt-get dist-upgrade
apt-get clean
exit

courtesy of Updating lxc image/container caches

One other note, there are two package candidates for installing the ping utility:

• iputils-ping - native Linux ping, preferred for Debian/Linux
• inetutils-ping - general gnu version, used on a variety of posix sytstems, less preferred

Some fix-ups in the process:

• apt-get install less
• dpkg-reconfigure locales
• useradd user

In a follow on post to my previous GUI From an LXC Container on the Host entry, here are some notes on linking sound in an LXC container to the host's sound system.

The primary reason for this was to get around Interactive Broker's broken sound requirements for really old libav libraries. Which is back to the version x53 libraries from the Debian Jessie / Ubuntu Precise 12.04 days. Long story short, I don't have this working yet. The basic issue I have is that, when running TWS in an LXC container, all the configuration menu options but one work. When I click on the 'Sound Manager' menu item, TWS locks up, and I don't see any log messages anywhere as to why this might happen. So there is something special in an LXC container environment which is locking up sound in TWS.

But I did get sound from FireFox / Youtube to work.

This on a Debian Bullseye system.

The basic solution is, in the LXC container configuration file, to mount the Pulse Audio socket from the user's account of the host into the container and set an environment for the user in the container. In this case, the user has UID of 1000:

lxc.mount.entry = /run/user/1000/pulse/native tmp/pulse/native none rw,bind,create=file 0 0
lxc.environment = PULSE_SERVER=unix:/tmp/pulse/native

For completeness, these can be added (helps with ALSA):

lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir
lxc.cgroup.devices.allow = c 116:* rwm

That is about all to make it all work. Some or all of the following package installs in the container might help:

sudo apt install \
  pulseaudio \
  pulseaudio-utils \
  pavucontrol \
  gstreamer1.0-libav gstreamer1.0-plugins-bad gstreamer1.0-pulseaudio \
  mpg123

For some machines with LXC containers, they have been running for a number of years. I want to take the easy way out and move the containers from one physical machine to another. At another time, I will rebuild the containers.

Since I am running BTRFS subvolumes for each container, I could be using BTRFS snapshot/send/receive commands to migrate/copy/replicate subvolumes. But before attempting that, I wanted to give the 'copy' a try. To do this properly, at the source, use the following -- with numeric-owner being a required paramenter -- command to collect the files:

tar --numeric-owner -czvf mycontainer.tar.gz /var/lib/lxc/my_container

At the destination, expand that file out:

tar --numeric-owner -xzvf mycontainer.tar.gz -C /var/lib/lxc/

The lxc users mailing list and Stack OverFlow were helpful.

Other stuff to do:

• Read up on CGroups in the Linux Programmer's Manual

In migrating from a very old version of LXC to a much newer version of LXC, I was getting errors. I needed to run a some debug to get a handle on errors:

lxc-start -n container -F --logpriority=DEBUG --logfile log

I had errors along the lines of:

Activating lvm and md swap...done.
Checking file systems...Segmentation fault (core dumped)
failed (code 139).

ServerFault had the solution: put "vsyscall=emulate" into /etc/default/grub, run 'update-grub' and reboot. Looks like I need to modernize my containers so I can eliminate this workaround, which may have some security considerations. There is a Debian Bug for this.

einstein home has a blog with some kernel references to the issue, in effect saying: "vsyscall is now disabled on latest linux distros". A lengthier LWN article at On vsyscalls and the vDSO. This works with kernel 4.14, my current version, but I see somewhere else that the workaround is entirely removed in kernel 4.15, at least in the Arch world. At bug 847154: "This breaks (e)glibc 2.13 and earlier".

Note, see newer notes at LXC Fresh Container Construction From Scratch for Proxmox.

• Configuring network namespaces in Linux
• manage kvm virtual machines

Policy Based Routing: using named routing table in Linux, a basic outline:

With Quagga, no. But you can implement it in the Linux shell. Examples
from Ubuntu:

1. First create a named routing table in /etc/iproute2/rt_tables
200 MY-TABLE

2. Then create routes in that table:
ip route add default via 172.16.0.1 table MY-TABLE

3. Finally, create the policy:
ip rule add from 10.0.0.0/29 table MY-TABLE

See all the "ip rule" commands for full details. I do a lot of this. 

I'm in the process of figuring out Docker, and how to make it run on Debian. These are just 'notes to self' to look at for another time. I am reverting to lxc (linux containers) for the time being.

• Debian package for Docker is still several versions behind (with the current appearing to be v1.50 from 2015/02/03, see Docker release notes). It is better to download the binary from Docker.
• Some notes using Docker in a multi-host environment: Multi-host Docker and OpenvSwitch
• Some web sites suggest 'apt-get install docker.io' on Debian Jessie to install Docker. Even the Docker site says this: Docker on Debian Jessie. But that command does not work as of this writing.
• Debian doesn't seem to have an official base image, just a semi-official base image, but really not really, based upon some reading which says that some kluges went into building the base image.
• Some other pointers for a Debian Jessie base image stuff: debian docker base image, Installing Docker on Debian Jessie (from 2013), Installing Docker in Debian (testing/Jessie) (also 2013) , CloudCreateDockerImage

But then, all this might be irrelevant. There is something called CoreOS, which is a ground up solution for running containers.

On my Debian 7 Wheezy systems running LXC, I have recently seen libvirt_lxc go up to 100% cpu utilization on a continuous basis. The brute force fix was to restart the guest. But that isn't a good idea.

There is a suggestion at [Bug 1005570] libvirt_lxc consumes 100% cpu - msg#00023 that:

When this 100% cpu problem occurs, can anyone tell me if they still see a process running as a child of the 'libvirt_lxc' process ? Whether a zombie or normal process.

Looking at the code, the only way I can see us getting into this problem, is if the guest closed its console (STDIO) streams, but did not exit. We should always get SIG_CHILD if the process exited, so I'm thinking perhaps the container's init process closed stdin/out/err, but kept running

When I logged in via the lxc console as root, the high cpu utilization went a way.

As a side note, to login as root via the lxc console, I had to add 'pts/0' to /etc/securetty. I also only have one tty listed in /etc/inittab:

# Normally not reached, but fallthrough in case of emergency.
z6:6:respawn:/sbin/sulogin
1:2345:respawn:/sbin/getty 38400 tty1
#2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6

Let me see if these settings actually help prevent further recurrence of the high cpu usage in libvirt_lxc.

Now that the networking side of things are complete with Quagga and OpenVSwitch, I can start working further up the technology layers. The next step is to work on virtualization technologies and their management. The focus of this article is to get Linux Containers (LXC) working within a Debian Wheezy 7.1 environment. Continue reading "Linux Containers (LXC) on Debian Wheezy 7.1..." »

As part of the virtualization solutions I'm working on, I wanted something a little lighter weight than a full Qemu/KVM solution, which has a higer overhead than a kernel oriented container mechanism. There are many services that can be run with a common operating system. Containers provides the ability for solid segregation of services at the price of very low overhead.

For a container style of virtualization, I had been looking at OpenVZ (which is used by Parallel's Virtuozzo, I believe), Linux-VServer, and VirtualBox. However, on Debian's Upgrade Path to Wheezy, I see that Debian is deprecating OpenVZ and Linux-VServer.

Independently of this, I had earlier come across lxc Linux Containers. As such, it looks like LXC is the path forward for light weight, container-based virtualization.