Raymond P. Burkholder - Things I Do

First of all, the obligatory caveat from 2023 where Proxmox developers discourage running Docker in LXC. Upgrades to Proxmox may break 'something', which will require remediation of the containers. The relationship between Proxmox, LXC and Docker is brittle.

And I totally agree not to install Docker directly on the Proxmox host, as Docker will conflict with many networking and functional operations.

However, the combination is just too enticing. What other mechanism is available to compartmentalize applications and provide GPU resources to each compartmentalized application? Putting LXC and Docker into a VM seems a bit 'heavy' just for the sake of softening some brittleness. All the same management has to take place within the VM.

Given the caveat, I'll see if I can make this work. Not so easy. Trying to run

docker run --rm hello-world

Yields an error:

docker: Error response from daemon: failed to mount /tmp/containerd-mount2030888385: 
mount source: "overlay", target: "/tmp/containerd-mount2030888385", 
fstype: overlay, flags: 0, 
data: "
  workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work,
  upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs,
  lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,userxattr", 
  err: permission denied

With an associated apparmor error in Proxmox:

audit: type=1400 audit(1774803476.655:145): 
  apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 
  profile="lxc-131_" 
  name="/tmp/containerd-mount2030888385/" 
  pid=1480790 comm="dockerd" fstype="overlay" srcname="overlay"

The simple solution is to set nesting=1 in the proxmox lxc options.

The next hurdle is that it may take a couple/several minutes for the Docker file to run when the container starts up. If so, you may see this:

> ps aux
root      41  0.0  0.0   2680  1808 ?    Ss   20:09   0:00 /bin/sh /usr/lib/ifupdown/wait-online.sh

If so, this can be disabled:

systemctl disable ifupdown-wait-online.service

In addition, systemd-networkd-wait-online may be waiting for an interface it doesn't manage. This will cause a startup delay of several minutes. Use the following to add some debugging and logging

systemctl edit systemd-networkd-wait-online.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

In my case, I then saw something like:

root@frigate01:~# systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured

Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: lo: link is ignored
Mar 29 20:38:44 frigate01 systemd-networkd-wait-online[97]: vlan60: link is not managed by networkd.

I have used a non-standard interface name. I resolved this by updating the edit with the following:

> systemctl edit systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=vlan60
#Environment=SYSTEMD_LOG_LEVEL=debug

The empty ExecStart line clears the original command parameters.

• How to Fix systemd-networkd-wait-online Service Timing Out During Boot
• systemd-networkd-wait-online.service(8) — Linux manual page

Some Docker commands:

docker run --rm -it hello-world bash

Docker installation is easy enough:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

This installs the latest stable release of:

• Docker CLI,
• Docker Engine,
• Docker Buildx,
• Docker Compose,
• containerd, and
• runc.

To get an idea of usage:

sh -c docker version

• ollama - framework for running models
• PyTorch Installation
• Hugging Face - models
• Open WebUI - extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It is built around universal standards, supporting Ollama and OpenAI-compatible Protocols (specifically Chat Completions).

How to Enable GPU Passthrough to LXC Containers in Proxmox indicates that the process of providing passthrough of a GPU to both an LXC container as well as a Virtual Machine is not possible as the two types of configurations conflict with each other.

As my own preference is to run whatever possible in LXC containers, I'll summarize the configuration I used, which is an amalgamation of configurations from several sites.

My current installation is ProxMox v9.1.6 with:

• ProArt Z890-CREATOR WIFI
• Intel(R) Core(TM) Ultra 9 285K
• Corsair CMP64GX5M2X6600C32 (128G 4400 MT/s) - ECC would have been nice
• NVIDIA Corporation AD103 [GeForce RTX 4070] (rev a1)

In BIOS/UEFI, enable these:

• VT-d / IOMMU
• Above 4G Decoding
• PCIe Native Power Management (if available)

Proxmox kernel parameters:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

update-grub
reboot

VFIO Binding - optional but recommended:

# /etc/modprobe.d/vfio.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Obtain Linux drivers from NVidia. The CUDA toolkit is not required. Only the drivers are required in ProxMox. Toolkits and add-ons are added within the container.

Install the drivers:

apt install build-essential
apt install pve-headers-$(uname -r)
sh NVIDIA-Linux-x86_64-595.58.03.run
# note, use the open kernel, rather than proprietary

Blacklist nouveau:

cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF

Test that the card is accessible:

nvidia-smi

Enable Data Persistence to prevent the GPU from re-initializing with each use.

nvidia-persistenced --persistence-mode
systemctl enable nvidia-persistenced

Then restart:

reboot

Identify the nvidia devices requiring passthrough:

root@host02:~# ls -al /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Mar 28 11:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Mar 28 11:58 /dev/nvidiactl
crw-rw-rw- 1 root root 505,   0 Mar 28 11:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 505,   1 Mar 28 11:58 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Mar 28 11:58 .
drwxr-xr-x 21 root root   5060 Mar 28 11:58 ..
cr--------  1 root root 508, 1 Mar 28 11:58 nvidia-cap1
cr--r--r--  1 root root 508, 2 Mar 28 11:58 nvidia-cap2

Note the numbers 195, 505 and 508 in this list (yours may be different).

Construct a container, and prior to starting, place the following into /etc/pve/lxc/<vmid>.conf (based upon the device listing above):

dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-modeset
dev3: /dev/nvidia-uvm
dev4: /dev/nvidia-uvm-tools
dev5: /dev/nvidia-caps/nvidia-cap1
dev6: /dev/nvidia-caps/nvidia-cap2

These lines are optional in the config file, one site talks about them by my container seems to work without them (they may be an old style cgroup2 style passthrough rather than the device oriented passthrough above):

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 505:* rwm
lxc.cgroup2.devices.allow: c 508:* rwm

Start the container and push the driver file into the container:

pct push <vmid> downloads/NVIDIA-Linux-x86_64-595.58.03.run /root/NVIDIA-Linux-x86_64-595.58.03.run

In the container, install the driver, minus the kernel module:

apt install kmod
sh NVIDIA-Linux-x86_64-595.58.03.run --no-kernel-modules

Run nvidia-smi in the container to confirm the card is reachable.

Add nvtop at the host or the container level to chart live GPU utllization:

apt install nvtop

Additional resources:

• Complete GPU passthrough guide for AI workloads, avoid the mistakes I made so you don't have to
• [TUTORIAL] NVIDIA drivers instalation Proxmox and CT
• How to Enable GPU Passthrough to LXC Containers in Proxmox - contains ollama startup examples with OpenWebUI

Another type of test to run when pytorch is installed:

python -c "import torch; print(torch.cuda.is_available())"

Some say this happens due to Wayland usage. My usage is Wayland/X11, not that it may make a difference. It is that way because I like that fact that the windows of an application get placed back to where they were when the application is re-opened.

The solution appears to be:

Tools -> Options -> General -> Enable 'Use LibreOffice dialogs'

In the Nanog email list, the following was posted as a summary of current tooling use for network management in Debian:

Linux has a bunch of different possible ways to administer all of this stuff.

• The most comprehensive CLI mechanism is iproute2 (the “ip” command and some related constructs).
• The most comprehensive and capable persistent configuration database mechanism is systemd-networkd.

Other persistent mechanisms include:

• Netplan (YAML based configurations that now days mostly get parsed into systemd-networkd files and then executed).
• Debian Traditional (the /etc/network/interfaces file and/or interfaces.d directory, ifup/ifdown/etc.). Lacks many features, but most can be worked around with iproute2 shell commands added to triggers in the file.
• Debian Traditional can be supplemented with ifupdown2 - ifupdown replacement from Cumulus Networks
• NetworkManager (semi-capable, but any capabilities it lacks are just hard to cope with).

My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done.

Overall, it’s a pretty comprehensive interface and provides good logs for troubleshooting in most circumstances.

When creating a Debian testing/forky LXC container on a Debian trixie machine, the following error may be encountered in the output:

I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://deb.debian.org/debian...
E: Couldn't find these debs: isc-dhcp-client
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian

This is a result of bug #1125011 in the Debian bug tracker.

There are several possible solutions:

• Manually apply the patches supplied by the Debian LXC team
• Probably might be solved by running lxc-create on a testing/forky machine, where the solution may have already been applied - I have not confirmed this
• Or it may work on a sid machine

If running Firefox on a Debian Linux machine, install virt-viewer:

sudo apt install virt-viewer

Ensure the VirtIO drivers and such have been installed in the virtual machine (if Windows) in order to provide SPICE services.

Then, in Firefox on your workstation:

• go into about:config, and add the key 'network.protocol-handler.expose.virt-viewer' as boolean and set to true
• go into about:preferences, and set "What should Firefox do with other files" to "Ask whether to open or save files".
• in Proxmox, open a SPICE based console for a virtual machine, which attempts a download or a run of a customized .vv file,
• Firefox will then request to open a Virt-Viewer file with Remote Viewer - at this point, you can set it as the default viewer, and it will show up in the application preferences

After following my own instructions for building my own LXC container template for ProxMox using the SID release, when the container started, the ProxMox logs would fill up with errors along the lines of:

apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 name="/run/credentials/systemd-journald.service/" flags="rw, move"

My Trixie template did not seem to offer up these types of errors. LXC containers were created with the 'Unpriviledged Container" setting to 1|yes.

Instead of going the last resort brute force and ignorance route of using the following configuration (see Fixing net.ipv4.ip_unprivileged_port_start and AppArmor Docker Errors in a Proxmox LXC for some background):

lxc.apparmor.profile: unconfined
features: keyctl=1,nesting=1

I took a more nuanced/detailed approach. AppArmor Denied Operation mount info failed flags match Error 13 provided a starting point for developing a solution.

After incrementally adding rules as new Apparmor DENIED statements occurred, this is the rule set which seems to resolve the errors. Once the container is created, these are the rules I add to the end of /etc/pve/lxc/<vmid>.conf:

lxc.apparmor.raw: mount options=(rw,move) -> /run/credentials/{,**},
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: userns create,

Restart the container, and the errors should no longer occur.

Don't try to place statements in /var/lib/lxc/<vmid>/config as it is over-written by ProxMox upon container startup. Rules are appended to that configuration.

I used the following for a trixie v13.3 version of a container:

lxc.apparmor.raw: mount fstype=ramfs -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /dev/shm/,
lxc.apparmor.raw: mount options=(ro, remount, bind) -> /dev/,
lxc.apparmor.raw: mount options=(rw, move) -> /dev/mqueue/,
lxc.apparmor.raw: mount options=(rw, move) -> /run/lock/,
lxc.apparmor.raw: mount options=(rw, move) -> /tmp/,
lxc.apparmor.raw: mount options=(ro, remount, noatime, bind) -> /,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec, remount, nosymfollow, bind) -> /run/systemd/mount-rootfs/run/credentials/systemd-networkd.service/,
lxc.apparmor.raw: userns create,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/namespace-{,**},
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/sys/net/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/uptime,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/slabinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/meminfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/swaps,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/loadavg,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/cpuinfo,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/diskstats,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec, remount, bind) -> /run/systemd/mount-rootfs/proc/stat,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /run/systemd/unit-root/proc/,
lxc.apparmor.raw: mount options=(ro, nosuid, nodev, noexec) -> /sys/kernel/config/,
lxc.apparmor.raw: mount options=(rw, nosuid, nodev, noexec) -> /sys/kernel/config/,

Pip vs Pipenv: Which is better and which to learn first compares pipenv vs the python3-pip and virtualenv package combo.

After referring to that, I think I'll just stick with the standard pip/virtualenv combo for now.

To get started:

# install basic packages
apt-get install python3 python3-pip virtualenv  python3-venv git

# create a project directory - example ansible
python3 -m venv ansible

# activate the project
cd ansible
source bin/activate

# example installation of packages
pip install ansible
pip install argcomplete
activate-global-python-argcomplete
source ~/.bash_completion
ansible-config init --disabled > ansible.cfg

# to deactivate the project
deactivate

# upgrade
python3 -m pip install --upgrade ansible

At some point, integrate python3-ansible-runner, the github source and links to documentation.

pct_id=101
pct_name=test01
pct create $pct_id /var/lib/pve/local-btrfs/template/cache/trixie-13-3-template.tar.xz  \
  -hostname $pct_name \
  -description 'demo build' \
  -onboot 1 \
  -startup up=3 \
  -ostype debian \
  -arch amd64 \
  -cores 2 \
  -memory 1024 \
  -nameserver 10.10.10.10 -searchdomain 'example.com' \
  -net0 name=vlan30,bridge=vmbr1,ip=dhcp,tag=30,type=veth \
  -rootfs local-btrfs:8,mountoptions="noatime;discard" \
  -swap 512

There are many articles available which discuss customizing a pre-existing Proxmox Container Template. Few, if any, discuss constructing an LXC container from scratch. Maybe because, fundamentally, a container template is just the rootfs as tarball, so building it is quite easy:

• Build a linux based virtual machine, I use Debian's recent release
• Install LXC and its template package
• Construct and initialize an LXC container
• Shut it down and and zip it up
• Copy it over to the ProxMox template directory

The details:

# build the linux vm - details not relevant here
# ssh into the vm, or start a command line
# install basic packages

sudo apt install --no-install-recommends lxc lxc-templates xz-utils bridge-utils wget debootstrap rsync

# basic container templates are in:
#   /usr/share/lxc/templates/ 
# for debian as well as other distributions

# create an lxc container, provide a list any additional packages

lxc-create --template debian --name trixie-template -- --release trixie --packages iputils-ping,vim-tiny

# start and attach to the container
lxc-start trixie-template
lxc-attach trixie-template

# prepare for generating template
apt clean
apt purge

# Remove SSH host keys to ensure unique keys for each clone:
rm /etc/ssh/ssh_host_*

# Empty the machine ID file:
truncate -s 0 /etc/machine-id

# clear history
unset HISTFILE
# truncate history
history -c
> ~/.bash_history
# the following has a space in front to prevent inclusion in the history
 shutdown -h now

# the shutdown returns to the virtual machine's prompt
# compress the directory structure

cd /var/lib/lxc/trixie-template/

# remove /dev files as they can't be created in an unprivileged container
# an example error message if not removed:
#   tar: ./rootfs/dev/urandom: Cannot mknod: Operation not permitted
# construction of a new container will re-create the directory and files

rm ./rootfs/dev/ptmx
rm ./rootfs/dev/zero
rm ./rootfs/dev/tty3
rm ./rootfs/dev/urandom
rm ./rootfs/dev/null
rm ./rootfs/dev/tty
rm ./rootfs/dev/console
rm ./rootfs/dev/tty4
rm ./rootfs/dev/tty2
rm ./rootfs/dev/random
rm ./rootfs/dev/tty1
rm ./rootfs/dev/full

# cd into rootfs and zip the container

cd rootfs
tar --xz --acls --numeric-owner -cf /var/local/trixie-13-3-template.tar.xz ./

# the xz file can be copied over to proxmox and placed into
# /var/lib/pve/local-btrfs/template/cache/
# for use as a template for container creation

During the first use of lxc-create to create the original container, packages are downloaded and installed to build the container. The packages and installation is cached for faster subsequent builds of the same container type.

If the cache becomes stale, it can be rebuilt by using --flush-cache in a manner similar to:

lxc-create --template debian --name trixie-template -- --release trixie --flush-cache --packages iputils-ping,vim-tiny,less

An existing cache can be updated with something like:

sudo chroot /var/cache/lxc/debian/rootfs-trixie-amd64
apt-get update
apt-get dist-upgrade
apt-get clean
exit

courtesy of Updating lxc image/container caches

One other note, there are two package candidates for installing the ping utility:

• iputils-ping - native Linux ping, preferred for Debian/Linux
• inetutils-ping - general gnu version, used on a variety of posix sytstems, less preferred

Some fix-ups in the process:

• apt-get install less
• dpkg-reconfigure locales
• useradd user

Something from a Debian mailing list:

I found the root cause, when testing 6.12.57 I installed the image then the headers and the NVIDIA DKMS module was not rebuilt because the matching linux- headers package was not installed at the time the kernel image was configured.

If I install the headers first and then the linux-image package, DKMS correctly builds the NVIDIA module and 6.12.63 works fine, so it doesn't look like a kernel regression after all.

I don't know if I should manually run dkms autoinstall myself after a kernel update (I never had to before) or if there was a bug during the install process of this update.

Makes sense, I had NVidia compile fail in a similar. This makes it obvious what I should have observed.

The difference between reactive and proactive monitoring comes down to tracking the right network performance metrics. Important metrics that consistently predict issues before they impact users include:

• Round-trip time variations: Subtle increases in RTT often precede more serious network congestion. Set baseline thresholds for different times of day and get alerted when patterns deviate
• Buffer utilization patterns: Most admins ignore this until it's too late. Monitoring buffer usage across key network devices reveals impending bottlenecks before they trigger packet loss
• Error rates on interfaces: Even small increases in CRC errors or interface discards can signal hardware issues days before actual failures occur
• TCP retransmission rates: This metric reveals application performance issues that users might experience from data transmission rates before they report them
• Network traffic symmetry: Unusual asymmetric traffic patterns often indicate security issues or misconfigured applications

Reference: Stop Reactive Network Troubleshooting: Monitor These 5 Metrics to Prevent Downtime

Cloud environments introduce unique challenges for network performance monitoring. Some standard metrics like latency and packet loss remain important, but you'll also want to focus on metrics specific to your cloud connectivity. For hybrid environments, these are good metrics:

• Inter-region latency: Especially important for globally distributed applications
• Connection establishment time: Often overlooked but crucial for microservice architectures
• Throughput consistency: More important than raw bandwidth in many cloud scenarios
• DNS resolution time: Can be a hidden bottleneck in cloud environments

It has been a while since I last setup NordVPN on a Debian Linux using StrongSwan. StrongSwan is now using 'native' files rather than the now deprecated ipsec files. NordVPN Example: How to connect to NordVPN with IKEv2/IPSec on Linux refers to the old format. Here is a new format.

Here is my take on a successful installation.

apt install \
  --no-install-recommends \
    strongswan \
    libstrongswan-standard-plugins \
    libstrongswan-extra-plugins \
    libcharon-extra-plugins
wget https://downloads.nordcdn.com/certificates/root.pem -O /etc/swanctl/x509ca/NordVPN.pem
sed -i 's/load = yes/load = no/' /etc/strongswan.d/charon/constraints.conf

An example /etc/swanctl/swanctl.conf file:

connections {
  nordvpn {
    version = 2
    proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-ecp521,aes192-sha256-modp3072,default
    rekey_time = 0s
    fragmentation = yes
    dpd_delay = 300s
    local_addrs = %defaultroute
    remote_addrs = 
    vips=0.0.0.0,::
    local {
      auth = eap-mschapv2
      eap_id = "<username>"
    }
    remote {
      auth = pubkey
      cacerts = /etc/swanctl/x509ca/NordVPN.pem
      id = %any
    }
    children {
      nordvpn {
        remote_ts = 0.0.0.0/0,::/0
        rekey_time = 0s
        dpd_action = clear
        esp_proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-modp3072,aes192-sha256-ecp256-modp3072,default
      }
    }
  }
}

secrets {
  eap-nordvpn {
    id = "<username>"
    secret = "<password>"
  }
}

If you have a local network to which you need access when the vpn is up, StrongSwan using route table 220 for forwarding. Use the following command to see current settings:

# ip rule list
0:      from all lookup local
220:    from all lookup 220
32766:  from all lookup main
32767:  from all lookup default

# ip route list table 220
default via 192.168.1.10 dev eth0 proto static src 

To add your local network to the route table. Additional subnets are added in a similar way. Change the interface name to suit your local circumstances. Use the updown Plugin for better control of the local routing.

ip route add table 192.168.1.0/24 dev wlan0

This may be required for changes made:

# systemctl restart strongswan

Tunnel related state and status commands:

sudo swanctl --load-conns
sudo swanctl --list-conns
sudo swanctl --list-certs
sudo swanctl --list-sas
sudo swanctl --initiate --child nordvpn
sudo swanctl --terminate --child nordvpn
sudo swanctl --reload-settings

References:

• How to Setup an IKEv2 VPN Connection on Arch Linux (Example: NordVPN) - primary configuration template
• Allow Strongswan roadwarrior to access local LAN - keeping local traffic out of the vpn