Orchestration

I found out the hard way that SaltStack and Debian no longer place nice together. I had upgraded a Debian installation from Bullseye to Bookworm, along with the resident Salt Minion. When attempting to use the minion, it no longer starts up, due to various imports no longer working. Which was due to the salt-minion not being upgraded. The error message would started this odyssey:

salt ImportError: cannot import name 'Markup' from 'jinja2'

Taking a look at the Debian Developer Information for Salt, the last version started in 'unstable' was 3004.1 back in December of 2022. This is now almost 8 months later and little or no movement. There was some mention in a ticket somewhere that Salt release cycles don't cater to Debian stable release cycles. Not sure if that is a legitimate reason or not, but, well, for whatever reason, SaltStack management in Debian is no longer a simple no brainer.

However, after a little digging, there is a way to run SaltStack versions 3006 (current as of this writing). It is simple to install on Bullseye, but not easily done on Bookworm.

On Bullseye (as root, or implies sudo):

# cd ~ # apt remove salt-minion salt-master # apt install curl # curl -L https://bootstrap.saltstack.com -o install_salt.sh # sh install_salt.sh -M onedir

The '-M' installs the salt master at the same time (for machines running master). If you forget to do that, you'll need to diagnose and fix the systemctl mask error with the following:

# apt install file # file /etc/systemd/system/salt-master.service # rm /etc/systemd/system/salt-master.service # systemctl daemon-reload # sh install_salt.sh -M onedir

The 'sh install_salt.sh -M onedir' should show a symlink to /dev/nul, which the 'rm ...' will fix.

On Bookworm, the bootstrap isn't scheduled to work till beginning of 2024 sometime I think with Salt 3007 or 3008 -- more info in [FEATURE REQUEST] Add Salt support for Debian 12 #64223 .

In the meantime, I had to cheat a bit:

• in /etc/debian_version, change 12.0 to 11.0
• in /etc/apt/sources.list, change bookworm to bullseye
• rm /etc/apt/sources.list.d/salt.list
• run apt update
• run the commands listed above for installing the one or both the salt services
• restore /etc/debian_version and /etc/apt/sources.list to their original content

I'm sure there are more elegant ways of doing this, but this worked to fake the needed version 11 in the installation script and directory traversal requirements

Note, more info on the Salt Install/Bootstrap Process.

Some choice extractions from Configuration Management is an AntiPattern:

.... you’ve got an operations team that bottlenecks all changes in production. They’re the only ones with commit access to the configuration management repository, and they have to make all changes in production. This is the anti-DevOps way of doing things.

The other option is that you expect developers to run configuration management on the clusters that they are responsible for. This is the much more DevOps way of doing things, but it suffers from a different problem. Now you have to teach all your engineers the DSL of your configuration management software of choice, and depending on how you deploy your code to production, every developer now has the power to take down your whole system with a poorly written configuration change.

... you just have a separate configuration management branch for every cluster you run in production, and restrict developers to running code on only the systems they manage ...

configuration management promises that you’ll know the complete state of your infrastructure, but it never works that way

The alternative is immutable infrastructure. If you’re running in the cloud, this means baking your AMI with your application code already on it. See, imagine a world where you finish writing the code, you push it into git, and it gets built into an RPM or Debian package. That package is installed on a Base-AMI that was carefully crafted by your security and performance engineers, and then an AMI, with your software installed, along with its dependencies is pushed out

Start with a base, or foundation image — this is either your handcrafted, optimized image or just the default image from an upstream vendor. ... Your base image should have the latest security updates as well as any base infrastructure packages that are run platform-wide. Things like your monitoring packages, or your service discovery. Now, you could use a configuration management tool to build your base image, but you could also use OS packages and a little python to install and configure your base image.

This enables continuous deployments because new code just goes through your pipeline and you don’t have to deal with old versions of libraries or dependencies or configuration that might have been left around.

Moving to immutable allows you to take advantage of some cool release strategies. The first one is a rolling release, where nodes are replaced one at a time. This can be handy when you have state on your cluster that needs to be preserved.

My favorite release strategy is the Blue/Green push. You have 100 nodes, you start up 100 new nodes, and then you move traffic to the new nodes. If you have issues, you can quickly move traffic back to the old nodes. After a reasonable window, say 1–3 hours, you shutdown the old nodes.

I run check_mk for monitoring some servers. Currently, the check_mk host uses ssh connections to acquire the data from the check_mk monitored host.

Journey into the SaltMine to keep nagios fed with check_mk shows some ways of not using ssh, but to use minion/master interactions to capture the data. I am leaning towards revisiting this by using SaltStack's inotify beacon to signal captured file changes, which then trigger events and orchestration to transfer the data from the minion/host to the check_mk/monitor. And I think it can be done in a way such that the salt master doesn't necessarily need to reside on the check_mk monitor. [as a note, the article shows some file locking mechanisms which might come in handy when I try to tackle this].

But, first, I wanted to prove the theory in a different scenario. This example uses three hosts:

• monitored host, which is running the salt-minion, and on to which the check_mk monitoring agent is to be installed,
• monitoring host, which is check_mk, and also has a salt-minion installed, and the
• salt-master, which controls the state and interactions between hosts

The monitoring host will use ssh to connect to the monitored host and access the agent. During the first ssh session, a manual intervention is typically required to confirm usage of the destination's public host key, which then goes into the ~/.ssh/known_hosts file. '-o StrictHostKeyChecking=no' could be used as a simple work-around, but is not very security conscious. Instead, I came up with a series of SaltStack events and states to get the monitored host's public key into the monitoring host's known_hosts file.

There are a number of key sets in use:

• When check_mk connects to an agent via ssh, it will typically use a local private key, and will require a shared public key in the monitored host's ~/.ssh/authorized_keys file. I use SaltStack states and pillars to distribute and install the public key, and make use of the "command="/usr/bin/check_mk_agent" option in the authorized_keys file
• Each host has a unique public/private key. SSH uses this to prevent man in the middle attacks, and to ensure the host hasn't changed. This blog entry is about getting this monitored host's public key into the monitoring host's known_hosts file.

I had thought that the Reactor engine could use pillar states when running states for minions. It may or may not have worked in the future. For the present, it is confirmed as not working: Pillars not available to reactor states #42762. This refers to another issue which supposedly resolves everything in v2017.7.1: Only force saltenv/pillarenv to be a string when not None #42433.

While discovering this issue, I have been thinking about ways to maintain temporary variables somewhere. I first looked at Thorium, but it doesn't seem to have a very flexible two-step look up system, as I need to access variables based upon minion_id, and service type.

The consul pillar state seems to be used only on the master, but I am thinking something should be available on the minion, or accessible from the minion. I thought of the sqlite state for this, but might be a bit heavy duty.

While researching this issue, I came across other interesting Salt articles, which I book mark here, for possible future reference:

• stretch repository is missing packages #42715: this is interesting because in order to load 2017.7.1, I would have to obtain it from the SaltStack repository (as debian doesn't have this yet), and since I run Stretch and Buster, this might cause some issues.
• SaltStack-able Microservices Architecture: Beating the challenge with SaltStack, upstart, consul, consul-template, logstash, etc. An interesting article as it introduces the concept of a jinja macro.
• Combine multiple pillar files under a single key: uses the 'multiple app needs logrotate entries' problem to introduce the concept of merge, os_family_map (which has another issue in github), and jinja 'do' statements.
• Salt consul pillar list of dicts #34458: not necessarily directly relevant, but does have a link to the YAML v1.2 Specifications.
• Using Consul as an External Pillar Source: a simple, detailed way of using Consul for pillar based data for common data as well as minion specific private data.
• Two entries here from the same blogger: Create self-managing servers with Masterless Minions and Using salt-ssh to install Salt, with a bonus reference: Getting redundancy and scalability with multiple master servers"
• Managing your computers with a Salt Master and Git.
• Some notes on using SaltStack: centers the discussion around roles.
• Using Consul as an External Pillar Source
• SaltStack orchestration with Beacons and Reactors
• Thorium and Salt API chapter extract from a book called "Mastering SaltStack".
• Network Automation at Scale: Up and Running in 60 Minutes: an excellent tutorial on orchestration at scale, from a RIPE 74 presentation.
• Salt-based secret storage solution - many of the chapters are incomplete, but this has content.

Dustin Spinhirne's OVN tutorial was recently mentioned on the OVS mailing list. His examples use a single host in an ESXi environment with virtual hosts simulated through namespace commands.

As an excuse to try out Vagrant automation skills, I created a Vagrant multi-host test environment running in VirtualBox. Interfaces are automatically created, ip addresses automatically assigned, and guests can have readable names. Previous articles here explain how I built a recent custom kernel and how I built the latest version of OVS/OVN. The resulting kernel and packages are used to spool up this Vagrant/VirtualBox test environment for playing with OVN.

OVS/OVN Vagrant on GitHub is an environment I have for building with current kernels, generating current OVS/OVN packages, and using them in an OVN Lab type environment. Continue reading "VirtualBox/Vagrant Test Environment for Dustin..." »

A little bit more automation: with each new kernel version that comes out, I need to update my base Vagrant image, and as a result, update the VirtualBox Guest Additions as well.

Here is my Vagrantfile to control the process. The VM is started three times, of which only two are really necessary (the first two). These are the steps: Continue reading "Multiple Provisioners in Vagrant" »

I have been using Salt for a number of infrastructure automation projects. Still a lot to learn. For me, it is an excellent 'as built' documentation tool. Not to mention the time saving and repeatability of building, rebuilding, and supporting infrastructure.

Some links I use, and want to keep around:

• rest_cherrypy: A REST API for Salt
• kitchen-salt: A Test Kitchen Provisioner for Salt. The provider works by generating a salt-minion config, creating pillars based on attributes in .kitchen.yml and calling salt-call. This provisioner is tested with kitchen-docker against CentOS, Ubuntu, and Debian.
• Salt Community Projects: The one currently at the top of the list: HubbleStack. "... on-demand profile-based auditing, real-time security event notifications, automated remediation, alerting and reporting"

Having gone through the exercise of getting Vagrant to build machines with multiple interfaces consistently, I am now ready for the next step.

The next step is being able create multiple interconnected machines. I tested the basic configuration of two machines in a previous blog entry. I could continue on with manually defining machines and their relationships, which becomes a lot of manual editing and book-keeping.

Instead, I wanted to be able to see if it could be done with some simple lists and some code. So... having not programmed in Ruby before (upon which Vagrant relies), I picked up the third edition of 'Beginning Ruby', took a read through, figured out the basics of the language, and was able to successfully program an auto-build configuration.

Two sets of definitions are required. A machine list, and a list of connections between each machine. My test example:

# a list of machines to create within VirtualBox
machineNames = ['edge01','edge02','core01','core02','host01']

# a list of connections to be made between machines
#   this example makes use of a series of point to point links
vlans = [
  ['edge01','core01'],['edge01','core02'],
  ['edge02','core01'],['edge02','core02'],
  ['core01','core02'],
  ['host01','core01'],['host01','core02']
  ]

In the example, there are a number of links, and each link is defined by identifying the machines taking part. For example, one link contains edge01 and core01. This link will have an ip subnet automatically assigned and will have a unique VirtualBox internal network.

By changing the machineNames list and the vlans list, a network of desired topology will be provisioned enabled with the 'vagrant up' command.

Here is the full Vagrantfile listing for the example:

# -*- mode: ruby -*-
# vi: set ft=ruby :

require 'ipaddr'

# a list of machines to create within VirtualBox
machineNames = ['edge01','edge02','core01','core02','host01']

# a list of connections to be made between machines
#   this example makes use of a series of point to point links
vlans = [
  ['edge01','core01'],['edge01','core02'],
  ['edge02','core01'],['edge02','core02'],
  ['core01','core02'],
  ['host01','core01'],['host01','core02']
  ]

# track the vlans to be used on each machine
class Machine
  attr_accessor :name, :interfaces
  def initialize(name)
    @name = name
    @interfaces = []
  end
end

# create each machine with an empty set of interfaces
machines = {}
machineNames.each do |name|
  puts 'machine name exists: #{name}, over-writing' if machines.has_key?(name)
  machines[name] = Machine.new( name )
end

# the subnet isn't used at the moment, but the class does provide a sequence of addresses
class IPAddressing
  attr_accessor :subnet
  def initialize(subnet)
    @subnet = subnet
    @nextAddress = subnet.succ  # don't use .1 as Vagrant emits warning messages
  end
  def next
    @nextAddress = @nextAddress.succ
  end
end

# assign vlan names and ip address ranges for each connection
ipAddresses = {}

ixNet = 101 #provide each link with a unique number
vlans.each do |vlan|
  net = "vlan#{ixNet}"
  ipAddresses[net] = IPAddressing.new( IPAddr.new "10.0.#{ixNet}.0/24" )
  vlan.each do |endPoint|
    if machines.has_key?(endPoint)
      machines[endPoint].interfaces << net
    else
      puts "endPoint #{endPoint} doesn't exist" if !machines.has_key?(endPoint)
    end
  end
  ixNet += 1
end

# generate connectivity information
maxConnections = 0
machines.values.each do |machine|
  maxConnections = (maxConnections > machine.interfaces.length) ? maxConnections : machine.interfaces.length
  out = "#{machine.name}: "
  out += machine.interfaces.join(', ')
  puts out
end

puts "maxConnections: #{maxConnections}"

Vagrant.configure("2") do |config|

  config.vm.box = "stretch"

  config.vm.box_url = "file://boxes/stretch.box"

  # ensure all connections are virtio
  config.vm.provider "virtualbox" do |v|

    cnt = 1; while ( cnt <= (maxConnections + 1) ) # the extra one is for the NAT interface
      v.customize ["modifyvm", :id, "--nictype#{cnt}", "virtio"]
      cnt += 1
    end
  end

  # provision the machines, NAT is automatically provisioned, and is not listed
  machines.values.each do |machine|
    config.vm.define machine.name do |instance|
      instance.vm.hostname = machine.name
      machine.interfaces.each do |interface|
        ipAddress = ipAddresses[interface].next.to_string
        instance.vm.network :private_network, virtualbox__intnet: interface, ip: ipAddress
        puts "#{machine.name}: #{interface} = #{ipAddress}"
      end
    end
  end

end

Here is the initial part of the output during 'vagrant up':

$ vagrant up
edge01: vlan101, vlan102
edge02: vlan103, vlan104
core01: vlan101, vlan103, vlan105, vlan106
core02: vlan102, vlan104, vlan105, vlan107
host01: vlan106, vlan107
maxConnections: 4
edge01: vlan101 = 10.0.101.2
edge01: vlan102 = 10.0.102.2
edge02: vlan103 = 10.0.103.2
edge02: vlan104 = 10.0.104.2
core01: vlan101 = 10.0.101.3
core01: vlan103 = 10.0.103.3
core01: vlan105 = 10.0.105.2
core01: vlan106 = 10.0.106.2
core02: vlan102 = 10.0.102.3
core02: vlan104 = 10.0.104.3
core02: vlan105 = 10.0.105.3
core02: vlan107 = 10.0.107.2
host01: vlan106 = 10.0.106.3
host01: vlan107 = 10.0.107.3
Bringing machine 'edge01' up with 'virtualbox' provider...
Bringing machine 'edge02' up with 'virtualbox' provider...
Bringing machine 'core01' up with 'virtualbox' provider...
Bringing machine 'core02' up with 'virtualbox' provider...
Bringing machine 'host01' up with 'virtualbox' provider...
... more stuff follows as the machines are built ....

After having gone through the basics of a Vagrant install, with more to do yet, I need to record some of the commands I used to get things going.

This solution is based upon using Debian Stretch/Testing as the 'box'. Which I build in a minimal configuration. I create a 'vagrant' user during the build process. Also ensure that the first network card is of type 'Paravirtualized Network (virtio-net)' when being built under VirtualBox.

Once built, and rebooted, login as 'vagrant':

mkdir /home/vagrant/.ssh
chmod 700 /home/vagrant/.ssh
pushd /home/vagrant/.ssh
wget --no-check-certificate \
  'https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub' \
  -O authorized_keys
chmod 600 /home/vagrant/.ssh/authorized_keys
chown -R vagrant /home/vagrant/.ssh
popd

In order to clear the way for vagrant automation, create the following file, as root (sudo doesn't appear to work):

cat <<EOT >> /etc/sudoers.d/10-sudo-overrides
Defaults      env_keep += "GIT_*"
Defaults>root env_keep += SSH_AUTH_SOCK
Defaults>root !requiretty
%sudo         ALL=NOPASSWD: ALL
EOT

VirtualBox extensions are required. They can be installed in one of two ways. The first is by using the gui to mount the cd and then run:

apt-get install linux-headers-$(uname -r) build-essential
# attach guest extensions
mount /dev/cdrom /media/cdrom
KERN_DIR=/usr/src/linux-headers-$(uname -r)/ \
  sh /media/cdrom/VBoxLinuxAdditions.runsh /media/cdrom/VBoxLinuxAdditions.run
umount /media/cdrom
apt-get remove binutils build-essential bzip2 cpp cpp-5 cpp-6 dpkg-dev fakeroot \
   g++ g++-6 gcc gcc-5 gcc-6 libalgorithm-diff-perl libalgorithm-diff-xs-perl \
  libalgorithm-merge-perl libasan2 libasan3 libatomic1 libc-dev-bin libc6-dev \
  libcc1-0 libcilkrts5 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-5-dev \
  libgcc-6-dev libgomp1 libisl15 libitm1 liblsan0 libmpc3 libmpfr4 libmpx0 libmpx2 \
  libperl5.24 libquadmath0 libstdc++-6-dev libtsan0 libubsan0 \
  linux-compiler-gcc-5-x86 linux-headers-4.7.0-1-amd64 \
  linux-headers-4.7.0-1-common linux-kbuild-4.7 \
  linux-libc-dev make manpages manpages-dev patch perl

The other way, courtesy of Creating a Base Box (and substituting the correct iso version):

wget http://download.virtualbox.org/virtualbox/4.3.8/VBoxGuestAdditions_4.3.8.iso
sudo mkdir /media/VBoxGuestAdditions
sudo mount -o loop,ro VBoxGuestAdditions_4.3.8.iso /media/VBoxGuestAdditions
sudo sh /media/VBoxGuestAdditions/VBoxLinuxAdditions.run
rm VBoxGuestAdditions_4.3.8.iso
sudo umount /media/VBoxGuestAdditions
sudo rmdir /media/VBoxGuestAdditions

Minimize the resulting box by removing packages and zeroing out unused space:

apt-get autoremove
apt-get purge
apt-get clean

dd if=/dev/zero of=/EMPTY bs=1M
rm -f /EMPTY

Shutdown the template. Remove the boot cd if one is attached. Ensure the first network port is of type NAT.

Inside a vagrant project directory, I create a template file, it has a default config, but if I need to diddle adjustments later, it is available:

# stretch.vf:
Vagrant::Config.run do |config|
  config.ssh.username = "vagrant"
end

I create a package, a place it in place I can reach it. It is local, but could be put in a remote repository if desired.

vagrant package --base stretch --vagrantfile stretch.vf
mkdir boxes
mv package.box boxes/stretch.box

These commands are the key to a re-usable environment:

vagrant init stretch file://boxes/stretch.box
vagrant up
vagrant ssh

Some other useful commands:

vagrant reload (after changing Vagrantfile), page 29
vagrant halt
vagrant suspend
vagrant destroy
vagrant box list
vagrant box remove 

And as a bonus check, once ssh'd into a guest, and a suitable port forward is in place, then check communication (without having to install a web server):

sudo python -m SimpleHTTPServer 80

Some Vagrant documentation:

• Vagrant by HashiCorp
• Friends of Vagrant

In my previous post, I indeed do VirtualBox networking the hard way. I also hinted that there was a simpler solution. I tried out the alternate solution, and was successful.

The solution is to use VirtualBox Paravirtualized Network (virtio) drivers. They appear to be applied in ascending order to the PCI bus, and are numbered that way:

vagrant@test01:~$ dmesg | grep eth
[    4.444193] virtio_net virtio0 enp0s3: renamed from eth0
[    4.454983] virtio_net virtio1 enp0s8: renamed from eth1
[    4.488878] virtio_net virtio2 enp0s9: renamed from eth2

vagrant@test01:~$ lspci
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:08.0 Ethernet controller: Red Hat, Inc Virtio network device
00:09.0 Ethernet controller: Red Hat, Inc Virtio network device

This is the Vagrantfile to ensure that additional network interfaces are applied with the appropriate type:

Vagrant.configure("2") do |config|
  config.vm.box = "stretch"

  config.vm.box_url = "file://boxes/stretch.box"

  config.vm.provider "virtualbox" do |v|
    #v.memory = VM_MEM_SIZE
    #v.cpus = VM_CPU_COUNT
    #v.customize ["modifyvm", :id, "--cpuexecutioncap", "#{VM_CPU_CAP}"]
    v.customize ["modifyvm", :id, "--nictype1", "virtio"]
    v.customize ["modifyvm", :id, "--nictype2", "virtio"]
    v.customize ["modifyvm", :id, "--nictype3", "virtio"]
    #v.customize ["storagectl", :id, "--name", "SATA Controller", "--hostiocache", "off"]
  end

  config.vm.define "test01" do |test01|
        test01.vm.hostname = "test01"
        test01.vm.network "private_network", virtualbox__intnet: "net1", ip: "192.168.35.23"
        test01.vm.network "private_network", virtualbox__intnet: "net2", ip: "192.168.45.34"
    end
  config.vm.define "test02" do |test02|
        test02.vm.hostname = "test02"
        test02.vm.network "private_network", virtualbox__intnet: "net1", ip: "192.168.35.24"
        test02.vm.network "private_network", virtualbox__intnet: "net2", ip: "192.168.45.33"

    end
end

With an acknowledgement to github comments for #2389

Link to VirtualBox: Chapter 6 - Virtual Networking

When using Vagrant to provision a test environment where each guest has a single network interface, life is good. However, when running Debian Stretch/Testing as a guest with multiple interfaces, the interactions between Vagrant and VirtualBox networking get a little weird.

As I write this, and thinking ahead a bit, I will need to try the virtio driver and see if I get something different, ie, a better result which doesn't require a work-a-round.

Anyway, fundamentally, VirtualBox, when emulating Intel network adapters, like the PRO/1000 MT, will assign them to PCI bus numbers in high number first. On boot up, these are the default assignments in the guest prior to renaming:

[    4.647350] e1000 0000:00:08.0 eth0: (PCI:33MHz:32-bit) 08:00:27:3f:00:28
[    4.647355] e1000 0000:00:08.0 eth0: Intel(R) PRO/1000 Network Connection
[    5.435048] e1000 0000:00:09.0 eth1: (PCI:33MHz:32-bit) 08:00:27:0e:e3:92
[    5.435051] e1000 0000:00:09.0 eth1: Intel(R) PRO/1000 Network Connection
[    6.097368] e1000 0000:00:11.0 eth2: (PCI:33MHz:32-bit) 08:00:27:22:7d:7c
[    6.097372] e1000 0000:00:11.0 eth2: Intel(R) PRO/1000 Network Connection

You can see that the PCI 00:11.0 is labelled as eth2, but is actually 'adapter 1' in the VirtualBox Network configuration list. This gets in the way of Vagrant's requirement of the first interface to be type of NAT. Under Stretch' standard renaming convention, eth0 will be renamed as enp0s8, and eth2 will be renamed as enp0s17. Due to lexical ordering, Vagrant will pick up eth0/enp0s8 as the first interface, which in fact, matches with 'Adapter 3' in the VirtualBox Network list, and therefore, won't be assigned a NAT address, and therefore results in no connectivity between Vagrant and the guest.

Here is a sample Vagrantfile used to generate the example:

Vagrant.configure("2") do |config|
  config.vm.box = "stretch"

  config.vm.box_url = "file://boxes/stretch.box"

  config.vm.define "test01" do |test01|
        test01.vm.hostname = "test01"
        test01.vm.network "private_network", virtualbox__intnet: "net1", ip: "192.168.35.23"
        test01.vm.network "private_network", virtualbox__intnet: "net2", ip: "192.168.45.34"
    end
  config.vm.define "test02" do |test02|
        test02.vm.hostname = "test02"
        test02.vm.network "private_network", virtualbox__intnet: "net1", ip: "192.168.35.24"
        test02.vm.network "private_network", virtualbox__intnet: "net2", ip: "192.168.45.33"

    end
end

The first port is an implied ':forwarded_port'. The two port definitions, one set for each test guest, create VirtualBox internal networks 'net1' and 'net2' on Adapters 2 and 3. When using 'virtualbox__intnet', be sure to put the ip address after 'virtualbox__intnet', otherwise a :hostonly network will be created. The VIRTUALBOX INTERNAL NETWORK documentation could therefore be deemed slightly in-accurate.

I am sure there is a better way to do it, but to do things the hard way, I ended up using udev in Stretch to rename the interfaces to the opposite order. These changes are put in the 'box' file so they are automatically available each time. This requires two changes in /etc/udev/rules.d:

vagrant@test02:~$ ls -alt /etc/udev/rules.d/
total 16
drwxr-xr-x 2 root root 4096 Oct 10 21:13 .
-rw-r--r-- 1 root root  196 Oct 10 21:13 76-net-names.rules
lrwxrwxrwx 1 root root    9 Oct 10 18:05 80-net-setup-link.rules -> /dev/null
-rw-r--r-- 1 root root  134 Oct 10 13:08 60-vboxadd.rules
drwxr-xr-x 4 root root 4096 Oct  7 15:40 ..

The first change is to add an 'over-ride the default' link to /dev/null for 80-net-setup-link.rules:

ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules

The default for that rule is at:

vagrant@test02:~$ sudo find / -name 80-net-setup-link.rules -print
/etc/udev/rules.d/80-net-setup-link.rules
/lib/udev/rules.d/80-net-setup-link.rules

You can see the original in /lib/udev/rules.d, with the over-ride in /etc/udev/rules.d.

The second change is with the content of /etc/udev/rules.d/76-net-names.rules:

vagrant@test02:~$ cat /etc/udev/rules.d/76-net-names.rules
SUBSYSTEM=="net", ACTION=="add", ATTR{ifindex}=="2", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", ATTR{ifindex}=="3", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", ATTR{ifindex}=="4", NAME="eth0"

Because the ifindex is hardcoded, and backwards, the box file needs a special configuration depending upon the number of interfaces to be expected. A bit painful. According to

 man /usr/share/doc/udev/README.Debian.gz

the following command is needed to update configurations:

update-initramfs -u

To obtain the ifindex attribute, this command proved useful (shown on a one interface system [the original box]):

root@stretch:/home/vagrant# apt-get install sysfsutils

root@stretch:/home/vagrant# systool -c net -v
Class = "net"

  Class Device = "eth0"
  Class Device path = "/sys/devices/pci0000:00/0000:00:11.0/net/eth0"
    addr_assign_type    = "0"
    addr_len            = "6"
    address             = "08:00:27:22:7d:7c"
    broadcast           = "ff:ff:ff:ff:ff:ff"
    carrier_changes     = "2"
    carrier             = "1"
    dev_id              = "0x0"
    dev_port            = "0"
    dormant             = "0"
    duplex              = "full"
    flags               = "0x1003"
    gro_flush_timeout   = "0"
    ifalias             =
    ifindex             = "2"
    iflink              = "2"
    link_mode           = "0"
    mtu                 = "1500"
    netdev_group        = "0"
    operstate           = "up"
    proto_down          = "0"
    speed               = "1000"
    tx_queue_len        = "1000"
    type                = "1"
    uevent              = "INTERFACE=eth0
IFINDEX=2"

    Device = "0000:00:11.0"
    Device path = "/sys/devices/pci0000:00/0000:00:11.0"

Some additional systool information:

• systool, a useful tool for SAN, as well for sysfs devices

Another diagnostic command, which shows some additional information, but I couldn't get the 'DEVPATH' to match properly. I don't know how to properly debug commands to find what matches and what doesn't.

root@stretch:/home/vagrant# udevadm info /sys/class/net/eth0
P: /devices/pci0000:00/0000:00:11.0/net/eth0
E: DEVPATH=/devices/pci0000:00/0000:00:11.0/net/eth0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=82545EM Gigabit Ethernet Controller (Copper) (PRO/1000 MT Single Port Adapter)
E: ID_MODEL_ID=0x100f
E: ID_NET_NAME_MAC=enx080027227d7c
E: ID_NET_NAME_PATH=enp0s17
E: ID_OUI_FROM_DATABASE=Cadmus Computer Systems
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=2
E: INTERFACE=eth0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eth0
E: TAGS=:systemd:
E: USEC_INITIALIZED=4238875

One udev command which did show some syntax errors:

root@test01:/home/vagrant# udevadm test /sys/class/pci_bus/0000\:00/device/0000:00:09.0
calling: test
version 231
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

=== trie on-disk ===
tool version:          231
file size:         7055294 bytes
header size             80 bytes
strings            1810454 bytes
nodes              5244760 bytes
Load module index
Found container virtualization none
timestamp of '/etc/systemd/network' changed
Parsed configuration file /lib/systemd/network/99-default.link
Parsed configuration file /etc/systemd/network/30-internet.link
Parsed configuration file /etc/systemd/network/20-internet.link
Parsed configuration file /etc/systemd/network/10-internet.link
Created link configuration context.
timestamp of '/etc/udev/rules.d' changed
Skipping overridden file: /lib/udev/rules.d/80-net-setup-link.rules.
Reading rules file: /lib/udev/rules.d/50-firmware.rules
Reading rules file: /lib/udev/rules.d/50-udev-default.rules
Reading rules file: /lib/udev/rules.d/55-dm.rules
Reading rules file: /lib/udev/rules.d/60-block.rules
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules
Reading rules file: /lib/udev/rules.d/60-drm.rules
Reading rules file: /lib/udev/rules.d/60-evdev.rules
Reading rules file: /lib/udev/rules.d/60-persistent-alsa.rules
Reading rules file: /lib/udev/rules.d/60-persistent-input.rules
Reading rules file: /lib/udev/rules.d/60-persistent-storage-dm.rules
Reading rules file: /lib/udev/rules.d/60-persistent-storage-tape.rules
Reading rules file: /lib/udev/rules.d/60-persistent-storage.rules
Reading rules file: /lib/udev/rules.d/60-persistent-v4l.rules
Reading rules file: /lib/udev/rules.d/60-serial.rules
Reading rules file: /etc/udev/rules.d/60-vboxadd.rules
Reading rules file: /lib/udev/rules.d/64-btrfs.rules
Reading rules file: /lib/udev/rules.d/70-debian-uaccess.rules
Reading rules file: /lib/udev/rules.d/70-mouse.rules
Reading rules file: /lib/udev/rules.d/70-power-switch.rules
Reading rules file: /lib/udev/rules.d/70-touchpad.rules
Reading rules file: /lib/udev/rules.d/70-uaccess.rules
Reading rules file: /lib/udev/rules.d/71-seat.rules
Reading rules file: /lib/udev/rules.d/73-seat-late.rules
Reading rules file: /lib/udev/rules.d/73-special-net-names.rules
Reading rules file: /lib/udev/rules.d/73-usb-net-by-mac.rules
Reading rules file: /lib/udev/rules.d/75-net-description.rules
Reading rules file: /lib/udev/rules.d/75-probe_mtd.rules
Reading rules file: /etc/udev/rules.d/76-net-names.rules
Reading rules file: /lib/udev/rules.d/78-sound-card.rules
Reading rules file: /lib/udev/rules.d/80-debian-compat.rules
Reading rules file: /lib/udev/rules.d/80-drivers.rules
Reading rules file: /lib/udev/rules.d/80-ifupdown.rules
Skipping empty file: /etc/udev/rules.d/80-net-setup-link.rules
Reading rules file: /lib/udev/rules.d/85-hwclock.rules
Reading rules file: /lib/udev/rules.d/90-console-setup.rules
Reading rules file: /lib/udev/rules.d/99-systemd.rules
rules contain 24576 bytes tokens (2048 * 12 bytes), 11090 bytes strings
1545 strings (19300 bytes), 982 de-duplicated (8774 bytes), 564 trie nodes used
IMPORT builtin 'hwdb' /lib/udev/rules.d/50-udev-default.rules:15
RUN 'kmod load $env{MODALIAS}' /lib/udev/rules.d/80-drivers.rules:5
created db file '/run/udev/data/+pci:0000:00:09.0' for '/devices/pci0000:00/0000:00:09.0'
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:09.0
DRIVER=e1000
ID_MODEL_FROM_DATABASE=82540EM Gigabit Ethernet Controller (PRO/1000 MT Desktop Adapter)
ID_PCI_CLASS_FROM_DATABASE=Network controller
ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
ID_VENDOR_FROM_DATABASE=Intel Corporation
MODALIAS=pci:v00008086d0000100Esv00008086sd0000001Ebc02sc00i00
PCI_CLASS=20000
PCI_ID=8086:100E
PCI_SLOT_NAME=0000:00:09.0
PCI_SUBSYS_ID=8086:001E
SUBSYSTEM=pci
USEC_INITIALIZED=3699833
run: 'kmod load pci:v00008086d0000100Esv00008086sd0000001Ebc02sc00i00'
Unload module index
Unloaded link configuration context.
root@test01:/home/vagrant# ls /run/udev/
control           data/             link.dvd          links/            static_node-tags/ tags/             watch/
root@test01:/home/vagrant# ls /run/udev/links
\x2fcdrom
\x2fdisk\x2fby-id\x2fata-VBOX_CD-ROM_VB2-01700376
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0-part1
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0-part2
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0-part5
\x2fdisk\x2fby-partuuid\x2ff3491df3-01
\x2fdisk\x2fby-partuuid\x2ff3491df3-02
\x2fdisk\x2fby-partuuid\x2ff3491df3-05
\x2fdisk\x2fby-path\x2fpci-0000:00:01.1-ata-2
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1-part1
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1-part2
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1-part5
\x2fdisk\x2fby-uuid\x2f326eca33-565a-4e9c-990d-f3393fde3b43
\x2fdisk\x2fby-uuid\x2fdba71b3f-d578-48b0-99f9-c2cc9945760b
\x2fdvd
\x2finput\x2fby-id\x2fusb-VirtualBox_USB_Tablet-event-mouse
\x2finput\x2fby-id\x2fusb-VirtualBox_USB_Tablet-mouse
\x2finput\x2fby-path\x2fpci-0000:00:04.0-event-mouse
\x2finput\x2fby-path\x2fpci-0000:00:04.0-mouse
\x2finput\x2fby-path\x2fpci-0000:00:06.0-usb-0:1:1.0-event-mouse
\x2finput\x2fby-path\x2fpci-0000:00:06.0-usb-0:1:1.0-mouse
\x2finput\x2fby-path\x2fplatform-pcspkr-event-spkr
\x2frtc

With some information supplied at:

• Debugging udev rules
• How to debug an udev rule (in /etc/udev/rules.d/…)
• How to check if a udev rule fired: which might be useful to see the properties when a network card is hot wired into a guest.
• writing udev rules: some useful background information
• opensuse: udev rule not triggered - good troubleshooting examples
• udev debugging toolbox: interesting command: 'udevadm info -e'
• wiki with lots of tricks
• 'man udev.conf' and adjust for more debug info, which should end up in syslog
• Scripting with udev
• Debian's Wiki on udev
• PacketPusher's comments on udev

From debugging udev rules, running udev in debug mode:

user@ubuntu:~$ sudo service udev stop
user@ubuntu:~$ sudo /lib/systemd/systemd-udevd --debug