Vagrant, VirtualBox, NAT, and Out Of Order Network Ports

Raymond P. Burkholder - Things I Do

Google Analytics

Google Custom Search


"... an engineer who is not only competent at the analytics and technologies of engineering, but can bring value to clients, team well, design well, foster adoptions of new technologies, position for innovations, cope with accelerating change and mentor other engineers" -- CACM 2014/12

Blog Content ©2022
Raymond Burkholder
All Rights Reserved

+1 (403) 633-7293
+1 (441) 705-7293
raymond@burkholder.net

Available for Contract Work:
* Linux: Debian
* Languages: C++, Python, Perl
* Android: SDK, NDK, Java, C++, Audio
* Virtualization: KVM, LXC, VirtualBox, libvirt, ProxMox VE
* Routing/Switching: FRFouting, Pimd, OpenvSwitch
* Security: nftables, StrongSwan
* Automation: PXEBoot, SaltStack, Vagrant
* Networks: BGP, OSPF, Multicast, VXLAN, TC
* C++ Libraries: Wt, Boost, wxWidgets, ChartDir, HDF5
* Cisco Routing, Switching, MPLS, VPLS, Firewall/VPN
* Network Monitoring: SNMP, Netflow, Nagios, Check_MK
* Database: Cassandra, PostgreSQL, Microsoft SQL

GitHub: rburkholder
LinkedIn
Twitter: @oneunified
Resume

Categories



All categories

Syndicate This Blog

Tuesday, October 11. 2016

Vagrant, VirtualBox, NAT, and Out Of Order Network Ports

When using Vagrant to provision a test environment where each guest has a single network interface, life is good. However, when running Debian Stretch/Testing as a guest with multiple interfaces, the interactions between Vagrant and VirtualBox networking get a little weird.

As I write this, and thinking ahead a bit, I will need to try the virtio driver and see if I get something different, ie, a better result which doesn't require a work-a-round.

Anyway, fundamentally, VirtualBox, when emulating Intel network adapters, like the PRO/1000 MT, will assign them to PCI bus numbers in high number first. On boot up, these are the default assignments in the guest prior to renaming: 

[    4.647350] e1000 0000:00:08.0 eth0: (PCI:33MHz:32-bit) 08:00:27:3f:00:28
[    4.647355] e1000 0000:00:08.0 eth0: Intel(R) PRO/1000 Network Connection
[    5.435048] e1000 0000:00:09.0 eth1: (PCI:33MHz:32-bit) 08:00:27:0e:e3:92
[    5.435051] e1000 0000:00:09.0 eth1: Intel(R) PRO/1000 Network Connection
[    6.097368] e1000 0000:00:11.0 eth2: (PCI:33MHz:32-bit) 08:00:27:22:7d:7c
[    6.097372] e1000 0000:00:11.0 eth2: Intel(R) PRO/1000 Network Connection

You can see that the PCI 00:11.0 is labelled as eth2, but is actually 'adapter 1' in the VirtualBox Network configuration list. This gets in the way of Vagrant's requirement of the first interface to be type of NAT. Under Stretch' standard renaming convention, eth0 will be renamed as enp0s8, and eth2 will be renamed as enp0s17. Due to lexical ordering, Vagrant will pick up eth0/enp0s8 as the first interface, which in fact, matches with 'Adapter 3' in the VirtualBox Network list, and therefore, won't be assigned a NAT address, and therefore results in no connectivity between Vagrant and the guest.

Here is a sample Vagrantfile used to generate the example: 

Vagrant.configure("2") do |config|
  config.vm.box = "stretch"

  config.vm.box_url = "file://boxes/stretch.box"

  config.vm.define "test01" do |test01|
        test01.vm.hostname = "test01"
        test01.vm.network "private_network", virtualbox__intnet: "net1", ip: "192.168.35.23"
        test01.vm.network "private_network", virtualbox__intnet: "net2", ip: "192.168.45.34"
    end
  config.vm.define "test02" do |test02|
        test02.vm.hostname = "test02"
        test02.vm.network "private_network", virtualbox__intnet: "net1", ip: "192.168.35.24"
        test02.vm.network "private_network", virtualbox__intnet: "net2", ip: "192.168.45.33"

    end
end

The first port is an implied ':forwarded_port'. The two port definitions, one set for each test guest, create VirtualBox internal networks 'net1' and 'net2' on Adapters 2 and 3. When using 'virtualbox__intnet', be sure to put the ip address after 'virtualbox__intnet', otherwise a :hostonly network will be created. The VIRTUALBOX INTERNAL NETWORK documentation could therefore be deemed slightly in-accurate.

I am sure there is a better way to do it, but to do things the hard way, I ended up using udev in Stretch to rename the interfaces to the opposite order. These changes are put in the 'box' file so they are automatically available each time. This requires two changes in /etc/udev/rules.d: 

vagrant@test02:~$ ls -alt /etc/udev/rules.d/
total 16
drwxr-xr-x 2 root root 4096 Oct 10 21:13 .
-rw-r--r-- 1 root root  196 Oct 10 21:13 76-net-names.rules
lrwxrwxrwx 1 root root    9 Oct 10 18:05 80-net-setup-link.rules -> /dev/null
-rw-r--r-- 1 root root  134 Oct 10 13:08 60-vboxadd.rules
drwxr-xr-x 4 root root 4096 Oct  7 15:40 ..

The first change is to add an 'over-ride the default' link to /dev/null for 80-net-setup-link.rules: 

ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules

The default for that rule is at: 

vagrant@test02:~$ sudo find / -name 80-net-setup-link.rules -print
/etc/udev/rules.d/80-net-setup-link.rules
/lib/udev/rules.d/80-net-setup-link.rules

You can see the original in /lib/udev/rules.d, with the over-ride in /etc/udev/rules.d.

The second change is with the content of /etc/udev/rules.d/76-net-names.rules: 

vagrant@test02:~$ cat /etc/udev/rules.d/76-net-names.rules
SUBSYSTEM=="net", ACTION=="add", ATTR{ifindex}=="2", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", ATTR{ifindex}=="3", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", ATTR{ifindex}=="4", NAME="eth0"

Because the ifindex is hardcoded, and backwards, the box file needs a special configuration depending upon the number of interfaces to be expected. A bit painful. According to 

 man /usr/share/doc/udev/README.Debian.gz

the following command is needed to update configurations: 


update-initramfs -u

To obtain the ifindex attribute, this command proved useful (shown on a one interface system [the original box]): 

root@stretch:/home/vagrant# apt-get install sysfsutils

root@stretch:/home/vagrant# systool -c net -v
Class = "net"

  Class Device = "eth0"
  Class Device path = "/sys/devices/pci0000:00/0000:00:11.0/net/eth0"
    addr_assign_type    = "0"
    addr_len            = "6"
    address             = "08:00:27:22:7d:7c"
    broadcast           = "ff:ff:ff:ff:ff:ff"
    carrier_changes     = "2"
    carrier             = "1"
    dev_id              = "0x0"
    dev_port            = "0"
    dormant             = "0"
    duplex              = "full"
    flags               = "0x1003"
    gro_flush_timeout   = "0"
    ifalias             =
    ifindex             = "2"
    iflink              = "2"
    link_mode           = "0"
    mtu                 = "1500"
    netdev_group        = "0"
    operstate           = "up"
    proto_down          = "0"
    speed               = "1000"
    tx_queue_len        = "1000"
    type                = "1"
    uevent              = "INTERFACE=eth0
IFINDEX=2"

    Device = "0000:00:11.0"
    Device path = "/sys/devices/pci0000:00/0000:00:11.0"

Some additional systool information:

Another diagnostic command, which shows some additional information, but I couldn't get the 'DEVPATH' to match properly. I don't know how to properly debug commands to find what matches and what doesn't. 

root@stretch:/home/vagrant# udevadm info /sys/class/net/eth0
P: /devices/pci0000:00/0000:00:11.0/net/eth0
E: DEVPATH=/devices/pci0000:00/0000:00:11.0/net/eth0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=82545EM Gigabit Ethernet Controller (Copper) (PRO/1000 MT Single Port Adapter)
E: ID_MODEL_ID=0x100f
E: ID_NET_NAME_MAC=enx080027227d7c
E: ID_NET_NAME_PATH=enp0s17
E: ID_OUI_FROM_DATABASE=Cadmus Computer Systems
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=2
E: INTERFACE=eth0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eth0
E: TAGS=:systemd:
E: USEC_INITIALIZED=4238875

One udev command which did show some syntax errors: 

root@test01:/home/vagrant# udevadm test /sys/class/pci_bus/0000\:00/device/0000:00:09.0
calling: test
version 231
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

=== trie on-disk ===
tool version:          231
file size:         7055294 bytes
header size             80 bytes
strings            1810454 bytes
nodes              5244760 bytes
Load module index
Found container virtualization none
timestamp of '/etc/systemd/network' changed
Parsed configuration file /lib/systemd/network/99-default.link
Parsed configuration file /etc/systemd/network/30-internet.link
Parsed configuration file /etc/systemd/network/20-internet.link
Parsed configuration file /etc/systemd/network/10-internet.link
Created link configuration context.
timestamp of '/etc/udev/rules.d' changed
Skipping overridden file: /lib/udev/rules.d/80-net-setup-link.rules.
Reading rules file: /lib/udev/rules.d/50-firmware.rules
Reading rules file: /lib/udev/rules.d/50-udev-default.rules
Reading rules file: /lib/udev/rules.d/55-dm.rules
Reading rules file: /lib/udev/rules.d/60-block.rules
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules
Reading rules file: /lib/udev/rules.d/60-drm.rules
Reading rules file: /lib/udev/rules.d/60-evdev.rules
Reading rules file: /lib/udev/rules.d/60-persistent-alsa.rules
Reading rules file: /lib/udev/rules.d/60-persistent-input.rules
Reading rules file: /lib/udev/rules.d/60-persistent-storage-dm.rules
Reading rules file: /lib/udev/rules.d/60-persistent-storage-tape.rules
Reading rules file: /lib/udev/rules.d/60-persistent-storage.rules
Reading rules file: /lib/udev/rules.d/60-persistent-v4l.rules
Reading rules file: /lib/udev/rules.d/60-serial.rules
Reading rules file: /etc/udev/rules.d/60-vboxadd.rules
Reading rules file: /lib/udev/rules.d/64-btrfs.rules
Reading rules file: /lib/udev/rules.d/70-debian-uaccess.rules
Reading rules file: /lib/udev/rules.d/70-mouse.rules
Reading rules file: /lib/udev/rules.d/70-power-switch.rules
Reading rules file: /lib/udev/rules.d/70-touchpad.rules
Reading rules file: /lib/udev/rules.d/70-uaccess.rules
Reading rules file: /lib/udev/rules.d/71-seat.rules
Reading rules file: /lib/udev/rules.d/73-seat-late.rules
Reading rules file: /lib/udev/rules.d/73-special-net-names.rules
Reading rules file: /lib/udev/rules.d/73-usb-net-by-mac.rules
Reading rules file: /lib/udev/rules.d/75-net-description.rules
Reading rules file: /lib/udev/rules.d/75-probe_mtd.rules
Reading rules file: /etc/udev/rules.d/76-net-names.rules
Reading rules file: /lib/udev/rules.d/78-sound-card.rules
Reading rules file: /lib/udev/rules.d/80-debian-compat.rules
Reading rules file: /lib/udev/rules.d/80-drivers.rules
Reading rules file: /lib/udev/rules.d/80-ifupdown.rules
Skipping empty file: /etc/udev/rules.d/80-net-setup-link.rules
Reading rules file: /lib/udev/rules.d/85-hwclock.rules
Reading rules file: /lib/udev/rules.d/90-console-setup.rules
Reading rules file: /lib/udev/rules.d/99-systemd.rules
rules contain 24576 bytes tokens (2048 * 12 bytes), 11090 bytes strings
1545 strings (19300 bytes), 982 de-duplicated (8774 bytes), 564 trie nodes used
IMPORT builtin 'hwdb' /lib/udev/rules.d/50-udev-default.rules:15
RUN 'kmod load $env{MODALIAS}' /lib/udev/rules.d/80-drivers.rules:5
created db file '/run/udev/data/+pci:0000:00:09.0' for '/devices/pci0000:00/0000:00:09.0'
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:09.0
DRIVER=e1000
ID_MODEL_FROM_DATABASE=82540EM Gigabit Ethernet Controller (PRO/1000 MT Desktop Adapter)
ID_PCI_CLASS_FROM_DATABASE=Network controller
ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
ID_VENDOR_FROM_DATABASE=Intel Corporation
MODALIAS=pci:v00008086d0000100Esv00008086sd0000001Ebc02sc00i00
PCI_CLASS=20000
PCI_ID=8086:100E
PCI_SLOT_NAME=0000:00:09.0
PCI_SUBSYS_ID=8086:001E
SUBSYSTEM=pci
USEC_INITIALIZED=3699833
run: 'kmod load pci:v00008086d0000100Esv00008086sd0000001Ebc02sc00i00'
Unload module index
Unloaded link configuration context.
root@test01:/home/vagrant# ls /run/udev/
control           data/             link.dvd          links/            static_node-tags/ tags/             watch/
root@test01:/home/vagrant# ls /run/udev/links
\x2fcdrom
\x2fdisk\x2fby-id\x2fata-VBOX_CD-ROM_VB2-01700376
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0-part1
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0-part2
\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VBa68e3639-072af7a0-part5
\x2fdisk\x2fby-partuuid\x2ff3491df3-01
\x2fdisk\x2fby-partuuid\x2ff3491df3-02
\x2fdisk\x2fby-partuuid\x2ff3491df3-05
\x2fdisk\x2fby-path\x2fpci-0000:00:01.1-ata-2
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1-part1
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1-part2
\x2fdisk\x2fby-path\x2fpci-0000:00:0d.0-ata-1-part5
\x2fdisk\x2fby-uuid\x2f326eca33-565a-4e9c-990d-f3393fde3b43
\x2fdisk\x2fby-uuid\x2fdba71b3f-d578-48b0-99f9-c2cc9945760b
\x2fdvd
\x2finput\x2fby-id\x2fusb-VirtualBox_USB_Tablet-event-mouse
\x2finput\x2fby-id\x2fusb-VirtualBox_USB_Tablet-mouse
\x2finput\x2fby-path\x2fpci-0000:00:04.0-event-mouse
\x2finput\x2fby-path\x2fpci-0000:00:04.0-mouse
\x2finput\x2fby-path\x2fpci-0000:00:06.0-usb-0:1:1.0-event-mouse
\x2finput\x2fby-path\x2fpci-0000:00:06.0-usb-0:1:1.0-mouse
\x2finput\x2fby-path\x2fplatform-pcspkr-event-spkr
\x2frtc

With some information supplied at:

From debugging udev rules, running udev in debug mode: 

user@ubuntu:~$ sudo service udev stop
user@ubuntu:~$ sudo /lib/systemd/systemd-udevd --debug
Posted by
Raymond P. Burkholder
in Vagrant at 12:24 | Comments (0)
Trackbacks
Trackback specific URI for this entry
No Trackbacks
Comments
Display comments as (Linear | Threaded)
No comments
Add Comment
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
To leave a comment you must approve it via e-mail, which will be sent to your address after submission.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA 1CAPTCHA 2CAPTCHA 3CAPTCHA 4CAPTCHA 5


 
   
 


Calendar

Back January '25 Forward
Mo Tu We Th Fr Sa Su
Friday, January 24. 2025
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Archives



Blog Links:
Open Access Journals
Linux Insides
Bit Twiddling
Frink
Quote Database
Nanex Research
Sergey Solyanik
Chris Donnan
BeyondVC
lifehacker
Ticker Sense
HeadRush
TraderFeed
The Daily WTF
Guy Kawaski
Steve Pavlina
Matt Cutts
Joel On Software
Julian Dunn
Steve Yegge
ZeroHedge

Reference
VIM Cheat Sheet
TMux Cheat Sheet
GDB Cheat Sheet
Git Branching