Networks

Following on from my previous post on building a FRR package series, this provides the process for installing an FRR (Free Range Routing) package. Some content is taken from Building_FRR_on_Debian8.md.

# update/upgrade existing packages
sudo apt update
sudo apt upgrade

# install a few necessary packages
sudo apt install iproute libc-ares2  libjson-c3

# install frr (users and groups installed via the package)
sudo dpkg -i frr_3.1-dev_amd64.deb

# create empty frr configuration files
sudo install -m 755 -o frr -g frr -d /var/log/frr
sudo install -m 775 -o frr -g frrvty -d /etc/frr
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/zebra.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/bgpd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ospfd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ospf6d.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/isisd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ripd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ripngd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/pimd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ldpd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/nhrpd.conf
sudo install -m 640 -o frr -g frrvty /dev/null /etc/frr/vtysh.conf

Enable routing by setting the following in /etc/sysctl.conf, then running 'sysctl -p'

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Configuration files can then be edited as appropriate, then the services started with:

sudo systemctl start frr

Basic instructions for building Debian packages for FRR (Free Range Routing), a fork of Quagga, can be found in the github repository as Building FRR on Debian 8 from Git Source.

# update/upgrade existing packages
apt update
apt upgrade

# install necessary packages
apt install \
 libsystemd-dev \
 linuxdoc-tools \
 build-essential \
 debhelper \
 git autoconf automake libtool make gawk \
 libreadline-dev texinfo libjson-c-dev pkg-config bison flex \
 python-pip libc-ares-dev python3-dev

# install newer pytest (>3.0) from pip
pip install pytest

# add frr groups and user (not sure if this step is actually needed)
addgroup --system --gid 92 frr
addgroup --system --gid 85 frrvty
adduser --system --ingroup frr --home /var/run/frr/  \
  --gecos "FRR suite" --shell /bin/false frr
usermod -a -G frrvty frr

# download source for master
git clone https://github.com/FRRouting/frr.git frr

# prepare build files
cd frr
./bootstrap.sh

# invoke debian package builder
dh binary-arch

As of this writing, the following packages are built, with the first being the one of interest:

frr_3.1-dev_amd64.deb
frr-dbg_3.1-dev_amd64.deb
frr-dbgsym_3.1-dev_amd64.deb

gcc -v for debian/stretch yields:

gcc version 6.3.0 20170516 (Debian 6.3.0-18)

From demystifying openvswitch (with the slide deck showing some interesting edit shortcuts with virsh, and slide 13 may be incorrect as trunk is default):

• ovs-vsctl: Set up, maintain, and inspect various openvswitch configurations. It provides a high level interface to the database to query and apply changes at run time. Operations are persistent across reboot. Example: ovs-vsctl --format=table --column=name,vlan_mode
• ovs-ofctl, ovs-dpctl: Administer and monitor flow entries. Two kinds of flows: openflows (flows managed at control plane) and datapath (kernel flow, cached version of openflow). ovs-ofctl speaks to openflow module. ovs-dpctl talks to kernel module.
• ovs-appctl: Sends commands to a running openvswitch to gather information which is not directly exposed to ovs-ofctl -- the Swiss Army knife of openflow troubleshooting. Example: ovs-appctl vlog/list, ovs-appctl vlog/set module(:facility(:level)) where level is emer, err, warn, info, dbg
• ovsdb-client: query the ovs database. Example: ovsdb-lient list-tables, ovsdb-client list-columns , ovsdb-client monitor --detach
• ovsdb-tool showlog

openvswitch deep dive on slide 14 and 15, shows relationship between commands and the daemons.

Built in to iproute2:

~$  ip tcp_metrics
192.168.4.2 age 336273.112sec source 192.168.4.1
10.151.0.10 age 17668.708sec cwnd 10 rtt 969us rttvar 788us source 10.9.2.21
10.152.0.10 age 8.368sec source 10.9.2.21

sysstat package (edit /etc/default/sysstat and change ENABLED from false to true, and change /etc/cron.d/sysstat to a different interval)

to look into:
• sysstat package: sar

netstat

netstat -s|grep -i retran
    19448 segments retransmitted
    161 times recovered from packet loss due to fast retransmit
    Detected reordering 3 times using reno fast retransmit
    1 timeouts after reno fast retransmit
    303 fast retransmits
    1 retransmits in slow start
    TCPRetransFail: 167
    TCPSynRetrans: 13966

More stats than you can shake a stick at: ss -ti

Papers on TCP:

• TCP Congestion Control on RTP Media Streams: using ffmpeg, tcptrace, mininet, wanem
• TCP Tuning Techniques for High-Speed Wide-Area Networks: presentation
• Identification of Hostile TCP Traffic Using Support Vector Machines
• SLAC Network Monitoring Tools: large list of curated tools

A couple of interesting email list articles regarding libopenvswitch. A recent one titled libopenvswitch in C++, which indicates there are some caveats: most of all the word 'public' used which is a c++ keyword.

That article put me onto a search of other libopenvswitch articles, and this one came up: creating a libopenvswitch-dev package. Which suggested there is quite a bit of work involved in producing a cohesive library.

In that article was a link to a detailed paper from 2011 by Intel called How to Write Shared LibrariesK. The paper covers the ELF structure, dynamic loader, the relocation process, as well as many C/C++ specific requirements. The biggest reason for this link is having to do with the ABI (Application Binary Interface) and it's stability.

2017/07/31 - Open source at work, things getting resolved quickly: a patch series

I write as co-chair of v6ops, under our charter element "comment to other working groups when it seems appropriate".

One outcome from at least v6ops and I think a couple of other WGs and RGs this week - may I suggest a change to Node Requirements and a corresponding change to the IPv6 Ready Logo?

General Category: "Good grief, it's 2017 for goodness' sake!"

Something that would be helpful in IPv6 deployment would be to turn IPv6 on by default in residential routers. Note that this is not the general case. I can think of routers, whose vendors have C's and J's, and probably H's, in their names, whose default configuration is as an Internet Host. The following would apply to devices that are configured by default as an IP router with routing enabled.

Please add a requirement of the general form:

• "If IPv4 router operation is enabled by default, enable IPv6 router operation by default."

Note that this is not as simple as it might sound. There are at least three configurations that must be allowed for upstream: bridging the ISP downstream and CPE downstream LANs, Address allocation via DHCP IA_NA, and address allocation via SLAAC, and on the CPE downstream LAN(s), address allocation via DHCP IA_NA and SLAAC. BBF TR-124 gives a flowchart for this or RFC 7084 defines the algorithms. The implementation is going to have to enable all three, see which works, and act accordingly.

There may also be considerations for PPPOE: if IPv4 is configured for PPPOE, turn it on for IPv6.

Good Grief. This is 2017 for goodness' sake, and there are implementations that turn IPv6 on by default and work. Do it. Just do it.

-- Fred Baker

With a caveat:

As a separate issue, from an operational point of view, implicit enabling of functionality in one area when it's explicitly enabled in another is something that needs to be handled carefully because otherwise you can end up violating the principal of least astonishment.

-- Nick Hilliard

And some miscellaneous links:

• IPv6 Neighbour Discovery Protocol

Question:

I want to classify/match packets in OVS based on some TCP options, this is currently not supported by OVS but it is supported by netfilter. The support for ConnTrack left me wondering if I could use netfilter to match that field and then use the result of that in OVS.

Answer:

There's no native integration, but I could imagine that if Netfilter ran on the packets first then modified the skb mark field, then OVS ran later on that packet then plausibly you could match on the pkt_mark.

"You need to increase the vlan-limit to be able to match or pop multiple VLAN tags. Otherwise the second tag is parsed as the Ethertype."

e.g.

# ovs-vsctl set Open_vSwitch . other_config:vlan-limit=2

github.com/openvswitch/ovs

ArchLinux has a reference page on a software access point. At the bottom, it references that a non-00 country code is required along with the installation of CRDA (Central Regulatory Domain Agent) for wireless networks is required to run in the 5g range, due to the fact that radars and such need to be detected and non-interfered with. That page also describes a mechanism for a wifi device to be both a client and access-point.

• WiFi hostapd configuration for 802.11ac networks
• Debian WiFi
• Intel Wireless WiFi Link, Wireless-N, Advanced-N, Ultimate-N devices for debian
• RTL8192CU and RTL8188CUS in Station and AP mode at the same time
• Debian access point with Hostapd and RTL8188CUS/RTL8192CU
• Raspberry Pi Into a WiFi Hotspot with Edimax Nano USB EW-7811Un (RTL8188CUS chipset)
• List of Wi-Fi Device IDs in Linux
• aircrack-ng
• Beginners guide to a custom 802.11ac setup: lots of good tricks using "COMPEX WLE900V5-23 miniPCIe module, AR9880, 802.11ac, 3*3MIMO"
• About hostapd
• Turn any computer into a wireless access point with Hostapd
• hostapd: IEEE 802.11 AP, IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator

• command: iw list - to show capabilities of hardware
• command: iw wlan0 info
• command: iw wlan0 link
• command: iw wlan0 station dump
• command: iw reg set {00|BM|US|CA} - set appropriate country code for operating mode
• command: iw get get - to see results
• command: lsusb - to show usb devices command: lspci: - to show pci devices
• command: lspci -vv -s 6e:00.0 - show particular pci device (change the id)
• command: nmcli - network manager command line more info
• command: cat /proc/net/wireless
• command: iwevent
• command: iwgetid
• command: iwconfig - ancient tool

• FreeWTP -- An Open Source CAPWAP WTP : not sure what it does
• wifi access point on same chip by "linux is wonderful"
• ath9k and hostapd hoot: for an older kernel version, but still workable, with a note about making it work with 802.11n.
• strong wifi encryption with hostapd, written by ibm, and discusses "pre-shared key by mac address"

Windows 7,8,9,10 are different animals. I was able to get Android, Mac, Linux and various cameras to associate using WPA2 encryption. Windows needs a particular wpa settings, otherwise things won't work. Troubleshooting Windows is next to impossible with its typical cryptic hex code patterns or this: "couldn't connect'. Windows is the most unhelpful operating system. Back in the early days it was helpful. But now, even for the technically inclined, Windows is a nightmare for troubleshooting corner cases.

Any way, here is a hostapd.conf file working with Windows 10 and the latest drivers (I tested with a couple different Atheros cards). It came down to using particular combinations of WPA parameters, specifically: TKIP and CCMP.

logger_syslog=-1
logger_syslog_level=2
#logger_stdout=-1
#logger_stdout_level=2
#sets the wifi interface to use, is wlan0 in most cases
driver=nl80211
#ieee80211n=1
#ht_capab=[HT40-][SHORT-GI-40][DSSS_CCK-40]
#ieee80211ac=1
#ieee80211d=1
#ieee80211h=1
#wmm_enabled=0
#wme_enabled=1
#country_code=CA
country_code=0
#sets the mode of wifi, depends upon the devices you will be using. It can be a,b,g,n. Setting to g ensures backward compatiblity.
hw_mode=g
#sets the channel for your wifi
channel=0
#macaddr_acl sets options for mac address filtering. 0 means "accept unless in deny list"
macaddr_acl=0
#setting ignore_broadcast_ssid to 1 will disable the broadcasting of ssid
ignore_broadcast_ssid=0
interface=wlp4s0
#sets the ssid of the virtual wifi access point
ssid=yourssid
bssid=02:00:00:00:10:01
#Sets authentication algorithm
#1 - only open system authentication
#2 - both open system authentication and shared key authentication
auth_algs=1
#####Sets WPA and WPA2 authentication#####
#wpa option sets which wpa implementation to use
#1 - wpa only
#2 - wpa2 only
#3 - both
wpa=3
#sets wpa passphrase required by the clients to authenticate themselves on the network
#sets wpa key management
#wpa_key_mgmt=WPA-PSK WPA-EAP WPA-PSK-SHA256 WPA-EAP-SHA256
wpa_passphrase=yourpassphrase
wpa_key_mgmt=WPA-PSK
#sets encryption used by WPA
wpa_pairwise=TKIP CCMP
#sets encryption used by WPA2
rsn_pairwise=CCMP

hostapd docs make mention that camp might need to be removed in order to function with certain windows computers.

It took a little while to determine the root cause, but, ...

I am rebuilding an old windows workstation into a Linux based router/firewall/access-point using hostapd to provide wireless interfaces. Normally, this is a straight-forward configuration. The twist in this case is that I am using Open vSwitch to handle layer 2 functions. The box has an existing Atheros AR5B22 based PCIe 1 card with two antenna connections on the rear face. Which translates into a Qualcomm Atheros AR9462 chipset.

I added the wireless interface to OVS on vlan 10 with a command like:

ovs-vsctl add-port ovsbr0 wlp4s0 tag=10

I used a simple hosted configuration file like:

# cat /etc/hostapd/hostapd.conf
interface=wlp4s0
driver=nl80211
ssid=test
auth_algs=1
wpa=1
#wpa_psk_file=/etc/hostapd/hostapd.psk
wpa_passphrase=testtest
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP TKIP
rsn_pairwise=CCMP

When running in debug mode:

hostapd -d -K -t /etc/hostapd/hostapd.conf

I was seeing EAPoL (Extensible Authentication Protocol over LAN) based retries. It took some research to come across the interaction about openvswitch and hostapd. Bottom line, is that hostapd uses a control channel to control the wireless device. EAPoL is a packet based function. AS such, hostapd needs to monitor the packets to pick out the EAPoL packets. A mechanism is built in for working with regular Linux bridged networking. That code does not work with Open vSwitch. Someone created a patch to work around the issue.

This patch isn't found in mainline hostapd, nor as a patch in the Debian repository. So... I need to apply it manually. I used the documentation found at Debian Building Tutorial as a starting point. The divergence is that the documentation uses a non-functional, deprecated tool called dpatch. I used quilt to handle patching.

As a sidebar, this server I am building uses LXC containers to segregate functionality, compartmentalise security, and to make it easier to keep the main install minimal. As patching and rebuilding the package requires a bunch of build tools, the tools can be easily installed, and then the whole build environment deleted when complete.

The annotated series of steps. I have a number of commented-out entries which I plan to play with later, once I understand more the nuances. One thing, there appears to be a lock out of 5gig frequencies when acting as a host.

apt update
apt install build-essential fakeroot devscripts # install tools
apt-get source hostapd  #obtain source
# review the source directory, then
apt-get build-dep wpa  # install the build dependencies
cd wpa-2.4/
debuild -b -uc -us  # build with existing source
dpkg -i ../hostapd_2.4-1_amd64.deb  # install the package as a test
apt install quilt  # install patch manager
quilt top  # inspect the current latest patch
ls -alt debian/patches/  # most patches kept here
ls -alt patches/  # some are here as well
quilt new rpb_hostapd_openvswitch  # create a new patch
# in the following edit, I removed the content, and pasted the source from
# https://github.com/helmut-jacob/hostapd/blob/master/src/drivers/linux_ioctl.c
quilt edit src/drivers/linux_ioctl.c  # source file to change
quilt refresh  # refresh
quilt top  # my new patch is at the top
cat debian/patches/rpb_hostapd_openvswitch |less  # this is my patch
quilt diff  # show the diff colorized
quilt push  # add the patch to the list of patches
dch -n   # update the changelog and version
debuild -b -uc -us  # build with the new patch
# the patch can then be applied and tested (direct from the lxc container):
dpkg -i /var/lib/lxc/apd/rootfs/usr/src/hostapd_2.4-1.1_amd64.deb

With the patch, clients can now successfully associate and authenticate with hostapd when the wireless port is connected to an Open vSwitch bridge.

hostapd sample configurations:

• Hostapd: from a Gentoo perspective, with an intro to multiple AP and some references to 802.11ac with DFS
• CRDA Regulatory Code - getting hostapd to run in the 5g bands
• Edison AP mode in 5GHz - putting two and two together
• w1.fi hostapd.conf example
• hostapd docs which include some 802.1x suggestions and for running with segregated plans. freeradius is also needed when running 802.1x style configurations as the radius protocol is used for communicating the configurations to end devices.
• simple hostapd/radius config
• openwrt / openvswitch: background information on the hostapd / openvswitch issue
• OpenFlow Isolation: use openflow / openvswitch to isolate wifi networks and users, which is an improvement on the psk per user and 802.1x per user vlan. with ovs-ofctl examples
• MAC address spoofing, and since the pre-shared key in a previous step might be MAC based, here is a way to get around the limitation.
• hostapd and dhcp: simple wifi config
• SDN: Establishing a Session Database for SDN Using 802.1X and Multiple Authentication Resources -- interesting presentation on open flow, 802.1x, sdn, radius, ...

• OpenFlow Header Files
• Revised OpenFlow Library (ROFL) - this looked like an excellent library, but I have problems trying to make the controller and the ethswitchd examples function with out crashing or not reacting as expected. And it uses and integrates it's own packet send/receive functions.
• libtins - packet crafting and sniffing. It would be interesting if the crafting portion was removed from the send/receive portion, which make a good open flow packet encoder/decoder.
• boost graph - max flow/min cut algorithms
• min cost max flow with negative weights
• amedama's github for open flow, with graph code
• runos - another controller, but with lots of other bits added
• logcabin - Raft protocol library
• OpenFlow 1.4.1 spec
• OpenFlow 1.5.1 spec
• beehive - Distributed SDN controller built on top of beehive. Open flow packet definitions base upon:
• Packet - "protocol buffer" for network protocol

Know of any c++ based packet serializers, coders/decoders, to process/build open flow packets?

Raft:

• consensus explained
• raft consensus algorithm

SDN Papers:

• Automatic belief network modeling via policy inference for SDN fault localisation: Springer Open Access paper on - Modeling via Policy Inference (called MPI) is a highly scalable, effective and flexible modeling approach to tackle fault localization challenges in a highly dynamic and agile SDN network.

Other openflow stuff:

• OpenFlow controller programming framework that provides everything needed to create OpenFlow controllers in Ruby
• Case Study with OpenWifi
• OpenFlow 1.3 for OpenWRT

Other not relevant C++ oriented stuff encountered:

• RapidJSON
• optitrack - natural point motion capture streaming library
• NatNet - the SDK
• NetworKit - social graph analysis
• list of cpp libraries
• Erasure Coding for storage devices with a paper
• mFAST - A FAST (FIX Adapted for STreaming) encoder/decoder)

To cut a short story even shorter, in the end I selected Digital Ocean's NetBox IP address management (IPAM) and data center infrastructure management (DCIM) tool. It uses PostgreSQL as it's backing database, which in my mind, was a significant deciding factor for me as it has built in field types to support both IPv4 and IPv6 style ip addresses and CIDRs. MySQL/MariaDB does not have that capability natively. Performing subnet/supernet queries in PostgreSQL is a great way to look for address and subnets easily and efficiently without resorting to third party libraries.

For the longer version of the story, here are some other IPAM tools I encountered in my travels:

• RackTables: a datacenter asset management system seeming to rely on MySQL. But it does seem to be more detail oriented to identifying and locating gear in racks.
• Netdot: NETwork Documentation Tool, a Perl based tool which, as a strength, can perform network scans with CDP/LLDP, switch forwarding tables, and router tables to build it's relationship database. I think it can use either MySQL or PostgreSQL. But I'm not sure if it is maintained much, because as of this date, the last release was 2014/08/12. There is a repository with some updates as of 2016/06/16.
• HaCi is an IP Address / Network Administration Tool with IPv6 support. This seems to be strictly an IP Address manager, with the last indicated revision of 2015/03/04.
• OpenNetAdmin provides a database managed inventory of an IP network. Unique to this product is a command line interface. Also unique, is the ability for tracking DNS records. It does have contexts to allow over-lapping addresses (a multi-tenant situation). But it seems to be oriented more towards MySQL. The features page lists other abilities. It looks featureful, however, the last git commit is listed as 2016/03/16.
• Not quite related, but here for the fact I encountered it: NetHead -- IP Allocation Tool is based on the original RIPE registry software.
• Wikiwand has a larger table oriented listing of IP Address Management tools, both commercial as well as open source, with an indication of databases solutions supported.
• nipap is a "sleek, intuitive and powerful IP address management system built to handle large amounts of IP addresses". Written in Python, using PostgreSQL for a data store, and supports IPv6. So a CLI is not unique thing, as nipap has one as well. VRF management is included.
• phpipam is one of the first IPAM solutions I heard about, but was one of the later ones to build in IPv6 capability. It has matured considerably, and is actively maintained.
• NOC Project is interesting as it knows about QinQ style networks. But it doesn't look like it is currently maintained.
• GestióIP is another address manager, but it doesn't seem to have recent activity either. And no special feature which stands out.
• NetMagis supports IPv4 and IPv6, generates DNS zone files, has an LDAP privilege system, can generate network maps, and has some sort of traffic map capability. The most recent github commit is about five months ago.

In looking for open source routing and open flow solutions, I came across Brian Linkletter's site called Open-Source Routing and Network Simulation. I never realised that there were so many network simulators out there. His site specializes in evaluating many of them.

Two reviews stand out. The first is DNS and BIND demonstration using the Cloonix network emulator which provides a mechanism for experimenting with the DNS protocol and the various open-source implementations of DNS including BIND.

The second interesting network emulator is the OFNet SDN network emulator. it comes prepackaged as an .ova file. The article indicates that it comes with the Floodlight and Beacon controllers. I wonder if Ryu could be used? Anyway, based upon the article, the emulator has visual tools for watching OpenFlow messages, evaluating topologies, and show traffic patterns. Further into the article, the author installs OpenDaylight, so it does look like other controllers can be trialled.

It’s usually best to leverage some sort of “smart” DNS to handle CNAME distribution, giving you the ability to weight your CNAME distribution vs. only using one CDN all the time, or prefer different CDNs in various global regions. I’ve had decent experience with Dyn here, but Route53 has all the features you’d want as well. If possible, write tooling towards your DNS provider’s API to automate your failovers.

Weight your distribution such that you never have one CDN turned off completely; you’ll want a small trickle of user traffic hitting every CDN so that the caches won’t be cold when you switch over to it.

Make sure you have a distributed metrics service (ThousandEyes, WebMetrics, et al) testing your CDNs individually as well as the external hostname.

Stay away from HTML- or Header-munging features when possible; stick with feature sets that are common (and implementable in similar ways) across your providers. (Similar advice goes for multi-vendor *anything*, TBH)

Some folks just let Cedexis do the work: www.cedexis.com