Networks

When using the BGP module in Free Range Routing, the 'network ' statement can be used in at least a couple of different roles. One generic role is for generating an advertisement for an aggregate prefix, say to an upstream. Even knowing that, one is also tempted to use a network statement to advertise connected prefixes.

The draw back to advertising connected prefixes is that the prefix is advertised even a related interface is not 'up'. This could lead to a blackhole scenario.

A better way to handle the advertisements of connected prefixes is to use the 'redistribute connected' command.

Even with the use of this command, there may be scenarios (which I need to test at some point) where the prefix is advertised or withdrawn depending upon the link state. Free Range Routing has an additional command which could be used to ensure link state checking: 'bgp network import-check'.

There is more about the Linux state checking flags in the Why Link-State Matters presentation from LinuxCon 2015.

In addition, the Free Range Routing developers have brought together some relevant sysctl settings.

This is another collection of random notes, this time, on how to build something on Linux somewhat resembling Cisco's Global Load Balancing capability, basically a continuation of my entry at Linux ifupdown2 VRRP.

Traditionally, one sets up VRRP using keepalived or the simpler vrrpd. This configuration is typically used when setting up (typically) two routers in an active/passive setup to act as a gateway for a network subnet. In essence, the two (or more) routers negotiate who will hold the gateway mac and ip address.

In other circumstances, it might be desired (and possible) to run active/active. This is a possibility when running containers on a host, and there are similar services running across the hosts. In this instance the same address can be assigned as a secondary address across multiple containers to load balance traffic.

And in even other cases, subnets may be stretched in a layer2 over layer3 encapsulated network across multiple hosts. And in this case, each host should be able to act as a gateway for the traffic local to it. It is this last example I am currently investigating.

Reynold's Blog has an entry called Configuring Cumulus Linux High Availability Layer 2 Network. The most interesting aspect of this post is reference to using 'address-virtual' commands when using ifupdown2 style /etc/network/interface structures:

address-virtual 00:00:5e:00:01:02 10.11.2.254/24

The ip and mac addresses are identical across interfaces sharing the gateway role. The mac address is a reserved range 00:00:5e:00:01:00 – 00:00:5e:00:01:ff for VRRP style operations. The ip address is the virtual ip address (VIP). This style of usage is explained more in Virtual Router Redundancy - VRR.

Or maybe I don't worry about this as Ethernet Virtual Private Network - EVPN has a section with asymmetric routing and symmetric routing which do not need vrrp style constructs.

Layer 3 routing on Cumulus Linux MLAG talks about VRR, the address-virtual, and FRR/route-map to obtain ECMP based load balancing. Now the question - how to get things to not need MLAG.

# netstat -rn -f inet
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.7.28.23      0.0.0.0         UG        0 0          0 vlan28
10.7.1.1        10.7.3.7        255.255.255.255 UGH       0 0          0 enp14s0f2
10.7.1.2        10.7.3.4        255.255.255.255 UGH       0 0          0 enp14s0f3
10.7.1.33       10.7.28.23      255.255.255.255 UGH       0 0          0 vlan28
10.7.1.34       10.7.28.23      255.255.255.255 UGH       0 0          0 vlan28
10.7.1.41       10.7.3.7        255.255.255.255 UGH       0 0          0 enp14s0f2
.....

# ls -l /sys/class/net
total 0
lrwxrwxrwx 1 root root    0 Mar  8 04:06 bond0 -> ../../devices/virtual/net/bond0
-rw-r--r-- 1 root root 4096 Mar  8 04:05 bonding_masters
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f0 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.0/net/enp12s0f0
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f1 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.1/net/enp12s0f1
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f2 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.2/net/enp12s0f2
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f3 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.3/net/enp12s0f3
lrwxrwxrwx 1 root root    0 Mar  8 04:05 enp14s0f0 -> ../../devices/pci0000:00/0000:00:03.2/0000:0e:00.0/net/enp14s0f0
lrwxrwxrwx 1 root root    0 Mar  8 04:05 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx 1 root root    0 Mar  8 04:05 ovsbr0 -> ../../devices/virtual/net/ovsbr0
lrwxrwxrwx 1 root root    0 Mar  8 04:06 ovs-system -> ../../devices/virtual/net/ovs-system
lrwxrwxrwx 1 root root    0 Mar  8 04:05 vlan19 -> ../../devices/virtual/net/vlan19

# cat /sys/class/net/enp8s0f0/statistics/tx_bytes
308623980

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.9.0.0        10.9.2.32       255.255.255.0   UG        0 0          0 enp1s0
10.9.1.1        10.9.2.32       255.255.255.255 UGH       0 0          0 enp1s0

# nstat -a
#kernel
IpInReceives                    156764             0.0
IpInDelivers                    156764             0.0
IpOutRequests                   151083             0.0
IpOutNoRoutes                   40                 0.0
.....

• netstat --statistics
• netstat -s -- protocol stats
• netstat -r -- routing info from kernel
• netstat -g -- mulicast info
• netstat -i -- stats
• netstat -ic -- monitor continuously
• netstat -e -- extra information
• netstat -o -- timer related information
• netstat -p -- associated PID
• netstat -l -- listening sockets
• ss --summary

• nload - console based histogram
• iftop - console based numeric traffic values
• iptraf - An IP traffic monitor that shows information on the IP traffic passing over your network.
• iptraf-ng - is a fork of the original project
• dstat - Combines vmstat, iostat, ifstat, netstat information and more
• sar

Diagnostics
• strace -e open nstat 2>&1 > /dev/null|grep /proc
• strace -e open netstat -s 2>&1 > /dev/null|grep /proc

I was looking through Cumulus ifupdown2 code and came across references to vrrpd and ifplugd. I have been using keepalived before becoming aware of what ifupdown2 can do. I am starting to fathom how things relate 'under the hood'.

Configuring Cumulus Linux High Availability Layer 2 Network – Part 2 introduces vrrp along with 'address-virtual' in the /etc/network/interfaces file:

auto brvlan.10
iface brvlan.10
 address 10.0.10.1/24
 address-virtual 00:00:5e:00:01:01 10.0.10.254/24

The MAC address range is reserved in an RFC, with the last byte adjustable, which now makes sense as to why there could only be 255 vrrp instances. Based upon current thinking, something I need to test, is ifupdown2 sets up ifplugd and vrrpd under the hood to handle migrating the virtual address from interface to interface.

I need to figure out how the configuration above correlates with vrrp specific settings as seen in ifupdown2 vrrpd.py source code.

When evaluating VRRP settings, referring to Configuring, Attacking and Securing VRRP on Linux may be helpful. One useful suggestion for some implementations is to put the virtual address on an untagged interface away from the 'protected' vlans in order to prevent spoofing of the VRRP packets.

Debian packages a very old version of vrrpd. It is best to use a more up-to-date version at fredbcode/Vrrpd

Right up my alley -- someone who is using SaltStack to provision networks. There are some ideas in which I might lift for some of my own work: Building your own sdn with debian linux salt stack and python.

• Netdev 2.1 conference report - an excellent and detailed summary of Netdev 2.1 Conference in Montreal from April 6 to 8 written by Brian Linkletter.
• Interactive map of Linux kernel
• Per-IP rate limiting with iptables -- need to figure out how to do this in nftables

Because they help decipher the operation of ifupdown2, frr, and other routing tools.

• Interface Configuration and Management
• Ethernet Virtual Private Network - EVPN
• Management VRF
• Virtual Router Redundancy - VRR
• Data Center Host to ToR Architecture
• Power over Ethernet - PoE
• Network Virtualization
• Redistribute Neighbor - uses ifplugd
• Equal Cost Multipath Load Sharing - Hardware ECMP
• Segment Routing
• VLAN-aware Bridge Mode for Large-scale Layer 2 Environments
• Data Center Layer 2 High Availability - validated design
• VXLAN Routing - probably associated with FRR symmetric,asymmetric routing

Third party blogs:

• BGP in EVPN-Based Data Center Fabrics - implies that EVPN with eBGP may not 'work out of the box', but seems to be functioning just fine with my implementation of Linux kernel 4.8/4.14, Free Range Routing 4.0.
• Layer 3 Leaf/Spine Fabric with EVPN / VXLAN control - with a note about using the "Common Spine ASN – Discrete Leaf ASN", with eBGP, which results in inherent ECMP (Equal Cost Multi Path) routing. (made possible with FRRouting as the underlying routing engine).
• VRF for Linux where David Ahearn discusses the rationale for adding VRF to the Linux kernel. There is also an entry about listening on ports in a VRF: "gives users and architects a choice: bind a socket per VRF or use a ‘VRF any’ socket. "
• Ifupdown2: Network Interface Manager slideshare from DebConf16, with a discussion of ifquery, policy manager, vrf, ...
• Using the Linux VRF Solution - comprehensive slide deck by David Ahearn on using VRF in Linux: systemd, iproute2, ifupdown2, ...
• Microservice Networking - Leveraging VRF on the host - another David Ahearn slide deck - discusses the use of /32 addresses in a container

Some links I have come across, but nothing will be easy to do

• open-ethernet/mlag: brings in a number of other libraries, and needs a controller to sync states
• MLAG on Linux - lessons learned, from the folks at Cumulus Networks. I downloaded their Cumulus VX to see about extracting the MLAG tools, but their level of integration is... extensive, and therefore hard to extract into something self contained. A paper to go along with the presentation. And a slide show which goes into excellent detail about how packets transition an MLAG solution, and how to stop duplication.

I found out today, via I, Admin, that the Linux kernel, regardless of whether iptables or nftables is installed or not, is tracking connections. That tends to indicate then, that adding rules via nftables will add very little additional overhead. So firewalling a box adds very little additional overhead.

The state of the connection table can be determined by installing:

sudo apt install conntrack

The -L option will show the current state of the table. The -E allows one to watch connection states as they change.

To watch and to sort the state of connections, use:

sudo apt install iptstate

Traffic bandwidth used via particular tcp sessions can be viewed with

sudo apt install tcptrack

In a nutshell, "state" is determined via the data collected by and rules determined by connection tracking. Connection tracking is mostly just a set of timeouts, thresholds, and verifications that help us determine if a packet is "likely" or, ideally "mostly guaranteed" to be part of a known/expected/established layer 4 session. These timeouts, thresholds, and verifications can be seen by catting the various files in /proc/sys/net/netfilter

But back to connection tracking, ... having made the explicit realization that connection tracking is 'always on', and that iptables and nftables only add rule into that path, it becomes easier to realize that network security becomes a very low overhead endeavor. There are very few reason's for not having an 'always on' security posture.

By running

sudo apt install conntrackd

and creating an appropriate conntrackd.conf file, it is possible to build a synchronized, multi-host, active/active firewall solution with little additional effort. conntrackd is designed to replicate the local connection tracking table to other hosts, and to merge other host's connection tracking tables into the local tables.

2018/01/15: Vincent Bernat, in comments below, indicates that "connection tracking is not automatically done. It is done only if the nf_conntrack and/or nf_conntrack_ipv6 modules are loaded (or builtin). This is usually only done by autoloading through the first iptables command". I will need to check that out to confirm from an installation perspective as well as a packet forwarding speed perspective.

The Energy Sciences Network has a number articles (plus many other things):

• Linux Tuning which includes TCP tuning entries, as well as congestion control entries.
• Even an entry on 40G/100G Tuning

2017/11/20 TCP Related: Coping with the TCP TIME-WAIT state on busy Linux servers by Vincent Bernat.

Etherate code on Github is supplemented with some documentation and comments at 53bits Etherate. Etherate sends traffic directly over Ethernet so it goes without saying that all tests between Tx and Rx hosts are run within the same layer 2 broadcast domain, it can not test over a layer 3 boundary.

TRex Low-Cost, High-Speed Stateful Traffic Generator

MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.

iperf3

NFPA: Network Function Performance Analyzer: is a publicly available, open-source measurement application, which is not only in accordance with standardized methodologies (RFC 2544), but also makes possible to comprehensively compare performance metrics of NFs in an exhaustive range of dimensions. Uses DPDK PktGen.

• TCP Statistic and Analysis Tool
• tcpick: is a textmode sniffer libpcap-based that can track, reassemble and reorder tcp streams
• tcpflow a tool similar to tcpick

To detect if a local interface is in promiscuous mode, run:

# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0    1500    41831      0      0 0         32796      0      0      0 BMRU
enp3s0    1500        0      0      0 0             0      0      0      0 BMU
enp4s0    1500  2847577      0 308126 0       2433906      0      0      0 BMRU
lo       65536        1      0      0 0             1      0      0      0 LRU
veth1n    1500  1354033      0 2752094 0       1469155      0  72704      0 BMRU

The above command, no interfaces are in promiscuous mode. By setting the mode, and running the check again, the interface is in promiscuous mode with the 'P' flag:

# ip link set dev enp4s0 promisc on
# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0    1500    41871      0      0 0         32833      0      0      0 BMRU
enp3s0    1500        0      0      0 0             0      0      0      0 BMU
enp4s0    1500  2847700      0 308126 0       2434033      0      0      0 BMPRU
lo       65536        1      0      0 0             1      0      0      0 LRU
veth1n    1500  1354033      0 2752094 0       1469161      0  72704      0 BMRU

List of flags:

• B: broadcast
• M: multicast
• P: promisc mode
• R: running
• U: up

While looking at the MACsec information, I came across information about LLDP (Link Layer Discovery Protocol) and how it can be used for packet pacing. This is relevant for me as I found that when doing IPSEC encryption from one device to another, it is easy to overrun the receiver's packet buffers if the packets cannot be filtered and decrypted fast enough.

Mellanox has an article called Quality of Service, which discusses Data Center Bridging (DCB), Priority-based Flow Control (PFC), Pause frames, and shared buffers. They use the open source open-lldp. Their hardware seems to be capable of sending pause frames when buffers get too full, something which would help when slower SoC cpus are processing encrypted packets.

The open-lldp wiki has an article on Data Center Bridging (DCB) on Linux. It has very generic notes. But since there is a lldptool-pfc man page, it appears as though functionality is indeed available in Linux.

When looking at some NetDevConf papers, I opened up MACsec, which is "Encryption for the wired LAN". The biggest advantage illustrated was in a cloud environment where traffic from one guest to another via a third party's infrastructure could be easily and application-transparently encrypted. The slide deck has a number of actual usage examples. Sabrina Dubroca writes an article on the redhat developers blog: MACsec: a different way to encrypt network traffic.

Which, come to think of it, would be a dead simple way to encrypt L2 WAN traffic.

• MACsec Implementation on Linux: an in depth tutorial written by costier ntework engineering.
• MACsec on Linux: another well documented example of using MACsec on Linux.

• OSRG gobgp - BGP implemented in go
• kmotiko openflow in go