Networks

The Energy Sciences Network has a number articles (plus many other things):

• Linux Tuning which includes TCP tuning entries, as well as congestion control entries.
• Even an entry on 40G/100G Tuning

Etherate code on Github is supplemented with some documentation and comments at 53bits Etherate. Etherate sends traffic directly over Ethernet so it goes without saying that all tests between Tx and Rx hosts are run within the same layer 2 broadcast domain, it can not test over a layer 3 boundary.

TRex Low-Cost, High-Speed Stateful Traffic Generator

MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.

iperf3

NFPA: Network Function Performance Analyzer: is a publicly available, open-source measurement application, which is not only in accordance with standardized methodologies (RFC 2544), but also makes possible to comprehensively compare performance metrics of NFs in an exhaustive range of dimensions. Uses DPDK PktGen.

• TCP Statistic and Analysis Tool
• tcpick: is a textmode sniffer libpcap-based that can track, reassemble and reorder tcp streams
• tcpflow a tool similar to tcpick

From a recent netfilter mailing list message:

Logging from network namespaces other than init has been disabled since kernel 3.10 in order to prevent host kernel log flooding from inside a container.

If you have kernel >= 4.11 or one with commit 2851940ffee3 ("netfilter: allow logging from non-init namespaces") backported, you can enable netfilter logging from other network namespaces by

 echo 1 >/proc/sys/net/netfilter/nf_log_all_netns

(the command must be issued from init_net).

Logging via NFLOG target and ulogd2 should work even without the sysctl mentioned above, IIRC.

From the netfilter mailing list:

... connection tracking helpers assignation can only be used with nft 0.8 and Linux 4.12.

You can set automatic assignment with:

echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper 

To detect if a local interface is in promiscuous mode, run:

# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0    1500    41831      0      0 0         32796      0      0      0 BMRU
enp3s0    1500        0      0      0 0             0      0      0      0 BMU
enp4s0    1500  2847577      0 308126 0       2433906      0      0      0 BMRU
lo       65536        1      0      0 0             1      0      0      0 LRU
veth1n    1500  1354033      0 2752094 0       1469155      0  72704      0 BMRU

The above command, no interfaces are in promiscuous mode. By setting the mode, and running the check again, the interface is in promiscuous mode with the 'P' flag:

# ip link set dev enp4s0 promisc on
# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0    1500    41871      0      0 0         32833      0      0      0 BMRU
enp3s0    1500        0      0      0 0             0      0      0      0 BMU
enp4s0    1500  2847700      0 308126 0       2434033      0      0      0 BMPRU
lo       65536        1      0      0 0             1      0      0      0 LRU
veth1n    1500  1354033      0 2752094 0       1469161      0  72704      0 BMRU

List of flags:

• B: broadcast
• M: multicast
• P: promisc mode
• R: running
• U: up

While looking at the MACsec information, I came across information about LLDP (Link Layer Discovery Protocol) and how it can be used for packet pacing. This is relevant for me as I found that when doing IPSEC encryption from one device to another, it is easy to overrun the receiver's packet buffers if the packets cannot be filtered and decrypted fast enough.

Mellanox has an article called Quality of Service, which discusses Data Center Bridging (DCB), Priority-based Flow Control (PFC), Pause frames, and shared buffers. They use the open source open-lldp. Their hardware seems to be capable of sending pause frames when buffers get too full, something which would help when slower SoC cpus are processing encrypted packets.

The open-lldp wiki has an article on Data Center Bridging (DCB) on Linux. It has very generic notes. But since there is a lldptool-pfc man page, it appears as though functionality is indeed available in Linux.

When looking at some NetDevConf papers, I opened up MACsec, which is "Encryption for the wired LAN". The biggest advantage illustrated was in a cloud environment where traffic from one guest to another via a third party's infrastructure could be easily and application-transparently encrypted. The slide deck has a number of actual usage examples. Sabrina Dubroca writes an article on the redhat developers blog: MACsec: a different way to encrypt network traffic.

Which, come to think of it, would be a dead simple way to encrypt L2 WAN traffic.

• MACsec Implementation on Linux: an in depth tutorial written by costier ntework engineering.
• MACsec on Linux: another well documented example of using MACsec on Linux.

• OSRG gobgp - BGP implemented in go
• kmotiko openflow in go

From the ovs-dev mailing list:

There are cases where the default gateway of a interface is in a different subnet than its IP address. Linux allows such configuration. For e.g, one could set the IP address of a Linux interface as 172.16.1.2/32 and then give it a default gateway of 172.16.1.1. This can be done for e.g. by running the following commands.

ifconfig eth0 172.16.1.2 netmask 255.255.255.255 broadcast 172.16.1.2 
route add 172.16.1.1 dev eth0 
route add default gw 172.16.1.1

The above configuration is what google cloud uses for its VMs.

In OVN static routes, we currently have the ability to specify the router port via which the packet needs to be pushed out to reach a next hop. But when support for IPv6 was added, we only allowed nexthops to be in the same subnet as one of the router's IP addresses.

This commit relaxes that restriction. When a outport is specified in static routes and when a nexthop is in a different subnet than any of the router IP addresses, we will assume that it is reachable from the first IP address of the router. Since this is a corner case, we just go with the first IP address. If it turns out that there are more cases, we can let users choose the IP address via which the destination is reachable.

From the netdev mailing list:

Ido says:

This patchset aims to prepare the mlxsw driver for VRF offload. The follow-up patchsets that introduce VRF support can be found here: github idosch-next.

The first four patches are mainly concerned with the netdevice notification block. There are no functional changes, but merely restructuring to more easily integrate VRF enslavement.

Patches 5-10 remove various assumptions throughout the code about a single virtual router (VR) and also restructure the internal data structures to more accurately represent the device's operation.

There are many different projects for controlling different aspects of the network in a datacenter. Many of them seem to be more oriented to the hypervisor side of things: how to get containers to talk to each other across a network. Those projects seem to make the assumption that a 'core network' already exists. Which implies that a network is partitioned into management by (at least) two groups of people?

What sort of solutions exist for network management end to end -- ie, being able to provision container networking, distributed networking within the hypervisor, routing/switching in the core, connectivity to the edge/exterior world? This would also include the ability of distributed security. Rather than central firewalls - security is provided at each entry point: each container/guest interface along with some distributed service function chaining to provide additional layers of security on particular flows.

To understand the challenges more intimately, I have been experimenting with these concepts. For instance, openvswitch is a popular platform for switching in hypervisor environments. Under the hood, it makes use of OpenFlow tables to control packet flows. It has a rich set of matches and actions.

The same development group have recently released OVN (open virtual networks), which provides a distributed abstraction for routing/switching and container provisioning. They have a ways to go before routing becomes a solved problem for resiliency. To me, OVN only solves part of the problem of networking throughout a datacenter infrastructure -- mostly having to do with the hypervisor side of things.

At another style of abstraction, Ryu appears to be a popular controller for OpenFlow style networking, and has an active user base. Not only does it know OpenFlow, it has been extended to handle openvswitch extensions, and even 'knows' how to integrate with the openvswitch ovsdb. Some users have integrated it with NetworkX to handle path calculations through a network.

I was initially tempted to run a single instance of Ryu to control various proof of concept routing/switching/hypervisor instances. The single instance of Ryu would use some graph theory to figure out which network element required which subset of functionality. That doesn't necessarily scale well, nor is it very resilient.

Taking a page from the OVN playbook where OVN has a central controller with agents on the hypervisors, something similar could be performed with Ryu. Rather than a central RYU controller, there would be a Ryu controller on each network element. Their effective SouthBound interface would be a distributed key/value store such as what could be provided by Consul. Each Ryu would calculate it's own participation in the network graph represented in the key/value store adjust local OpenFlow rules to suit. This provides distributed control resiliently with a distributed northbound interface for intent based network provisioning.

As part of it's feature set, Consul can also perform health monitoring and some statistics gathering. These statistics and state management can be used with the local path calculations to perform load-balancing and multi-path planning. Theoretically speaking.

I am wondering if this concept can handle distributed MLAG/LACP and VRRP style resiliency functionality. The distributed key/value store is the mechanism for signalling. Would this work?

Who is working on similar functionality?

• linux multipath routing load balancing
• Load balanced, redundant network configuration for Linux using ECMP, Quagga, BGP and OSPF

To help with my diagnosing of multicast with my article Multicast PIMD, a few links I used:

• pimd debug help
• troglobit multicast how to
• rfc4601: Protocol Independent Multicast - Sparse Mode (PIM-SM)
• Network virtualization with VXLAN
• Troubleshooting Multicast with Linux
• iperf for multicast testing
• omping: test ip multicast
• Enable Multicast Routing in the Kernel

• <vans163> anyone worked with connecting OVS/OVN to physical networks?
• <vans163> for example to have dedicated servers be on private lans with virtual ones
• <vans163> I think a programmable router is needed for this?
• <vans163> Like is this needed barefoot chip?
• <russellb> you can do physical network integration in 4 ways
• <russellb> 1) OVN supports top-of-rack based L2 gateways based on the hardware_vtep ovsdb schema
• <russellb> 2) OVN supports L2 gateways via a linux host running ovn-controller
• <russellb> 3) OVN supports connecting VMs directly to a physical network - you do this by adding a special "localnet" port to a logical switch, which tells OVN that the logical switch is actually direct connectivity to a physical network
• <russellb> 4) OVN supports L3 gateway routers that can route between a virtual, overlay network, and a physical network
• <russellb> vans163: github gist
• <russellb> that's a quick overview of #3
• <vans163> that #3 I assume will connect you to a physical vlan, but you still need something programable that you can have define that VLAN in the physical router/switch (if tagging)
• <russellb> yes, you need to also provision your underlay for that vlan
• <russellb> via whatever means suits you
• <russellb> ovn can't do it
• <russellb> you use ansible by chance?
• <russellb> it has some switch config modules you can use for this sort of thing ...
• <russellb> ansible network automation
• <russellb> but there are no shortage of ways to do this stuff

I am using check_mk to monitor some networks, and wanted to do something similar to what Cisco's IP SLA does, and what SmokePing does: check for a rough measure of latency, loss, and jitter on a network. As check_mk is Python based, I wanted something in Python, which I could retrofit to act as a check_mk agent.

I found some code at: Route Reflector, which may, in turn, may have had some of its origins at ping.py – Python Implementation of the ping command.

According to the notes in the code, the original code has been through a number of authors and iterations.

My iteration allows:

• a different nagios output (for check_mk)
• provide a source ip address and source description
• rrd code commented out, but can be uncommented for use

Under normal circumstances, when doing a wildcard (default) bind, and a packet is sent to a destination ip address, the source address will be chosen based upon which is 'closest' (from a routing table perspective) to the destination.

In my tests, I need to test network quality from a number of interfaces on the same device. Therefore, I need to set the source of address of the ping. In addition, with the source address set, the packet has to egress that same interface (routing will cause the egress to be out the interface 'closest' to the destination).

The solution, on a linux based platform, is to use policy based routing.

For a policy based routing example on a device with two interfaces:

• local interface eth0: 10.1.1.1/30
• other end: 10.1.1.2/30 (the default gateway for this example)
• local interface eth1: 10.1.2.1/30
• other end: 10.1.2.2/30
• destination host: 10.1.3.1

Linux iproute2 commands to provide the appropriate routing:

ip route add 0/0 via 10.1.1.2 
ip route add 0/0 via 10.1.2.2 dev eth1 table 6 
ip route add from 10.1.2.1/32 lookup 6 priority 100

A description of the above statements:

• The first default route is the main system default route.
• The second default route is placed into table 6 as a special lookup.
• Then a policy is added meaning that anything from 10.1.2.1/32 uses table 6 for destionation lookup
• Table 6 has the default route out eth1 for sending traffic originating from 10.1.2.1/32

An example ping would be:

/usr/bin/python ~/ping/ping.py -c 20 -t 1 -n int_eth1 -s 10.1.2.1 -d 10.1.3.1

The ping will go out interface eth1 with the policy-based route in place. Without the policy based route in place, the ping will go out eth0.

The code can be found at: Github, rburkholder, ping.py

More info on policy routing:

• jumanjiman / policy-based-routing
• linux-ip.net: IP Routing
• Policy Routing With Linux - Online Edition by Matthew G. Marsh

Other interesting tools:

• github: Python netlink library — Linux network setup and monitoring
• python: Service Function Chaining API for OpenStack Neutron
• Test driving Policy Based Routing on Linux: uses mininet as a test environment

Some other python ping results:

• Another pure python ping implementation using raw socket.
• Pinging with Python
• Ping, Latency, Jitter, and Packet Loss