Networks

On 06/14/2018 09:22 PM, someone wrote:
> So I have to ask, why is it advantageous to put this in a container 
> rather than just run it directly
> on the container's host? 

Most any host now-a-days has quite a bit of horse power to run services. All those services could be run natively all in one namespace on the same host, or ...

I tend to gravitate towards running services individually in LXC containers. This creates a bit more overhead than running chroot style environments, but less than running full fledged kvm style virtualization for each service.

I typically automate the provisioning and the spool up of the container and its service. This makes it easy to rebuild/update/upgrade/load-balance services individually and enmasse across hosts.

By running BGP within each container, BGP can be used to advertise the loopback address of the service. I go one step further: for certain services I will anycast some addresses into bgp. This provides an easy way to load balance and provide resiliency of like service instances across hosts,

Therefore, by running BGP within the container, and on the host, routes can be distributed across a network with all the policies available within the bgp protocol. I use Free Range Routing, which is a fork of Quagga, to do this. I use the eBGP variant (RFC 7938) for the hosts and containers , which allows for the elimination of the extra overhead of OSPF or similar internal gateway protocol.

Stepping away a bit, this means that BGP is used in a tiered scenario. There is the regular eBGP with the public ASN for handling DFZ-style public traffic. For internal traffic, private eBGP ASNs are used for routing traffic between and within hosts and containers.

With recent improvements to Free Range Routing and the Linux Kernel, various combinations of MPLS, VxLAN, EVPN, and VRF configurations can be used to further segment and compartmentalize traffic within a host, and between containers. It is now very easy to run vlan-less between hosts through various easy to configure encapsulation mechanisms. To be explicit, this relies on and contributes to a resilient layer 3 network between hosts, and eliminates the bothersome layer 2 redundancy headaches with spanning tree and such.

That was a very long winded way to say: keep a very basic host configuration running a minimal set of functional services, and re-factor the functionality and split it across multiple containers to provide easy access to and maintenance of individual services like dns, smtp, database, dashboards, public routing, private routing, firewalling, monitoring, management, ...

There is a higher up-front configuration cost, but over the longer term, if configured via automation tools like Salt or similar, maintenance and security is improved.

It does require a different level of sophistication with operational staff.

Other BGP routing Daemons:

• exabgp - python based
• gobgp - go based
• bird - c based
• bagpipe - python based, in openstack project

Other mailing list comments:

Our use case was both on exporting service IPs as well as receiving routes from ToRs. Exa is more geared towards the former than the latter. Rather then working on getting imports and route installation through Exa, we found it simpler with BIRD exporting the service IP from it bound to a loopback to run local healthchecks on the nodes and then have them yank the service IP from the loopback on failing healthchecks in order to stop exporting.

The intent of the original post was vague. Like a lot of people, I would not run a full BGP router in a container. Now, if the purpose is to inject or learn a handful of routes in order to do limited host routing, I can see the need. A route-server or a looking glass in a container would be fine, or something to perform analysis on the routing table, but not anything that has to route actual traffic.

I use ExaBGP to inject routes, perfect tool for that. If routes have to be received (not my use case) it makes more sense, as stated by previous posts, to use Quagga or BIRD. Which one is better : easy : if you like Cisco better, use Quagga. If you like Juniper better, use BIRD

BIRD looking glass looks very good

When using the BGP module in Free Range Routing, the 'network ' statement can be used in at least a couple of different roles. One generic role is for generating an advertisement for an aggregate prefix, say to an upstream. Even knowing that, one is also tempted to use a network statement to advertise connected prefixes.

The draw back to advertising connected prefixes is that the prefix is advertised even a related interface is not 'up'. This could lead to a blackhole scenario.

A better way to handle the advertisements of connected prefixes is to use the 'redistribute connected' command.

Even with the use of this command, there may be scenarios (which I need to test at some point) where the prefix is advertised or withdrawn depending upon the link state. Free Range Routing has an additional command which could be used to ensure link state checking: 'bgp network import-check'.

There is more about the Linux state checking flags in the Why Link-State Matters presentation from LinuxCon 2015.

In addition, the Free Range Routing developers have brought together some relevant sysctl settings.

This is another collection of random notes, this time, on how to build something on Linux somewhat resembling Cisco's Global Load Balancing capability, basically a continuation of my entry at Linux ifupdown2 VRRP.

Traditionally, one sets up VRRP using keepalived or the simpler vrrpd. This configuration is typically used when setting up (typically) two routers in an active/passive setup to act as a gateway for a network subnet. In essence, the two (or more) routers negotiate who will hold the gateway mac and ip address.

In other circumstances, it might be desired (and possible) to run active/active. This is a possibility when running containers on a host, and there are similar services running across the hosts. In this instance the same address can be assigned as a secondary address across multiple containers to load balance traffic.

And in even other cases, subnets may be stretched in a layer2 over layer3 encapsulated network across multiple hosts. And in this case, each host should be able to act as a gateway for the traffic local to it. It is this last example I am currently investigating.

Reynold's Blog has an entry called Configuring Cumulus Linux High Availability Layer 2 Network. The most interesting aspect of this post is reference to using 'address-virtual' commands when using ifupdown2 style /etc/network/interface structures:

address-virtual 00:00:5e:00:01:02 10.11.2.254/24

The ip and mac addresses are identical across interfaces sharing the gateway role. The mac address is a reserved range 00:00:5e:00:01:00 – 00:00:5e:00:01:ff for VRRP style operations. The ip address is the virtual ip address (VIP). This style of usage is explained more in Virtual Router Redundancy - VRR.

Or maybe I don't worry about this as Ethernet Virtual Private Network - EVPN has a section with asymmetric routing and symmetric routing which do not need vrrp style constructs.

Layer 3 routing on Cumulus Linux MLAG talks about VRR, the address-virtual, and FRR/route-map to obtain ECMP based load balancing. Now the question - how to get things to not need MLAG.

# netstat -rn -f inet
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.7.28.23      0.0.0.0         UG        0 0          0 vlan28
10.7.1.1        10.7.3.7        255.255.255.255 UGH       0 0          0 enp14s0f2
10.7.1.2        10.7.3.4        255.255.255.255 UGH       0 0          0 enp14s0f3
10.7.1.33       10.7.28.23      255.255.255.255 UGH       0 0          0 vlan28
10.7.1.34       10.7.28.23      255.255.255.255 UGH       0 0          0 vlan28
10.7.1.41       10.7.3.7        255.255.255.255 UGH       0 0          0 enp14s0f2
.....

# ls -l /sys/class/net
total 0
lrwxrwxrwx 1 root root    0 Mar  8 04:06 bond0 -> ../../devices/virtual/net/bond0
-rw-r--r-- 1 root root 4096 Mar  8 04:05 bonding_masters
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f0 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.0/net/enp12s0f0
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f1 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.1/net/enp12s0f1
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f2 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.2/net/enp12s0f2
lrwxrwxrwx 1 root root    0 Mar  8 04:06 enp12s0f3 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.3/net/enp12s0f3
lrwxrwxrwx 1 root root    0 Mar  8 04:05 enp14s0f0 -> ../../devices/pci0000:00/0000:00:03.2/0000:0e:00.0/net/enp14s0f0
lrwxrwxrwx 1 root root    0 Mar  8 04:05 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx 1 root root    0 Mar  8 04:05 ovsbr0 -> ../../devices/virtual/net/ovsbr0
lrwxrwxrwx 1 root root    0 Mar  8 04:06 ovs-system -> ../../devices/virtual/net/ovs-system
lrwxrwxrwx 1 root root    0 Mar  8 04:05 vlan19 -> ../../devices/virtual/net/vlan19

# cat /sys/class/net/enp8s0f0/statistics/tx_bytes
308623980

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.9.0.0        10.9.2.32       255.255.255.0   UG        0 0          0 enp1s0
10.9.1.1        10.9.2.32       255.255.255.255 UGH       0 0          0 enp1s0

# nstat -a
#kernel
IpInReceives                    156764             0.0
IpInDelivers                    156764             0.0
IpOutRequests                   151083             0.0
IpOutNoRoutes                   40                 0.0
.....

• netstat --statistics
• netstat -s -- protocol stats
• netstat -r -- routing info from kernel
• netstat -g -- mulicast info
• netstat -i -- stats
• netstat -ic -- monitor continuously
• netstat -e -- extra information
• netstat -o -- timer related information
• netstat -p -- associated PID
• netstat -l -- listening sockets
• ss --summary

• nload - console based histogram
• iftop - console based numeric traffic values
• iptraf - An IP traffic monitor that shows information on the IP traffic passing over your network.
• iptraf-ng - is a fork of the original project
• dstat - Combines vmstat, iostat, ifstat, netstat information and more
• sar

Diagnostics
• strace -e open nstat 2>&1 > /dev/null|grep /proc
• strace -e open netstat -s 2>&1 > /dev/null|grep /proc

I was looking through Cumulus ifupdown2 code and came across references to vrrpd and ifplugd. I have been using keepalived before becoming aware of what ifupdown2 can do. I am starting to fathom how things relate 'under the hood'.

Configuring Cumulus Linux High Availability Layer 2 Network – Part 2 introduces vrrp along with 'address-virtual' in the /etc/network/interfaces file:

auto brvlan.10
iface brvlan.10
 address 10.0.10.1/24
 address-virtual 00:00:5e:00:01:01 10.0.10.254/24

The MAC address range is reserved in an RFC, with the last byte adjustable, which now makes sense as to why there could only be 255 vrrp instances. Based upon current thinking, something I need to test, is ifupdown2 sets up ifplugd and vrrpd under the hood to handle migrating the virtual address from interface to interface.

I need to figure out how the configuration above correlates with vrrp specific settings as seen in ifupdown2 vrrpd.py source code.

When evaluating VRRP settings, referring to Configuring, Attacking and Securing VRRP on Linux may be helpful. One useful suggestion for some implementations is to put the virtual address on an untagged interface away from the 'protected' vlans in order to prevent spoofing of the VRRP packets.

Debian packages a very old version of vrrpd. It is best to use a more up-to-date version at fredbcode/Vrrpd

Right up my alley -- someone who is using SaltStack to provision networks. There are some ideas in which I might lift for some of my own work: Building your own sdn with debian linux salt stack and python.

• Netdev 2.1 conference report - an excellent and detailed summary of Netdev 2.1 Conference in Montreal from April 6 to 8 written by Brian Linkletter.
• Interactive map of Linux kernel
• Per-IP rate limiting with iptables -- need to figure out how to do this in nftables
• Using a dummy network interface - “fake” network interface through a module called “dummy”

Because they help decipher the operation of ifupdown2, frr, and other routing tools.

• Interface Configuration and Management
• Ethernet Virtual Private Network - EVPN
• Management VRF
• Virtual Router Redundancy - VRR
• Data Center Host to ToR Architecture
• Power over Ethernet - PoE
• Network Virtualization
• Redistribute Neighbor - uses ifplugd
• Equal Cost Multipath Load Sharing - Hardware ECMP
• Segment Routing
• VLAN-aware Bridge Mode for Large-scale Layer 2 Environments
• Data Center Layer 2 High Availability - validated design
• VXLAN Routing - probably associated with FRR symmetric,asymmetric routing

Third party blogs:

• BGP in EVPN-Based Data Center Fabrics - implies that EVPN with eBGP may not 'work out of the box', but seems to be functioning just fine with my implementation of Linux kernel 4.8/4.14, Free Range Routing 4.0.
• Layer 3 Leaf/Spine Fabric with EVPN / VXLAN control - with a note about using the "Common Spine ASN – Discrete Leaf ASN", with eBGP, which results in inherent ECMP (Equal Cost Multi Path) routing. (made possible with FRRouting as the underlying routing engine).
• VRF for Linux where David Ahearn discusses the rationale for adding VRF to the Linux kernel. There is also an entry about listening on ports in a VRF: "gives users and architects a choice: bind a socket per VRF or use a ‘VRF any’ socket. "
• Ifupdown2: Network Interface Manager slideshare from DebConf16, with a discussion of ifquery, policy manager, vrf, ...
• Using the Linux VRF Solution - comprehensive slide deck by David Ahearn on using VRF in Linux: systemd, iproute2, ifupdown2, ...
• Microservice Networking - Leveraging VRF on the host - another David Ahearn slide deck - discusses the use of /32 addresses in a container

Some links I have come across, but nothing will be easy to do

• open-ethernet/mlag: brings in a number of other libraries, and needs a controller to sync states
• MLAG on Linux - lessons learned, from the folks at Cumulus Networks. I downloaded their Cumulus VX to see about extracting the MLAG tools, but their level of integration is... extensive, and therefore hard to extract into something self contained. A paper to go along with the presentation. And a slide show which goes into excellent detail about how packets transition an MLAG solution, and how to stop duplication.

I found out today, via I, Admin, that the Linux kernel, regardless of whether iptables or nftables is installed or not, is tracking connections. That tends to indicate then, that adding rules via nftables will add very little additional overhead. So firewalling a box adds very little additional overhead.

The state of the connection table can be determined by installing:

sudo apt install conntrack

The -L option will show the current state of the table. The -E allows one to watch connection states as they change.

To watch and to sort the state of connections, use:

sudo apt install iptstate

Traffic bandwidth used via particular tcp sessions can be viewed with

sudo apt install tcptrack

In a nutshell, "state" is determined via the data collected by and rules determined by connection tracking. Connection tracking is mostly just a set of timeouts, thresholds, and verifications that help us determine if a packet is "likely" or, ideally "mostly guaranteed" to be part of a known/expected/established layer 4 session. These timeouts, thresholds, and verifications can be seen by catting the various files in /proc/sys/net/netfilter

But back to connection tracking, ... having made the explicit realization that connection tracking is 'always on', and that iptables and nftables only add rule into that path, it becomes easier to realize that network security becomes a very low overhead endeavor. There are very few reason's for not having an 'always on' security posture.

By running

sudo apt install conntrackd

and creating an appropriate conntrackd.conf file, it is possible to build a synchronized, multi-host, active/active firewall solution with little additional effort. conntrackd is designed to replicate the local connection tracking table to other hosts, and to merge other host's connection tracking tables into the local tables.

2018/01/15: Vincent Bernat, in comments below, indicates that "connection tracking is not automatically done. It is done only if the nf_conntrack and/or nf_conntrack_ipv6 modules are loaded (or builtin). This is usually only done by autoloading through the first iptables command". I will need to check that out to confirm from an installation perspective as well as a packet forwarding speed perspective.

The Energy Sciences Network has a number articles (plus many other things):

• Linux Tuning which includes TCP tuning entries, as well as congestion control entries.
• Even an entry on 40G/100G Tuning

2017/11/20 TCP Related: Coping with the TCP TIME-WAIT state on busy Linux servers by Vincent Bernat.

Etherate code on Github is supplemented with some documentation and comments at 53bits Etherate. Etherate sends traffic directly over Ethernet so it goes without saying that all tests between Tx and Rx hosts are run within the same layer 2 broadcast domain, it can not test over a layer 3 boundary.

TRex Low-Cost, High-Speed Stateful Traffic Generator

MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.

iperf3

NFPA: Network Function Performance Analyzer: is a publicly available, open-source measurement application, which is not only in accordance with standardized methodologies (RFC 2544), but also makes possible to comprehensively compare performance metrics of NFs in an exhaustive range of dimensions. Uses DPDK PktGen.

• TCP Statistic and Analysis Tool
• tcpick: is a textmode sniffer libpcap-based that can track, reassemble and reorder tcp streams
• tcpflow a tool similar to tcpick

To detect if a local interface is in promiscuous mode, run:

# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0    1500    41831      0      0 0         32796      0      0      0 BMRU
enp3s0    1500        0      0      0 0             0      0      0      0 BMU
enp4s0    1500  2847577      0 308126 0       2433906      0      0      0 BMRU
lo       65536        1      0      0 0             1      0      0      0 LRU
veth1n    1500  1354033      0 2752094 0       1469155      0  72704      0 BMRU

The above command, no interfaces are in promiscuous mode. By setting the mode, and running the check again, the interface is in promiscuous mode with the 'P' flag:

# ip link set dev enp4s0 promisc on
# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0    1500    41871      0      0 0         32833      0      0      0 BMRU
enp3s0    1500        0      0      0 0             0      0      0      0 BMU
enp4s0    1500  2847700      0 308126 0       2434033      0      0      0 BMPRU
lo       65536        1      0      0 0             1      0      0      0 LRU
veth1n    1500  1354033      0 2752094 0       1469161      0  72704      0 BMRU

List of flags:

• B: broadcast
• M: multicast
• P: promisc mode
• R: running
• U: up

While looking at the MACsec information, I came across information about LLDP (Link Layer Discovery Protocol) and how it can be used for packet pacing. This is relevant for me as I found that when doing IPSEC encryption from one device to another, it is easy to overrun the receiver's packet buffers if the packets cannot be filtered and decrypted fast enough.

Mellanox has an article called Quality of Service, which discusses Data Center Bridging (DCB), Priority-based Flow Control (PFC), Pause frames, and shared buffers. They use the open source open-lldp. Their hardware seems to be capable of sending pause frames when buffers get too full, something which would help when slower SoC cpus are processing encrypted packets.

The open-lldp wiki has an article on Data Center Bridging (DCB) on Linux. It has very generic notes. But since there is a lldptool-pfc man page, it appears as though functionality is indeed available in Linux.

When looking at some NetDevConf papers, I opened up MACsec, which is "Encryption for the wired LAN". The biggest advantage illustrated was in a cloud environment where traffic from one guest to another via a third party's infrastructure could be easily and application-transparently encrypted. The slide deck has a number of actual usage examples. Sabrina Dubroca writes an article on the redhat developers blog: MACsec: a different way to encrypt network traffic.

Which, come to think of it, would be a dead simple way to encrypt L2 WAN traffic.

• MACsec Implementation on Linux: an in depth tutorial written by costier ntework engineering.
• MACsec on Linux: another well documented example of using MACsec on Linux.