Networks

Found on a email list:

The Netgear GS108T is my typical go-to "not a dumb switch". 8 ports for about $80.

Make sure you get the v3 if you want most of the modern IPv6 L2 features (you also get some very limited L3 capabilities).

Extra bonus with the GS108Tv3, and anything else based on the RTL8380, is that you can run OpenWrt on it.

   [ Noah Meyerhans ]
   * net: Use fq_codel as the default network qdisc (Closes: #890343)

From ticket #890343:

by default Debian uses the pfifo_fast network queuing algorithm:

# tc -s qdisc show
[...]
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 413728102 bytes 475015 pkt (dropped 0, overlimits 0 requeues 785)
 backlog 0b 0p requeues 785

The systemd source package contains this file:

  ./systemd-237/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel

whose purpose is to set the default queuing algorithm to fq_codel.  
According to the NEWS file, this is a better alternative:

        * The default sysctl.d/ snippets will now set:

                net.core.default_qdisc = fq_codel

          This selects Fair Queuing Controlled Delay as the default
          queuing discipline for network interfaces. fq_codel helps
          fight the network bufferbloat problem. It is believed to be
          a good default with no tuning required for most workloads.
          Downstream distributions may override this choice. On 10Gbit
          servers that do not do forwarding, "fq" may perform better.
          Systems without a good clocksource should use "pfifo_fast".

However the 50-default.conf file is not in the Debian binary package.
Is this intentional or an omission?
Could it be possible to enable fq_codel by default?

Flexible routing has now come down in price with the Raspberry Pi Compute Module 4 IoT Router Carrier Board Mini which takes a Raspberry Pi Compute Module 4 and places it on a Compute Module 4 IoT Router Carrier Board Mini.

Here is someone building a Raspberry Pi Firewall and Router with DF Robot Dual NIC with Ubuntu.

How to get stats at all of the different points in the stack to track down reasons for dropped packets:

• ethtool -S for h/w and driver
• tc -s for drops by the qdisc
• /proc/net/softnet_stat for drops at the backlog layer
• netstat -s for network and transport layer
• ip -s

The key is the 'ether' specifier. The -e option shows the mac address in the output:

$ sudo tcpdump -n -i vlan50 ether host e4:54:e8:29:44:2d or ether host ff:ff:ff:ff:ff:ff -vv -e

The country code is not read from ACPI but from the EEPROM of the WiFi-card (0x0 "World Regulatory Domain" by default).

How to Build your Own Wireless Router (Part 3) - kernel requires code patch to enable 5GHz operations.

ath10k locks to regulatory domain US on ACPI platforms

From Phoronix - Linux 5.6 To Bring FQ-PIE Packet Scheduler To Help Fight Bufferbloat, there are some example configurations for using different buffering mechanisms (Implementing the Flow Queue PIE AQM in the Linux Kernel).

For wireless clients:

net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control = bbr

and fq_codel (codel) is better suited for router.

Another possibility:

I had pretty amazing results with cake. I prefer it over fq_codel and it is now part of main line since 4.19.

Cake - Common Applications Kept Enhanced

In the past the internet would just grind to a halt when the Apple devices started cloud syncing. They would totally swamp the outgoing bandwidth. Now nobody notices when an Apple device is syncing.

Open firmware for small routers for congestion management:

The OpenWrt issue with 4MB has to do with using newer kernels. For those devices, DD-WRT is a better bet (still on kernel 3.10).

Courtesy of 3 quick ways to reduce your attack surface on Linux, a command to identify open ports and identify the associated application:

$ ss -tulnp --no-header | awk '{print($1, $5, $7)}'
udp 0.0.0.0:32770 users:(("vlc",pid=2786557,fd=39))
udp 0.0.0.0:32771 users:(("vlc",pid=2786557,fd=40))
udp 127.0.0.1:123
udp 0.0.0.0:123
udp 0.0.0.0:631
udp 0.0.0.0:37019
udp 0.0.0.0:5353
udp 0.0.0.0:39276 users:(("vlc",pid=2786557,fd=50))
udp 0.0.0.0:39277 users:(("vlc",pid=2786557,fd=51))
tcp 0.0.0.0:61209
tcp 127.0.0.1:25
tcp 127.0.0.1:5433
tcp 0.0.0.0:8794
tcp 127.0.0.1:3493
tcp 127.0.0.1:5101 users:(("ssh",pid=899751,fd=4))
tcp 127.0.0.1:5102 users:(("ssh",pid=2187130,fd=4))
tcp 127.0.0.1:5201 users:(("ssh",pid=2186389,fd=5))
tcp 127.0.0.1:5202 users:(("ssh",pid=2186389,fd=7))
tcp 0.0.0.0:22
tcp 127.0.0.1:631
tcp 127.0.0.1:5432
...

In building an openflow controller, the controller needs to perform some packet interception, inspection, and modification activities. This is easy from a UDP perspective. But when the openflow engine forwards the complete packet, mac addresses and all, the packets need to be decoded. And for TCP, there are various connection oriented states which need to be taken in to account when trying to perform deep packet inspection.

This means creating a purpose built TCP stack to perform the state management, or someone make use of existing tools to perform the state management. In my searches, I came across:

• chobits/tapip - a user-mode TCP/IP stack based on linux tap device - with a basic state machine - but I don't need the tun/tap integration as I already have the buffered inbound and outbound raw packets.
• tass-belgium/picotcp - PicoTCP is a free TCP/IP stack implementation - this is better as it has an api for providing timers and packet-in/packet-out flows. Documentation looks good.
• hacker's userspace TCP/IP stack with a five part TCP stack tutorial at www.saminiir.com. It has a simple and self contained state machine.
• lwIP mirror - an embeddable tcp stack with a number of bundled applications.

The book "TCP/IP Illustrated, Volume 1, 2e" discusses timers extensively, something useful, as I am considering putting in the basic state handling code manually, rather than using one of the above libraries.

I saw a debian bug report Installation was successfully at BananaPi. When initially looking at the Banana Pi, as it has a version with a number of network interfaces, I didn't readily see the tooling necessary for booting images.

The bug report shows that Debian now has native images available, and similar images can be found in the Buster and Sid distributions. There are also Beagle Bone Black images in there.

Random links I picked up for building custom installs:

• Building a pure Debian armhf rootfs - good writeup by Tim Fletcher
• DebianInstallerArmOtherPlatforms - Howto install Debian/armel with a custom kernel - links on the page lead to other excellent articles
• debian debootstrap - from one of the primary contributors to Banana Pi builds
• Banana Pi BPI-R2 Wiki - much updated since I last landed on it
• Searching testing people for hdmi + wifi in Kernel 4.16 - forum entry
• How to build an Ubuntu/Debian SD image from scratch - with some bonding notes
• Banana Pi BPI-R64 - wonder when the release date is

Kind of related, Redhat has a summary article of Introduction to Linux interfaces for virtual networking. Excellent summary of many interface types in Linux.

2019/02/18 From a different direction, via the Debian Bugs list, I've come across Lamobo R1 which is another name for the Banana Pi R1. The page has some additional information, and by cruising to the main page, additional AllWinner SoC info can be obtained.

In addition, Armbian Build on GitHub offers the code for building Arm kernels on these types of boards.

Kernel CI shows results of Kernel Builds on various devices, with an example entry of Kernel 5.0 booting on Banana Pi R2

Sometimes, you just don't know what is on your network. A couple ways of finding out include using nmap or arp-scan.

An easy install:

sudo apt install arp-scan nmap

And an easy use with some basic defaults (the Unknown ones are LXC containers with manual mac addresses, and Cadmus Computer Systems is a VirtualBox device):

$sudo arp-scan --interface=vlan90 --localnet
Interface: vlan90, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9.5 with 256 hosts (https://github.com/royhills/arp-scan)
10.55.90.1      00:13:3b:0f:59:24       Speed Dragon Multimedia Limited
10.55.90.14     08:00:27:de:fc:9d       Cadmus Computer Systems
10.55.90.21     0a:00:55:90:00:ab       (Unknown)
10.55.90.22     0a:00:55:90:00:0c       (Unknown)
10.55.90.23     0a:00:55:90:00:23       (Unknown)
10.55.90.31     f0:ad:4e:03:64:7f       Globalscale Technologies, Inc.

8 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9.5: 256 hosts scanned in 3.155 seconds (81.14 hosts/sec). 6 responded

With nmap, it knows about the mis-named mac:

$ sudo nmap -sn 10.55.90.0/24
Starting Nmap 7.70 ( https://nmap.org ) at 2018-09-29 20:25 MDT
Nmap scan report for 10.55.90.1
Host is up (0.00032s latency).
MAC Address: 00:13:3B:0F:59:24 (Speed Dragon Multimedia Limited)
Nmap scan report for 10.55.90.14
Host is up (-0.10s latency).
MAC Address: 08:00:27:DE:FC:9D (Oracle VirtualBox virtual NIC)
Nmap scan report for 10.55.90.21
Host is up (-0.100s latency).
MAC Address: 0A:00:55:90:00:AB (Unknown)
Nmap scan report for 10.55.90.22
Host is up (-0.100s latency).
MAC Address: 0A:00:55:90:00:0C (Unknown)
Nmap scan report for 10.55.90.23
Host is up (-0.087s latency).
MAC Address: 0A:00:55:90:00:23 (Unknown)
Nmap scan report for 10.55.90.31
Host is up (0.00074s latency).
MAC Address: F0:AD:4E:03:64:7F (Globalscale Technologies)
Nmap scan report for 10.55.90.10
Host is up.
Nmap done: 256 IP addresses (7 hosts up) scanned in 3.31 seconds

With these specific commands, arp-scan may see things which nmap may not, ie, firewall that are blocking pings. nmap probably has a mode for scanning similarly to arp-scan.

From the iovisor-dev mailing list:

>I am trying to collect IPFIX flow data from the linux host interface.

Why IPFIX and not sFlow or netflow ?

>Can someone guide me the best way to collect the data using XDP.

It depends a bit on you setup. Assuming you want to do this "inline" on the box receiving the traffic. Then you should know/learn, that XDP cannot allocate a new packet (that e.g. could be used sending IPFIX/sFlow info directly). Instead, I would use the perf-ringbuffer to store sampled-packets (via copy), and then code a userspace program that reads from this perf-ringbuffer, and it will communicate with the central IPFIX/sFlow server.

>Any samples for reference will be a great help.

From XDP howto use the perf-ringbuffer via bpf_perf_event_output, samples are avail here:

• https://github.com/torvalds/linux/blob/master/samples/bpf/xdp_sample_pkts_kern.c
• https://github.com/torvalds/linux/blob/master/samples/bpf/xdp_sample_pkts_user.c

Notice, there are also plenty of BCC examples using the perf-ringbuffer, look for BCC code with:

   BPF_PERF_OUTPUT(events);
   events.perf_submit(ctx, data, sizeof(struct data_t));

On 06/14/2018 09:22 PM, someone wrote:
> So I have to ask, why is it advantageous to put this in a container 
> rather than just run it directly
> on the container's host? 

Most any host now-a-days has quite a bit of horse power to run services. All those services could be run natively all in one namespace on the same host, or ...

I tend to gravitate towards running services individually in LXC containers. This creates a bit more overhead than running chroot style environments, but less than running full fledged kvm style virtualization for each service.

I typically automate the provisioning and the spool up of the container and its service. This makes it easy to rebuild/update/upgrade/load-balance services individually and enmasse across hosts.

By running BGP within each container, BGP can be used to advertise the loopback address of the service. I go one step further: for certain services I will anycast some addresses into bgp. This provides an easy way to load balance and provide resiliency of like service instances across hosts,

Therefore, by running BGP within the container, and on the host, routes can be distributed across a network with all the policies available within the bgp protocol. I use Free Range Routing, which is a fork of Quagga, to do this. I use the eBGP variant (RFC 7938) for the hosts and containers , which allows for the elimination of the extra overhead of OSPF or similar internal gateway protocol.

Stepping away a bit, this means that BGP is used in a tiered scenario. There is the regular eBGP with the public ASN for handling DFZ-style public traffic. For internal traffic, private eBGP ASNs are used for routing traffic between and within hosts and containers.

With recent improvements to Free Range Routing and the Linux Kernel, various combinations of MPLS, VxLAN, EVPN, and VRF configurations can be used to further segment and compartmentalize traffic within a host, and between containers. It is now very easy to run vlan-less between hosts through various easy to configure encapsulation mechanisms. To be explicit, this relies on and contributes to a resilient layer 3 network between hosts, and eliminates the bothersome layer 2 redundancy headaches with spanning tree and such.

That was a very long winded way to say: keep a very basic host configuration running a minimal set of functional services, and re-factor the functionality and split it across multiple containers to provide easy access to and maintenance of individual services like dns, smtp, database, dashboards, public routing, private routing, firewalling, monitoring, management, ...

There is a higher up-front configuration cost, but over the longer term, if configured via automation tools like Salt or similar, maintenance and security is improved.

It does require a different level of sophistication with operational staff.

Other BGP routing Daemons:

• exabgp - python based
• gobgp - go based
• bird - c based
• bagpipe - python based, in openstack project

Other mailing list comments:

Our use case was both on exporting service IPs as well as receiving routes from ToRs. Exa is more geared towards the former than the latter. Rather then working on getting imports and route installation through Exa, we found it simpler with BIRD exporting the service IP from it bound to a loopback to run local healthchecks on the nodes and then have them yank the service IP from the loopback on failing healthchecks in order to stop exporting.

The intent of the original post was vague. Like a lot of people, I would not run a full BGP router in a container. Now, if the purpose is to inject or learn a handful of routes in order to do limited host routing, I can see the need. A route-server or a looking glass in a container would be fine, or something to perform analysis on the routing table, but not anything that has to route actual traffic.

I use ExaBGP to inject routes, perfect tool for that. If routes have to be received (not my use case) it makes more sense, as stated by previous posts, to use Quagga or BIRD. Which one is better : easy : if you like Cisco better, use Quagga. If you like Juniper better, use BIRD

BIRD looking glass looks very good

When using the BGP module in Free Range Routing, the 'network ' statement can be used in at least a couple of different roles. One generic role is for generating an advertisement for an aggregate prefix, say to an upstream. Even knowing that, one is also tempted to use a network statement to advertise connected prefixes.

The draw back to advertising connected prefixes is that the prefix is advertised even a related interface is not 'up'. This could lead to a blackhole scenario.

A better way to handle the advertisements of connected prefixes is to use the 'redistribute connected' command.

Even with the use of this command, there may be scenarios (which I need to test at some point) where the prefix is advertised or withdrawn depending upon the link state. Free Range Routing has an additional command which could be used to ensure link state checking: 'bgp network import-check'.

There is more about the Linux state checking flags in the Why Link-State Matters presentation from LinuxCon 2015.

In addition, the Free Range Routing developers have brought together some relevant sysctl settings.

This is another collection of random notes, this time, on how to build something on Linux somewhat resembling Cisco's Global Load Balancing capability, basically a continuation of my entry at Linux ifupdown2 VRRP.

Traditionally, one sets up VRRP using keepalived or the simpler vrrpd. This configuration is typically used when setting up (typically) two routers in an active/passive setup to act as a gateway for a network subnet. In essence, the two (or more) routers negotiate who will hold the gateway mac and ip address.

In other circumstances, it might be desired (and possible) to run active/active. This is a possibility when running containers on a host, and there are similar services running across the hosts. In this instance the same address can be assigned as a secondary address across multiple containers to load balance traffic.

And in even other cases, subnets may be stretched in a layer2 over layer3 encapsulated network across multiple hosts. And in this case, each host should be able to act as a gateway for the traffic local to it. It is this last example I am currently investigating.

Reynold's Blog has an entry called Configuring Cumulus Linux High Availability Layer 2 Network. The most interesting aspect of this post is reference to using 'address-virtual' commands when using ifupdown2 style /etc/network/interface structures:

address-virtual 00:00:5e:00:01:02 10.11.2.254/24

The ip and mac addresses are identical across interfaces sharing the gateway role. The mac address is a reserved range 00:00:5e:00:01:00 – 00:00:5e:00:01:ff for VRRP style operations. The ip address is the virtual ip address (VIP). This style of usage is explained more in Virtual Router Redundancy - VRR.

Or maybe I don't worry about this as Ethernet Virtual Private Network - EVPN has a section with asymmetric routing and symmetric routing which do not need vrrp style constructs.

Layer 3 routing on Cumulus Linux MLAG talks about VRR, the address-virtual, and FRR/route-map to obtain ECMP based load balancing. Now the question - how to get things to not need MLAG.