Networks

• Routing In Fat Trees (rift): addresses the demands of routing in Clos and Fat-Tree networks via a mixture of both link-state and distance-vector techniques colloquially described as 'link-state towards the spine and distance vector towards the leafs'. RIFT uses this hybrid approach to focus on networks with regular topologies with a high degree of connectivity, a defined directionality, and large scale.
• Network Virtualization Overlay Solution using EVPN explores the various tunnel encapsulation options over IP and their impact on the EVPN control-plane and procedures
• Interconnect Solution for EVPN Overlay networks

For Android:

wifianalyzer for Android, shows what station IDs are on what channels, handles 2.4g and 5G connections, etc. Doesn't provide mapping, just shows "from where I am right know, what channels have which stations as what strengths?"

Nanobeam:

NanoBeam® AC is a directional antenna/radio integrated unit and is intended as a point to point or point-to-multipoint WISP client radio. The one feature you can get from it very cheaply is a directional, 2x2 MIMO 5.x GHz band spectrum analyzer that sees things *which are not 802.11 or wifi based. sample images

Highly useful for tracking down a specific source of non-wifi 5 GHz band interference. There's all sorts of random consumer grade things people can buy and introduce into an environment which do not broadcast MAC addresses or SSIDs, and do not show up on purely 802.11(abgn/ac) based tools.

It will of course also see hidden SSIDs and standard+non-standard 802.11abgn(ac) emitters.

There are also 2.4 GHz versions of similar products which will let you find non-802.11 emitters in the 2300 to 2500 MHz band. At $79 a lot less expensive than a "real" spectrum analyzer.

You can get DC PoE injectors for them which will connect to a Makita drill battery if you want to make it portable and wander around with a laptop.

Some entries on DDNS (Dynamic Domain Name Service). Standards based DDNS makes use of RFC 2136 as a specification.

• Updating DNS Entries (with nsupdate or alternative implementations) – Run Your Own DynDNS Service, with links to other related sites about DDNS. But dated back to 2013.
• SaltStack DDNS State: SaltStack is Python based, and uses a python library called dnspython to handle ddns updates.
• dnspython s a DNS toolkit for Python. It supports almost all record types. It can be used for queries, zone transfers, and dynamic updates. It supports TSIG authenticated messages and EDNS0.
• dnsupdate.py: a python successor to nsupdate.

As a side note, I need to keep this handy for another project: DNS in OpenVPN: a better approach, which discusses using multiple copies of dnsmasq, spread over several sites, for maintaining zone based queries. The most interesting aspect is that of including reverse .in-addr.arpa entries in the configuration files, something into which I need to look.

• Best Current Operational Practice for Operators: IPv6 prefix assignment for end-users - persistent vs non-persistent, and what size to choose
• draft-palet-v6ops-p2p-from-customer-prefix-01: Using /64 from Customer Prefix for the Inter-Router Link
• Jool: an Open Source SIIT and NAT64 for Linux
• Understanding IPv6 – 7 Part Series - from Networking with FISH - "This has been a very popular series for people completely new to IPv6 and still working to wrap their head around it all. I wrote this blog series I wish had been out there when I was new to IPv6. I hope you enjoy it tons."
• Small / Home Office network IPv6 wish list - n this post, I’d like to focus on efforts to bring greater IPv6 support to Small Office / Home Office (SOHO) routers, which differ from standard home routers as they need to accommodate broader functionality. My hope is that router manufacturers will incorporate these features to improve IPv6 support.

From a listgroup:

• Paris Traceroute
• tcptraceroute

Traceroute on Linux/Unix boxes is generally using UDP by default. Try using ICMP, by adding the -I (capital i) option to traceroute:

traceroute -I 52.74.124.136

A TCP connection is uniquely identified by the combination of four numbers: The source IP address, the source port, the destination IP address and the destination port. You used the word session, but sessions happen above TCP in the stack and may use more than one TCP connection. Every packet in the connection contains all four numbers and no packet from any other connection contains the same four numbers.

If you want to track the connections, you capture the packets at each point in the path (router products have vendor-specific ways of doing this) and see which unique sets of the four numbers went through which router and router interface.

If you want to -test- which path a TCP connection -would- take, Ruairi's afore-mentioned tcptraceroute is the way to go. The regular traceroute with modern Linux servers also supports the "-T" flag which does the same thing. It works just like regular traceroute but uses synthetic TCP SYN packets instead of ICMP or UDP packets, allowing the packets to pass firewalls which would otherwise block the trace.

Bear in mind that in each case you will likely only see the path taken at the IP level. Underlying transits at the Ethernet or MPLS level are intentionally invisible to the endpoints.

In the data center context, enabling sFlow continuously captures packets from all paths and can be used to trace multi-path packet flows, whether layer 2 (MLAG/LAG), or layer 3 (ECMP). sFlow reports physical switch ports and captures Ethernet packet headers, so you can relate paths to MPLS labels, Ethernet headers, IP headers, TCP/UDP headers, VxLAN tunnels, etc.

The following article provides an example: Troubleshooting Connectivity Problems

Since I plan on using this similar combination in some upcoming configurations, I'll keep the reference here, and hopefully add details once I've worked through them.

• Using VLANs with OVS and libvirt by Scott Lowe back in 2012, referenced in an Replace Linux Bridge with Openvswitch on Ubuntu (an OVS mailing list entry).

The question:

What is the best way to model quota through ovs? Right now the best thing we could come up with is to periodically poll a flow stat and install a drop flow when the flow stat goes past the quota.

The answer given (which provides some food for thought on better mechanisms):

Open vSwitch includes sFlow / NetFlow instrumentation. Using flow data to drive your quota controller reduces the complexity of the OpenFlow rules since you no longer need granular rules to provide the traffic measurements.

The following article describes building a quota controller using sFlow:

Real Time Visibility and Control

Bypassing OpenFlow controller and injecting flow table entries via a code call:

> Can someone refer to the code where a  new flow is inserted to
> openflow tables in the OVS, as a consequence action of a received
> message from a controller?

handle_flow_mod() in ofproto.c

Another similar request:

> My project was previously calling system commands like ‘ovs-ofctl add-flow’ directly from C, 
> but we would like to do this programmatically now, by calling into the openvswitch library. 
> After looking at this for a bit and what methods I would have to call, I’ve realized this is 
> non trivial, and I can’t find any easily exposed methods to add and delete flows. 
> Does anyone have an example anywhere of calling into openvswitch methods 
> directly to add/remove flows? So far the best way I can find is calling into 
> ovn/controller/ofctrl.h:ofctrl_add_flow(…), but this will require quite a bit of legwork.

One way:

There are plenty of controllers out there, including a couple in C - libfluid

Another way:

Here’s how I do it.  It’s essentially the same as what the ‘ovs-ofctl add-flow’ command does under the hood.

1. Build a string formatted as the “ovs-ofctl” command would do, 
         for example “table=1,cookie=0xdeadbeef,in_port=1,actions=resubmit(,2)”
2. Pass this to parser_ofp_flow_mod_str().  The parser will magically parse the string in to a rich struct.
3. Pass the struct to ofputil_encode_flow_mod().  This returns a struct with a serialized OpenFlow message buffer.
4. Send this buffer on the wire either via a socket you manage yourself or let OpenVswitch do it for you.

Added 2017/11/13, the second link shows a terrific example usage for ovs-dpctl

> I have a problem that when i use the ovs-dpctl to add a flow into datapath, 
> it occurs "ovs-dpctl: parsing flow key (Invalid argument)"
>
>     example:
>    root@jlt:~# ovs-dpctl add-flow system@myDP "in_port(1),eth_type(0x800),ipv4
> (src=172.31.110.4,dst=172.31.110.5)" 2
> ovs-dpctl: parsing flow key (Invalid argument)

You may be able to use 'dmesg' to see which key is missing.

See the talk by Joe Stringer:
Youtube

And the blogpost (shameless plug for myself):
Direct Kernel OVS Flow Programming

A follow up from Ben indicated though:

I believe that this particular error comes from the userspace parser, not the kernel.

Another addition on 2017/11/13:

> For us, newbies, examples are extremely valuable, more than a thousand
> words.
> If some kind soul has an example on how to insert a new interface (s1-eth3)
> in a single switch (s1) with two interfaces (s1-eth2 and s1-eth3) please
> share with me.

If you run something like this:
        ovs-vsctl -vjsonrpc -- add-bond s1 s1-eth3 s1-eth2 s1-eth3
then you will see what ovs-vsctl does to insert such a bond, logged as
the jsonrpc module.

2017/11/25 Addition 1:

ovs-save is a shell script, using `ovs-ofctl dump-flows br` to write flows
into a file, and then use `ovs-ofctl add-flows br FILE` to add them.

2017/11/25 Addition 2:

The OpenFlow Spec does not directly provide the mechanism for the multi-path load balancing and also Ryu does not provide it. I guess it is depending on your application design...

But for the beginning, how about using the Group action(the "select" type)? The selection algorithm is depending on your switch though, it provides an easy way to do load balancing.

Please refer to the section "5.6.1 Group Types" in the OpenFlow Spec 1.3.5 for the details: https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-switch-v1.3.5.pdf

2017/12/07 dpctl: Support flush conntrack by 5-tuple

With this patch, "flush-conntrack" in ovs-dpctl and ovs-appctl accept a conntrack 5-tuple to delete the conntrack entry specified by the 5-tuple. For example, user can use the following command to flush a conntrack entry in zone 5.

$ ovs-dpctl flush-conntrack zone=5 \
    'ct_nw_src=10.1.1.2,ct_nw_dst=10.1.1.1,ct_nw_proto=17,ct_tp_src=2,ct_tp_dst=1'
$ ovs-appctl dpctl/flush-conntrack zone=5 \
    'ct_nw_src=10.1.1.2,ct_nw_dst=10.1.1.1,ct_nw_proto=17,ct_tp_src=2,ct_tp_dst=1'

2018/06/11: "I suggest using ofproto/trace to figure out what's going on. See ovs-vswitchd(8) if you're not already familiar with it."

2018/06/11: "You can see the history of the configuration database by running "ovsdb-tool -mm show-log" on it. This might reveal what is happening, too."

"ovs-vsctl --may-exist add-port {{ info.ovs_bridge }} {{ interface }} {{ info.ovs_options }} -- set interface {{ interface }} type=internal"

2018/11/25 - ovs-testcontroller

A Check_MK plugin which is supplied as part of the standard install but not 'turned on' is the 'smart' agent. It is a plugin which resides on the target with hard-drives to be monitored. Temperatures and other useful statistics can be collected.

The plugin can be found in the share/check_mk/agents/plugins/ directory amongst a number of other useful plugins.

It needs to be copied to the /usr/lib/check_mk_agent/plugins/ directory on the target to be monitored.

In order to be useful, the 'smartmontools' package needs to be installed as a pre-requisite.

On one of my systems, I get output some output similar to:

SMART WDC_WD1003FZEX-00K3CA0_WD-WCC6Y3PN9XKC Stats
  OK - Powered on: 3529 hours, Power cycles: 51, Reallocated sectors: 0, 
  Reallocated events: 0 (was 0 during discovery: normalized value looks OK), 
  Spin retries: 0, Pending sectors: 0, UDMA CRC errors: 0
Temperature SMART WDC_WD1003FZEX-00K3CA0_WD-WCC6Y3PN9XKC
  OK - 30 °C

Values are charted as well.

To get at various hardware sensors like temperatures and voltages, it is typically necessary to install the 'lm-sensors' package. Once the package is installed, sensors can be auto-detected with:

sensors-detect --auto

A variation of that is:

yes | sensors-detect

At the end of the output, this will probably generate an output similar to:

To load everything that is needed, add this to /etc/modules:
#----cut here----
# Chip drivers
coretemp
#----cut here----

'modprobe coretemp' can be used to obtain instant gratification by loading the module interactively, instead of at the next reboot.

On a simple system, some values can be seen:

# sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)
temp2:        +29.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +50.0°C  (high = +105.0°C, crit = +105.0°C)
Core 0:        +49.0°C  (high = +105.0°C, crit = +105.0°C)
Core 1:        +48.0°C  (high = +105.0°C, crit = +105.0°C)

For obtaining an output suitable for scraping by a check_mk agent, a different format can be emitted:

# sensors -u
acpitz-virtual-0
Adapter: Virtual device
temp1:
  temp1_input: 27.800
  temp1_crit: 105.000
temp2:
  temp2_input: 29.800
  temp2_crit: 105.000

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:
  temp1_input: 49.000
  temp1_max: 105.000
  temp1_crit: 105.000
  temp1_crit_alarm: 0.000
Core 0:
  temp2_input: 48.000
  temp2_max: 105.000
  temp2_crit: 105.000
  temp2_crit_alarm: 0.000
Core 1:
  temp3_input: 48.000
  temp3_max: 105.000
  temp3_crit: 105.000
  temp3_crit_alarm: 0.000

While writing this article, a few interesting web sites were encountered:

• Arch Linux' Sensors Page
• Plugins for Check_MK
• Check_MK plugin: lmsensors: a place to which I need to return and try installing his plugin for more information display in Check_MK.
• Earlier Article I wrote doing something similar but different.

I use an app called 'psensor' for watching system temperatures in a chart form.

OMD is a collection of monitoring tools which include the Check_MK monitoring application. Check_MK is a series of Python scripts and agents designed to simplify the use of a Nagios based scanning engine.

I have a small server running Debian Buster. I have used LXC to build a container. I used the following command to build a container:

lxc-create -n omd  -f /var/lib/lxc/persist/omd/config -t debian -B btrfs -- -r stretch

I have a BTRFS filesystem mounted on /var/lib/lxc. the -B btrfs creates a BTRFS subvolume for the container, which allows snapshots and the like.

The config file looks something like:

# cat /var/lib/lxc/persist/omd/config 
# 20171015
# lxc-create -n omd  -f /var/lib/lxc/persist/omd/config -t debian -B btrfs -- -r stretch
lxc.utsname = omd
lxc.start.auto = 1
lxc.start.order = 5
lxc.start.delay = 1

lxc.network.type = veth
lxc.network.name = eth50
lxc.network.veth.pair = veth-omd-v50
lxc.network.hwaddr = 0a:00:40:50:00:xx
lxc.network.ipv4 = 10.0.10.201/24
lxc.network.ipv4.gateway = 10.0.10.254
lxc.network.script.up = /etc/lxc/ovsup.sh
lxc.network.script.down = /etc/lxc/ovsdn.sh
lxc.network.flags = up

I am using Open vSwitch to provide bridging capabilities to the container and its environs. This particular container, using the lxc.start.auto value, will auto-start when the host restarts.

Yesterday, I attempted to install the Buster version of OMD, but some package dependencies were stale (repository) , and since I didn't have time to troubleshoot that, I went with the stable Stretch version:

apt install nano bash-completion gnupg2 wget
wget --no-check-certificate https://labs.consol.de/repo/stable/RPM-GPG-KEY
apt-key add RPM-GPG-KEY
echo "deb http://labs.consol.de/repo/stable/debian stretch main" > /etc/apt/sources.list.d/labs-consol-stable.list
apt update
apt install omd

It is as easy as that.

To configure a monitoring instance:

omd setup
omd create site01
omd config site01
omd start site01
su - site01
nano etc/check_mk/main.mk

During the 'omd create site01' step, I select the Nagios 3 engine, and set the Check_MK interface as default.

The next experiment is to install the Raw Version and see how that differs.

Monitoring buffer queues has had a recent update on Cumulus running on Mellanox swtiches using watermark values to determine what happens during microbursts.

Also on the same site is a page devoted to documenting Buffer Queue Sizes in various switch lines.

I am more into nftables now-a-days (which just had release 0.8 in the last couple days), but maybe these iptables/xtaqbles thingies can be made to work in nftables:

• Xtables-Addons On Centos 6 & Iptables GeoIP Filtering
• search for 'tarpit nftables' and see if there is anything, it is an interesting sticky concept
• A series of Xtables-addons. It would be interesting which of them could be re-implemented with nftables functionality.

port number with port name:

# ovs-ofctl dump-ports-desc <ovs bridge>

dump flows:

# ovs-ofctl dump-flows <ovs bridge>

Dump OVSDB table:

# ovsdb-client dump 

Obtain schema of OVSDB:

# ovsdb-client get-schema --pretty

Some background info on OVSDB from Brent Salisbury's Blog: Getting Started with OVSDB As Introduction to Open vSwitch says:

Like OSPF, OpenFlow directly influences the forwarding behavior of a networking devices (albeit way differently than OSPF) but isn’t able to actually change the configuration of that device. OSPF cannot disable a router’s interface, or create a GRE tunnel, for instance. For that, we need a management plane protocol.

In the world of OVS, this role is filled by OVSDB. This allows us to use well-understood wire protocols (namely JSON-RPC) to send commands to an OVS instance to do things like create tunnels, turn on/off certain features, get configuration data, and more.

I have wondered about the 'tcp' vs 'ptcp' syntax. Well the answer:

“Manager” refers to an OVSDB client. The “ptcp::" syntax means that OVS is passively listening on that port and local address for incoming JSON-RPC data.

“Controller” refers to an OpenFlow controller. The “tcp:" syntax means that OVS proactively reaches out to a controller at that address to establish the relationship. (Standard port of 6633)

2018/01/20:

ovs-vsctl get Interface <iface> status

2018/03/29

ovsdb-client -f list dump 

2018/06/11

# ovs-appctl dpctl/show
system@ovs-system:
        lookups: hit:393756304 missed:4746333 lost:0
        flows: 31
        masks: hit:1365293746 total:4 hit/pkt:3.43
        port 0: ovs-system (internal)
        port 1: ovsbr0 (internal)
        port 2: enp5s0
        port 3: vlan20 (internal)
        port 4: enp1s0
        port 5: enp2s0
        port 6: enp3s0
        port 7: vlan30 (internal)
        port 8: wlp4s0
        port 9: vlan40 (internal)
        port 10: vlan50 (internal)

# ovs-ofctl show ovsbr0
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000000ec4d20451
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(enp5s0): addr:00:0e:c4:d2:04:54
     config:     0
     state:      0
     current:    100MB-FD COPPER AUTO_NEG
     advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
     supported:  10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
     speed: 100 Mbps now, 1000 Mbps max
 2(vlan20): addr:a2:58:96:ce:5c:40
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(enp1s0): addr:00:0e:c4:d2:04:51
     config:     0
     state:      0
     current:    100MB-FD COPPER AUTO_NEG
     advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
     supported:  10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
     speed: 100 Mbps now, 1000 Mbps max
....
....

2018/06/21

# ovs-appctl dpif/dump-flows ovsbr0
recirc_id(0),in_port(9),eth(src=e6:64:dc:6f:7d:87,dst=44:80:eb:1c:1b:69),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:54, used:8.756s, flags:R., actions:8
recirc_id(0),in_port(13),eth(src=0a:00:40:50:00:ce,dst=b6:4d:8d:6b:37:5a),eth_type(0x0806), packets:0, bytes:0, used:never, actions:10
recirc_id(0),in_port(2),eth(src=84:2b:2b:47:2e:4f,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=199.68.195.98,tip=199.68.195.107,op=1/0xff), packets:2, bytes:120, used:0.836s, actions:push_vlan(vid=20,pcp=0),1,pop_vlan,3
recirc_id(0),in_port(7),eth(src=7e:00:6a:f3:12:91,dst=80:fa:5b:43:c6:db),eth_type(0x0800),ipv4(frag=no), packets:2341, bytes:1954768, used:0.008s, flags:SFPR., actions:5
recirc_id(0),in_port(5),eth(src=80:fa:5b:43:c6:db,dst=7e:00:6a:f3:12:91),eth_type(0x0806), packets:0, bytes:0, used:never, actions:7
....

Redhat has an excellent entry on Open vSwitch: Overview of 802.1ad (QinQ) Support.

A key point raised is:

Enabling QinQ support in OVS is a simple configuration change. You must set the vlan-limit option to a value of 0 (unlimited) or 2. By default vlan-limit has a value of 1, which means only one VLAN tag will be inspected. This preserves backward compatibility with older OVS releases.

$ ovs-vsctl set Open_vSwitch . other_config:vlan-limit=2

The article continues with ovs-ofctl and ovs-vsctl examples.