Networks

Windows 7,8,9,10 are different animals. I was able to get Android, Mac, Linux and various cameras to associate using WPA2 encryption. Windows needs a particular wpa settings, otherwise things won't work. Troubleshooting Windows is next to impossible with its typical cryptic hex code patterns or this: "couldn't connect'. Windows is the most unhelpful operating system. Back in the early days it was helpful. But now, even for the technically inclined, Windows is a nightmare for troubleshooting corner cases.

Any way, here is a hostapd.conf file working with Windows 10 and the latest drivers (I tested with a couple different Atheros cards). It came down to using particular combinations of WPA parameters, specifically: TKIP and CCMP.

logger_syslog=-1
logger_syslog_level=2
#logger_stdout=-1
#logger_stdout_level=2
#sets the wifi interface to use, is wlan0 in most cases
driver=nl80211
#ieee80211n=1
#ht_capab=[HT40-][SHORT-GI-40][DSSS_CCK-40]
#ieee80211ac=1
#ieee80211d=1
#ieee80211h=1
#wmm_enabled=0
#wme_enabled=1
#country_code=CA
country_code=0
#sets the mode of wifi, depends upon the devices you will be using. It can be a,b,g,n. Setting to g ensures backward compatiblity.
hw_mode=g
#sets the channel for your wifi
channel=0
#macaddr_acl sets options for mac address filtering. 0 means "accept unless in deny list"
macaddr_acl=0
#setting ignore_broadcast_ssid to 1 will disable the broadcasting of ssid
ignore_broadcast_ssid=0
interface=wlp4s0
#sets the ssid of the virtual wifi access point
ssid=yourssid
bssid=02:00:00:00:10:01
#Sets authentication algorithm
#1 - only open system authentication
#2 - both open system authentication and shared key authentication
auth_algs=1
#####Sets WPA and WPA2 authentication#####
#wpa option sets which wpa implementation to use
#1 - wpa only
#2 - wpa2 only
#3 - both
wpa=3
#sets wpa passphrase required by the clients to authenticate themselves on the network
#sets wpa key management
#wpa_key_mgmt=WPA-PSK WPA-EAP WPA-PSK-SHA256 WPA-EAP-SHA256
wpa_passphrase=yourpassphrase
wpa_key_mgmt=WPA-PSK
#sets encryption used by WPA
wpa_pairwise=TKIP CCMP
#sets encryption used by WPA2
rsn_pairwise=CCMP

• OSRG gobgp - BGP implemented in go
• kmotiko openflow in go

From the ovs-dev mailing list:

There are cases where the default gateway of a interface is in a different subnet than its IP address. Linux allows such configuration. For e.g, one could set the IP address of a Linux interface as 172.16.1.2/32 and then give it a default gateway of 172.16.1.1. This can be done for e.g. by running the following commands.

ifconfig eth0 172.16.1.2 netmask 255.255.255.255 broadcast 172.16.1.2 
route add 172.16.1.1 dev eth0 
route add default gw 172.16.1.1

The above configuration is what google cloud uses for its VMs.

In OVN static routes, we currently have the ability to specify the router port via which the packet needs to be pushed out to reach a next hop. But when support for IPv6 was added, we only allowed nexthops to be in the same subnet as one of the router's IP addresses.

This commit relaxes that restriction. When a outport is specified in static routes and when a nexthop is in a different subnet than any of the router IP addresses, we will assume that it is reachable from the first IP address of the router. Since this is a corner case, we just go with the first IP address. If it turns out that there are more cases, we can let users choose the IP address via which the destination is reachable.

From the netdev mailing list:

Ido says:

This patchset aims to prepare the mlxsw driver for VRF offload. The follow-up patchsets that introduce VRF support can be found here: github idosch-next.

The first four patches are mainly concerned with the netdevice notification block. There are no functional changes, but merely restructuring to more easily integrate VRF enslavement.

Patches 5-10 remove various assumptions throughout the code about a single virtual router (VR) and also restructure the internal data structures to more accurately represent the device's operation.

There are many different projects for controlling different aspects of the network in a datacenter. Many of them seem to be more oriented to the hypervisor side of things: how to get containers to talk to each other across a network. Those projects seem to make the assumption that a 'core network' already exists. Which implies that a network is partitioned into management by (at least) two groups of people?

What sort of solutions exist for network management end to end -- ie, being able to provision container networking, distributed networking within the hypervisor, routing/switching in the core, connectivity to the edge/exterior world? This would also include the ability of distributed security. Rather than central firewalls - security is provided at each entry point: each container/guest interface along with some distributed service function chaining to provide additional layers of security on particular flows.

To understand the challenges more intimately, I have been experimenting with these concepts. For instance, openvswitch is a popular platform for switching in hypervisor environments. Under the hood, it makes use of OpenFlow tables to control packet flows. It has a rich set of matches and actions.

The same development group have recently released OVN (open virtual networks), which provides a distributed abstraction for routing/switching and container provisioning. They have a ways to go before routing becomes a solved problem for resiliency. To me, OVN only solves part of the problem of networking throughout a datacenter infrastructure -- mostly having to do with the hypervisor side of things.

At another style of abstraction, Ryu appears to be a popular controller for OpenFlow style networking, and has an active user base. Not only does it know OpenFlow, it has been extended to handle openvswitch extensions, and even 'knows' how to integrate with the openvswitch ovsdb. Some users have integrated it with NetworkX to handle path calculations through a network.

I was initially tempted to run a single instance of Ryu to control various proof of concept routing/switching/hypervisor instances. The single instance of Ryu would use some graph theory to figure out which network element required which subset of functionality. That doesn't necessarily scale well, nor is it very resilient.

Taking a page from the OVN playbook where OVN has a central controller with agents on the hypervisors, something similar could be performed with Ryu. Rather than a central RYU controller, there would be a Ryu controller on each network element. Their effective SouthBound interface would be a distributed key/value store such as what could be provided by Consul. Each Ryu would calculate it's own participation in the network graph represented in the key/value store adjust local OpenFlow rules to suit. This provides distributed control resiliently with a distributed northbound interface for intent based network provisioning.

As part of it's feature set, Consul can also perform health monitoring and some statistics gathering. These statistics and state management can be used with the local path calculations to perform load-balancing and multi-path planning. Theoretically speaking.

I am wondering if this concept can handle distributed MLAG/LACP and VRRP style resiliency functionality. The distributed key/value store is the mechanism for signalling. Would this work?

Who is working on similar functionality?

• linux multipath routing load balancing
• Load balanced, redundant network configuration for Linux using ECMP, Quagga, BGP and OSPF

To help with my diagnosing of multicast with my article Multicast PIMD, a few links I used:

• pimd debug help
• troglobit multicast how to
• rfc4601: Protocol Independent Multicast - Sparse Mode (PIM-SM)
• Network virtualization with VXLAN
• Troubleshooting Multicast with Linux
• iperf for multicast testing
• omping: test ip multicast
• Enable Multicast Routing in the Kernel

• <vans163> anyone worked with connecting OVS/OVN to physical networks?
• <vans163> for example to have dedicated servers be on private lans with virtual ones
• <vans163> I think a programmable router is needed for this?
• <vans163> Like is this needed barefoot chip?
• <russellb> you can do physical network integration in 4 ways
• <russellb> 1) OVN supports top-of-rack based L2 gateways based on the hardware_vtep ovsdb schema
• <russellb> 2) OVN supports L2 gateways via a linux host running ovn-controller
• <russellb> 3) OVN supports connecting VMs directly to a physical network - you do this by adding a special "localnet" port to a logical switch, which tells OVN that the logical switch is actually direct connectivity to a physical network
• <russellb> 4) OVN supports L3 gateway routers that can route between a virtual, overlay network, and a physical network
• <russellb> vans163: github gist
• <russellb> that's a quick overview of #3
• <vans163> that #3 I assume will connect you to a physical vlan, but you still need something programable that you can have define that VLAN in the physical router/switch (if tagging)
• <russellb> yes, you need to also provision your underlay for that vlan
• <russellb> via whatever means suits you
• <russellb> ovn can't do it
• <russellb> you use ansible by chance?
• <russellb> it has some switch config modules you can use for this sort of thing ...
• <russellb> ansible network automation
• <russellb> but there are no shortage of ways to do this stuff

I am using check_mk to monitor some networks, and wanted to do something similar to what Cisco's IP SLA does, and what SmokePing does: check for a rough measure of latency, loss, and jitter on a network. As check_mk is Python based, I wanted something in Python, which I could retrofit to act as a check_mk agent.

I found some code at: Route Reflector, which may, in turn, may have had some of its origins at ping.py – Python Implementation of the ping command.

According to the notes in the code, the original code has been through a number of authors and iterations.

My iteration allows:

• a different nagios output (for check_mk)
• provide a source ip address and source description
• rrd code commented out, but can be uncommented for use

Under normal circumstances, when doing a wildcard (default) bind, and a packet is sent to a destination ip address, the source address will be chosen based upon which is 'closest' (from a routing table perspective) to the destination.

In my tests, I need to test network quality from a number of interfaces on the same device. Therefore, I need to set the source of address of the ping. In addition, with the source address set, the packet has to egress that same interface (routing will cause the egress to be out the interface 'closest' to the destination).

The solution, on a linux based platform, is to use policy based routing.

For a policy based routing example on a device with two interfaces:

• local interface eth0: 10.1.1.1/30
• other end: 10.1.1.2/30 (the default gateway for this example)
• local interface eth1: 10.1.2.1/30
• other end: 10.1.2.2/30
• destination host: 10.1.3.1

Linux iproute2 commands to provide the appropriate routing:

ip route add 0/0 via 10.1.1.2 
ip route add 0/0 via 10.1.2.2 dev eth1 table 6 
ip route add from 10.1.2.1/32 lookup 6 priority 100

A description of the above statements:

• The first default route is the main system default route.
• The second default route is placed into table 6 as a special lookup.
• Then a policy is added meaning that anything from 10.1.2.1/32 uses table 6 for destionation lookup
• Table 6 has the default route out eth1 for sending traffic originating from 10.1.2.1/32

An example ping would be:

/usr/bin/python ~/ping/ping.py -c 20 -t 1 -n int_eth1 -s 10.1.2.1 -d 10.1.3.1

The ping will go out interface eth1 with the policy-based route in place. Without the policy based route in place, the ping will go out eth0.

The code can be found at: Github, rburkholder, ping.py

More info on policy routing:

• jumanjiman / policy-based-routing
• linux-ip.net: IP Routing
• Policy Routing With Linux - Online Edition by Matthew G. Marsh

Other interesting tools:

• github: Python netlink library — Linux network setup and monitoring
• python: Service Function Chaining API for OpenStack Neutron
• Test driving Policy Based Routing on Linux: uses mininet as a test environment

Some other python ping results:

• Another pure python ping implementation using raw socket.
• Pinging with Python
• Ping, Latency, Jitter, and Packet Loss

How do I implement something like this in nftables:

iptables -A INPUT -m conntrack --ctstate INVALID -m limit --limit 3/m --limit-burst 5 -j LOG --log-level debug --log-prefix "INVALID DROP: "
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

nft add rule filter input ct state invalid \
        limit rate 3/minute burst 5 packets \
        log level debug prefix \"INVALID DROP: \" counter
nft add rule filter input ct state invalid counter drop

Note that rule counters are optional in nftables, unlikely iptables where we always have them on.

One can look to /sys/class/net for all things network information wise:

z800:/# ls /sys/class/net
enp1s0  enp2s0  lo

z800:/# ls /sys/class/net/enp1s0
addr_assign_type  address    carrier          dev_id    device   duplex  gro_flush_timeout  ifindex  link_mode  name_assign_type  operstate     phys_port_name  power       queues  statistics  tx_queue_len  uevent
addr_len          broadcast  carrier_changes  dev_port  dormant  flags   ifalias            iflink   mtu        netdev_group      phys_port_id  phys_switch_id  proto_down  speed   subsystem   type

z800:/# cat /sys/class/net/enp1s0/ifindex 
3

z800:/# cat /sys/class/net/enp1s0/iflink
3

z800:/# cat /sys/class/net/enp1s0/operstate
up

z800:/# cat /sys/class/net/enp1s0/statistics/
collisions           rx_compressed        rx_errors            rx_length_errors     rx_packets           tx_carrier_errors    tx_errors            tx_packets           
multicast            rx_crc_errors        rx_fifo_errors       rx_missed_errors     tx_aborted_errors    tx_compressed        tx_fifo_errors       tx_window_errors     
rx_bytes             rx_dropped           rx_frame_errors      rx_over_errors       tx_bytes             tx_dropped           tx_heartbeat_errors  

z800:/# cat /sys/class/net/enp1s0/statistics/rx_bytes 
26730983086

z800:/# cat /sys/class/net/enp1s0/speed
100

z800:/# cat /sys/class/net/enp1s0/mtu
1500

Better late than never, but I stumbled over a blog article at Cumulus where they introduce ifupdown2: a new network interface manager. This is meant to replace the existing ifupdown manager in Debian.

Their description of this new version:

Existing tools for network interface configuration have several shortcomings when applied to network switches. These include the lack of ability to handle interface dependencies, incremental updates to interface configuration without disruption, and interface configuration validation ... ifupdown2 solves these problems through an implementation based on dependency graph.

By looking at their code, maybe I can solve another problem I have. I have been using SaltStack to solve some NetDevOps related configuration tasks. There are built-in state managers for the standard Debian /etc/network/interfaces and ifupdown configurations. But lately, I have been integrating OpenVSwitch related configurations. The problem is that OVS maintains state through a database file. OVS does allow configuration commands through /etc/nework/interfaces: README.Debian for openvswitch-switch. I have also resorted to placing openvswitch commands in /etc/rc.local.

So.... by looking through the code in Cumulus' module, maybe I can reuse their code to come up with some sort of custom SaltStack State Module.

Some links:

• ifupdown 2.0.1 documentation
• iproute2 overview: good description, and good one-liner examples (not completely related to ifupdown)
• netdev.01 meeting slides

The Linux kernel has some powerful features in it. It seems that one can't always find them directly. Sometimes the problem of identification needs to be tackled obliquely. I have known about the existence of TC (Traffic Control via the IPRoute2 facilities) and eBPF (Berkeley Packet Filter).

A newly formed group, IOVisor, well maybe a few years old, but with shiny new mailing list, is building an interesting mix of tools for accessing and utilizing those sub-systems for NFV (Network Function Virtualization) scenarios. They have some demo go and c code at github. From a techy perspective, the most informative page on their web site, although sparse, is theirdownload page where they have links to the bpf man page and tc-bpf man page.

tc, which is for traffic control in the linux kernel, has powerful filter and action built-ins for packet processing, and is mostly known for its rate limiting and congestion control capabilities.

bpf is used in a number of subsystems in the kernel. Effectively, it can execute dynamically loaded code at the machine code level, and can therefore be very fast in execution. Some call it a virtual machine environment in the kernel. Human readable code can be compiled with a llvm or gcc add on, loaded into the kernel, execution paths are validated, and then converted to machine code instructions, and then executed. It makes for very fast and efficient packet handling.

It looks like the IOVisor people are attempting to harness that capability for higher level functionality.

When running dynamic routing protocols, it is desirable to have a unique loopback address on each device taking part in the routing domain. OSPF uses the loopback as an identifier. BGP uses the loopback address as a next-hop destination.

By default, and by standard, Linux has lo addressed with 127.0.0.1. It is tagged as a host only address, as it is only reachable within the host.

$ ip addr show dev lo
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

To add an ip address which would be visible from the network:

$ ip addr add 10.0.0.1/32 dev lo
$ ip addr show dev lo
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.1/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

In the show command, 10.0.0.1/32 is shown as global, and therefore reachable from other subnets.

And the address can be removed just as easily:

$ ip addr delete 10.0.0.1/32 dev lo
$ ip addr show dev lo
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

Linux Kernel 4.3 has had some changes made for VXLAN and MPLS. A Merge Branch 'lwtunnel' has been merged. The patch set is listed at LWN as [PATCH net-next 00/22] Lightweight & flow based encapsulation. The interesting aspect of the code is that the documentation presents examples for creating VXLAN and MPLS light weight tunnels:

 VXLAN:
   ip route add 40.1.1.1/32 encap vxlan id 10 dst 50.1.1.2 dev vxlan0
 MPLS: 
    ip route add 10.1.1.0/30 encap mpls 200 via inet 10.1.1.1 dev swp1

vrf-lite has also been made available. iproute2 tools are used to provide the functionality. It appears to be functional for ipv4 only for now. Utilities such as tcpdump, ping, tc, and netfilter are vrf-lite aware.

Kernel v4.3 will have something called Identifier Locator Addressing. This works with ipv6 only. From the article:

... each task in the data center is assigned a unique identifier that is not tied to any specific location in the net. That identifier is built into that task's IPv6 network address; the networking subsystem then does the necessary magic to route packets properly between them, changing the routing as needed as the task moves between machines.

Another addition to kernel v4.3 is a ipvlan driver A pdf with background information. Fundamentally, this is for l2/l3 based namespaces to share the mac address of a physical port (for instances where external switch policy only allows a single mac address per port).

Creating Overlay Networks Using Intel Ethernet Converged Network Adapters

IpRoute2 Stuff:

• baturin.org: lots of good recipes, including namespace support, l2tpv3, vxlan, multicast, and network event monitoring
• IPROUTE2 Utility Suite Howto: kernel circa 2.2 - 2.6. usual stuff plus ip tunnel, ip rule, ip monitor, multiple route tables
• How To Use IPRoute2 Tools to Manage Network Configuration on a Linux VPS: Digital Ocean related notes for their VP service
• xmodulo: Linux TCP/IP networking: net-tools vs. iproute2
• Information about gmane.linux.network: mailing list for Linux network development
• Simulating VRF Lite with iproute2
• Linux Advanced Routing and Traffic Control HOWTO
• Application Layer Packet Classifier for Linux: obsolete, and even going to Clear Foundation results in kernel v2 stuff. But the original l7-filter page has links to various firewall applications, including something from ZeroShell, which side-tracked me to Kerberos Protocol Tutorial.
• iproute2: Linux Foundation
• iproute2 From Wikipedia
• Linux NetDev Mailing List
• mplsadm: a cached page for mplsadm, really old, not really relevant, but an indicator for manually maintaining mpls labels