Raymond P. Burkholder - Things I Do - OpenFlow

Using Quilt to Patch a Debian Package - hostapd

nospam@example.com (Raymond P. Burkholder) — Sat, 27 May 2017 05:46:12 +0000

It took a little while to determine the root cause, but, ...

I am rebuilding an old windows workstation into a Linux based router/firewall/access-point using hostapd to provide wireless interfaces. Normally, this is a straight-forward configuration. The twist in this case is that I am using Open vSwitch to handle layer 2 functions. The box has an existing Atheros AR5B22 based PCIe 1 card with two antenna connections on the rear face. Which translates into a Qualcomm Atheros AR9462 chipset.

I added the wireless interface to OVS on vlan 10 with a command like:

ovs-vsctl add-port ovsbr0 wlp4s0 tag=10

I used a simple hosted configuration file like:

# cat /etc/hostapd/hostapd.conf
interface=wlp4s0
driver=nl80211
ssid=test
auth_algs=1
wpa=1
#wpa_psk_file=/etc/hostapd/hostapd.psk
wpa_passphrase=testtest
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP TKIP
rsn_pairwise=CCMP

When running in debug mode:

hostapd -d -K -t /etc/hostapd/hostapd.conf

I was seeing EAPoL (Extensible Authentication Protocol over LAN) based retries. It took some research to come across the interaction about openvswitch and hostapd. Bottom line, is that hostapd uses a control channel to control the wireless device. EAPoL is a packet based function. AS such, hostapd needs to monitor the packets to pick out the EAPoL packets. A mechanism is built in for working with regular Linux bridged networking. That code does not work with Open vSwitch. Someone created a patch to work around the issue.

This patch isn't found in mainline hostapd, nor as a patch in the Debian repository. So... I need to apply it manually. I used the documentation found at Debian Building Tutorial as a starting point. The divergence is that the documentation uses a non-functional, deprecated tool called dpatch. I used quilt to handle patching.

As a sidebar, this server I am building uses LXC containers to segregate functionality, compartmentalise security, and to make it easier to keep the main install minimal. As patching and rebuilding the package requires a bunch of build tools, the tools can be easily installed, and then the whole build environment deleted when complete.

The annotated series of steps. I have a number of commented-out entries which I plan to play with later, once I understand more the nuances. One thing, there appears to be a lock out of 5gig frequencies when acting as a host.

apt update
apt install build-essential fakeroot devscripts # install tools
apt-get source hostapd  #obtain source

# review the source directory, then
apt-get build-dep wpa  # install the build dependencies
cd wpa-2.4/
debuild -b -uc -us  # build with existing source
dpkg -i ../hostapd_2.4-1_amd64.deb  # install the package as a test
apt install quilt  # install patch manager
quilt top  # inspect the current latest patch
ls -alt debian/patches/  # most patches kept here
ls -alt patches/  # some are here as well

quilt new rpb_hostapd_openvswitch  # create a new patch
# in the following edit, I removed the content, and pasted the source from
# https://github.com/helmut-jacob/hostapd/blob/master/src/drivers/linux_ioctl.c
quilt edit src/drivers/linux_ioctl.c  # source file to change, leave function linux_master_get in place
quilt refresh  # refresh
quilt top  # my new patch is at the top
cat debian/patches/rpb_hostapd_openvswitch |less  # this is my patch
quilt diff  # show the diff colorized
quilt push  # add the patch to the list of patches
dch -n   # update the changelog and version

debuild -b -uc -us  # build with the new patch

# the patch can then be applied and tested (direct from the lxc container):
dpkg -i /var/lib/lxc/apd/rootfs/usr/src/hostapd_2.4-1.1_amd64.deb

With the patch, clients can now successfully associate and authenticate with hostapd when the wireless port is connected to an Open vSwitch bridge.

hostapd sample configurations:

Hostapd: from a Gentoo perspective, with an intro to multiple AP and some references to 802.11ac with DFS
CRDA Regulatory Code - getting hostapd to run in the 5g bands
Edison AP mode in 5GHz - putting two and two together
w1.fi hostapd.conf example
hostapd docs which include some 802.1x suggestions and for running with segregated plans. freeradius is also needed when running 802.1x style configurations as the radius protocol is used for communicating the configurations to end devices.
simple hostapd/radius config
openwrt / openvswitch: background information on the hostapd / openvswitch issue
OpenFlow Isolation: use openflow / openvswitch to isolate wifi networks and users, which is an improvement on the psk per user and 802.1x per user vlan. with ovs-ofctl examples
MAC address spoofing, and since the pre-shared key in a previous step might be MAC based, here is a way to get around the limitation.
hostapd and dhcp: simple wifi config
SDN: Establishing a Session Database for SDN Using 802.1X and Multiple Authentication Resources -- interesting presentation on open flow, 802.1x, sdn, radius, ...
Ath9k Chipsets

2018/09/16 Other configs (for 5ghz):

2020/08/21 From the mailing list:

For "normal" wifi you can't simply bridge the wlan interface of the client. You can bridge it on the AP. This is, because in wifi you don't have a source-MAC and a destination-MAC, but also a transmitter-MAC and a receiver-MAC. As an optimisation, in ap2sta direction the destination and the receiver are a single field. In sta2ap direction, the source and the transmitter are a single field
Take a look at 4addr mode which solves the issue by having actually all 4 fields. This is a linux specific implementation and may not be available on all types of cards.
An alternative would be to have a routed connection between the two devices and bridge your ovs-instances via GRETAP.

2020/12/09 - some additional settings and notes - on Debian hostapd gets installed with out a configuration file. Once the configuration file is in place, run the following to auto-start hostapd:

sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl start hostapd

The commands are from: Failed to start hostapd.service: Unit hostapd.service is masked. #1018

Another note: many/most/all Intel adapters are designed for client side operations. Use an ath5k/ath9k/ath10k adapters for server/host side operations. The key aspect here is that the channel=0 entry in the configuration file will result in errors and exit of hostapd as it cannot find survey data when mapping the channels. The channel parameter needs to be set to 1, 7, or 11 (non overlapping channels). For example:

ACS: Unable to collect survey data
ACS: All study options have failed
Interface initialization failed

Some interesting points from Intel Wireless 7260 as an access point:

First, the card is said “dual-band” but you can only uses one band at a time because there is only one radio. Almost all wireless cards have this limitation. If you want to use both the 2.4 GHz band and the less crowded 5 GHz band, two cards are usually needed.

c++ libraries useful for openflow

nospam@example.com (Raymond P. Burkholder) — Thu, 11 May 2017 13:05:21 +0000

OpenFlow Header Files
Revised OpenFlow Library (ROFL) - this looked like an excellent library, but I have problems trying to make the controller and the ethswitchd examples function with out crashing or not reacting as expected. And it uses and integrates it's own packet send/receive functions.
libtins - packet crafting and sniffing. It would be interesting if the crafting portion was removed from the send/receive portion, which make a good open flow packet encoder/decoder.
boost graph - max flow/min cut algorithms
min cost max flow with negative weights
amedama's github for open flow, with graph code
runos - another controller, but with lots of other bits added
logcabin - Raft protocol library
OpenFlow 1.4.1 spec
OpenFlow 1.5.1 spec
beehive - Distributed SDN controller built on top of beehive. Open flow packet definitions base upon:
Packet - "protocol buffer" for network protocol

Know of any c++ based packet serializers, coders/decoders, to process/build open flow packets?

Raft:

SDN Papers:

Automatic belief network modeling via policy inference for SDN fault localisation: Springer Open Access paper on - Modeling via Policy Inference (called MPI) is a highly scalable, effective and flexible modeling approach to tackle fault localization challenges in a highly dynamic and agile SDN network.

Other openflow stuff:

OpenFlow controller programming framework that provides everything needed to create OpenFlow controllers in Ruby
Case Study with OpenWifi
OpenFlow 1.3 for OpenWRT

Other not relevant C++ oriented stuff encountered:

RapidJSON
optitrack - natural point motion capture streaming library
NatNet - the SDK
NetworKit - social graph analysis
list of cpp libraries
Erasure Coding for storage devices with a paper. More about erasure codes via Hacker News to an article called A Gentle Introduction to Erasure Codes
mFAST - A FAST (FIX Adapted for STreaming) encoder/decoder)

Algebra, Graphs, ...

Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers, and related algorithms

Network Simulators

nospam@example.com (Raymond P. Burkholder) — Mon, 27 Mar 2017 19:27:14 +0000

In looking for open source routing and open flow solutions, I came across Brian Linkletter's site called Open-Source Routing and Network Simulation. I never realised that there were so many network simulators out there. His site specializes in evaluating many of them.

Two reviews stand out. The first is DNS and BIND demonstration using the Cloonix network emulator which provides a mechanism for experimenting with the DNS protocol and the various open-source implementations of DNS including BIND.

The second interesting network emulator is the OFNet SDN network emulator. it comes prepackaged as an .ova file. The article indicates that it comes with the Floodlight and Beacon controllers. I wonder if Ryu could be used? Anyway, based upon the article, the emulator has visual tools for watching OpenFlow messages, evaluating topologies, and show traffic patterns. Further into the article, the author installs OpenDaylight, so it does look like other controllers can be trialled.

OpenFlow References

nospam@example.com (Raymond P. Burkholder) — Mon, 14 Nov 2016 02:29:15 +0000

Intent Based Networking: published by ONF as a higher layer abstraction to network configuration
ONF Technical Library: with technical specifications for the various versions of OpenFlow. Follow the 'SDN Resources' button on the upper right for more resources.
Skydive: open source real-time network topology and protocols analyzer. It aims to provide a comprehensive way of understanding what is happening in the network infrastructure. Skydive is SDN-agnostic but provides SDN drivers in order to enhance the topology and flows informations.
Faucet: OpenFlow controller for a layer 2 switch based on Waikato University's Valve. It handles MAC learning and supports VLANs and ACLs. It is developed as an application for the Ryu OpenFlow Controller . Examples make use of the Northbound Networks Zodiac FX OpenFlow Switch at $AUD 99. It doesn't do much, if anything with encapsulations (like MPLS or VXLAN or GENEVE) -- various references indicate 'on the to-do list'. Also, in the main faucet github page, they mentioned that Table-Type-Patterns weren't used, which I am starting to think are important for larger, more complicated configurations.
Inside OpenFlow: is dedicated to providing a broad array of code examples showing how networking task are implemented with OpenFlow technology.
ATOM: hackable open source text editor, useful for Python coding, has auto-completion
Postman: Create and send any HTTP request using the awesome Postman request builder. Write your own test cases to validate response data, response times, and more! Good for testing and working with REST interfaces.
The Benefits of Multiple Flow Tables and TTPs where VRFs and VLAN defined by OpenFlow multiple tables are described.
Allied Telesis x510 and x930 series openflow capable switches.
NoviFlow switches
Netronome PCIe cards with off-load: Agilio CX cards
Colfax Direct: HPC and Data Center Gear
OpenNFP: collaborative research in the area of network function processing in server networking hardware

A very detailed and useful paper: Sofware-Defined Networking: A Comprehensive Survey

OpenFlow Thinkings

nospam@example.com (Raymond P. Burkholder) — Sun, 13 Nov 2016 20:20:33 +0000

t has taken a while, but I think I understand how some flavours of SDN work.

In CiscoLand, at the hardware level, CEF is a flow based forwarding mechanism. All the higher layer routing protocols like bgp, ospf, rip, etc, are used to build that flow table. I think Cisco's ACI is designed to manipulate those flow tables directly through some set of off-box oob controllers. These flows are monitored through Netflow, sFlow, OpenFlow telemetry commands, and the like. So flows are integral to how networks work. How they are controlled is the question.

I used to be of the mentality that this central control created points of failure in the network. But if boxes can maintain state during a controller failure or outage, then it isn't so bad. In addition, many initial talking points mention that flow tables needed to be dynamically populated, which requires punting packets to the controller, which really slows things down.

But many networks have a reasonably static topology. This enables flow bundles to be pre-populated and thus hardware forwarding can be used for the majority of the traffic, if the flow rules are general enough. From a high level perspective, then the issues arise when network failures occur. How does the network heal itself around those failures points? Some answers I am still looking into.

In the open source world, Linux, in the kernel forwarding plane, also maintains a flow table. sFlow can be used to monitor flows. There are netflow add-ons to monitor those flows. Static routes and things like Quagga are used to define the rules for how flows are setup.

So, enters the concept of OpenFlow, which is a set of rules used to manage those flow tables directly. Either locally, or through some off-box controller. (I have been looking into Ryu at the moment).

When using layer 3 network protocols like bgp, et.al., the flow rules are typically defined based upon destination ip address.

With OpenFlow, flows can be specified on a number of different combinations of addresses, ports, mac addresses, encapsulations, ...

Now take that concept, and make the realization that you can create a distributed firewall, because the network itself can provide firewall and associated security features. Rather than creating flow rules based upon destination address, ... with the added granularity, flow rules can be specified that only specific types of traffic (eg port 80, or esp, or ...) go through various parts of the network, and traffic not matching those flow rules go somewhere else, say like a honey-pot or something.

And with various forms of encapsulation from edge to edge, various forms of vrf/vpn/segmentation are inherently supplied.

So, it is conceivable that the central points of congestion like big central firewalls can be eliminated, and the security and privacy can be distributed throughout the network. Kind of like MPLS/VPLS on steroids.

From the open source perspective, I have just started going through the examples for OpenVSwitch (OVS) and its companion product Open Virtual Networking OVN at Spinhirne's Blog . The examples show some single points of failures, but I think I know how to make those better. It is looking promising based upon what I've seen so far, but have more under-the-hood looking to do.

OpenFlow on Debian Stretch with OpenVSwitch, RYU, and Mininet

nospam@example.com (Raymond P. Burkholder) — Mon, 31 Oct 2016 00:54:03 +0000

The most succinct demonstration of OpenFlow on OpenVSwitch is the Openflow version 1.3 tutorial on SDN Hub.

They make use of a pre-packaged VM for their demo. Which is easy enough to do with basic packages:

$ apt-get update
$ apt-get install openvswitch-switch python-openvswitch  ryu-bin python-ryu mininet tshark

The openvswitch installation will install and start up the openvswitch service. OpenVSwitch does regular switching and can also act as an OpenFlow agent. For the purposes of the demonstration, openvswitch should be set for OpenFlow version 1.3.

$ ovs-vsctl set bridge s1 protocols=OpenFlow13

An OpenFlow agent needs to be controlled by an OpenFlow controller. A large number of them are available. OpenSource. ClosedSource. or in between.

I use SaltStack for provisioning, automation and orchestration. It is a Python based tool. To stay with the theme of Python, I have selected RYU as a controller, which is developed in Python. The RYU install comes with a number of example scripts. A simple switch controller can be started with:

$ ryu-manager --verbose  /usr/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py

It listens on port 6653 and 6633:

t# netstat -ln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6653            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
udp        0      0 0.0.0.0:68              0.0.0.0:*

Mininet is a nice little utility which makes use of Linux' namespace and cgroups to setup emulated hosts and uses openvswitch to handle switching. This example makes use of a simple switch with three hosts. When starting up, it initializes the topology, and then provides a prompt for running commands.

$ mn --topo single,3 --mac --controller remote --switch ovsk,protocols=OpenFlow13
*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2 h3
*** Adding switches:
s1
*** Adding links:
(h1, s1) (h2, s1) (h3, s1)
*** Configuring hosts
h1 h2 h3
*** Starting controller
c0
*** Starting 1 switches
s1 ...
*** Starting CLI:
mininet> h1 ping h2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=19.2 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.312 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.082 ms
^C
--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2028ms
rtt min/avg/max/mdev = 0.082/6.534/19.209/8.963 ms
mininet>

Additional ports for communicating:

$ netstat -ln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6653            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:6634            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
udp        0      0 0.0.0.0:68              0.0.0.0:*

Dumping flows of the resulting actions:

# ovs-ofctl dump-flows s1 -O OpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x0, duration=185.861s, table=0, n_packets=24, n_bytes=1856, priority=0 actions=CONTROLLER:65535
 cookie=0x0, duration=180.186s, table=0, n_packets=5, n_bytes=378, priority=1,in_port=2,dl_dst=00:00:00:00:00:01 actions=output:1
 cookie=0x0, duration=180.181s, table=0, n_packets=4, n_bytes=336, priority=1,in_port=1,dl_dst=00:00:00:00:00:02 actions=output:2

Continue reading "OpenFlow on Debian Stretch with OpenVSwitch, RYU, and Mininet"