When using the BGP module in Free Range Routing, the 'network The draw back to advertising connected prefixes is that the prefix is advertised even a related interface is not 'up'. This could lead to a blackhole scenario.
A better way to handle the advertisements of connected prefixes is to use the 'redistribute connected' command.
Even with the use of this command, there may be scenarios (which I need to test at some point) where the prefix is advertised or withdrawn depending upon the link state. Free Range Routing has an additional command which could be used to ensure link state checking: 'bgp network import-check'.
There is more about the Linux state checking flags in the
Why Link-State Matters presentation from LinuxCon 2015.
In addition, the Free Range Routing developers have brought together some
relevant sysctl settings.
Monday, April 30. 2018
Linux Link State and Free Range Routing
Linux Drive Access
When a drive has lots of activity for seemingly no reason, what tools are available to troubleshoot? Here are a few possibilities.
Debian provides a tool called iotop, via the package iotop, which, by default, provides a 'top'-like experience. A different command line experience can be achieved through something like (which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used):
# iotop -o -b -d 10 Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND Total DISK READ : 0.00 B/s | Total DISK WRITE : 2041.86 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 1837.68 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 19967 be/4 root 0.00 B/s 2041.86 B/s 0.00 % 0.00 % conntrackd -C /etc/conntrackd/conntrackd.conf 14023 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % perl /usr/sbin/x2gocleansessions
More use cases are available at How to find which process is regularly writing to disk?
General statistics, which could be coupled with a 'wait', or in this case it refreshes itself with -k:
Continue reading "Linux Drive Access" »# iostat -xk 2 /dev/sdb Linux 4.14.0-0.bpo.3-amd64 (host01.ny1) 05/26/18 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.31 0.00 0.66 0.09 0.00 98.94 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 1.78 0.24 6.80 98.41 432.03 150.76 0.60 85.97 114.91 84.95 2.13 1.50
Network Operations Industry Trends
Based upon the Call for Proposals for ONS Europe 2018 to be held in Amsterdam in September 2018, this is what the industry appears to be thinking about:
- Networking Latest Trends
- Disaggregation of the Data Plane – Programmable Open Hardware including Silicon & White Boxes + Open Forwarding Innovations/Interfaces
- The Latest in Standards and Open Source Interworking to Drive Industry Forward
- Artificial Intelligence, Machine Learning, and Deep Learning Applied to Networking
- SD-WAN, IIOT, Data Insights, Business Intelligence
- Cloud Native, Kubernetes, and Network Automation
- Containers, Microservices, and the Service Mesh
- Networking Business and Operations
- High Availability, Scalability, Security in Network Platforms
- Next Generation Automated Architecture (Enterprise, Carriers, Cloud)
- Business Case & ROI on Transformation, Next Gen Services Based on SDN/NFV
- Lessons Learned from Migrating/Integrating Legacy PNF/VNF/OSS/BSS to Automated Systems
- Migration of Data Center to Virtualized Software Defined Lessons and Roadmaps
- The Road to 5G deployment; POCs, Pilots, Partners, and Roll-out
- (Technical) Service Provider & Cloud Networking
- Policy, Closed Loop Automation, and Zero Touch
- Edge Computing; Requirements, Tools, and Emerging Solutions
- NFV Security
- Building High Availability in SDN/NFV Solutions
- 5G Application Use Cases; What Comes First?
- The Shift to Microservice-based VNFs
- Multi-Cloud, and the Benefits it Brings
- (Business & Architecture) Service Provider & Cloud Networking
- SDN/NFV Learnings to Date, ROI, TCO, Agility Gains, etc.
- Business Case and Strategy for 5G Workloads; IoT, VR/AR, Connected Cars, etc.
- Software Defined Packet Optical
- Mobile Edge Computing/4G Video/CDN
- Planning the Migration – Legacy Workload Migration, and Integration of New Systems with Existing Carrier OSS/BSS/FCAPS Systems
- (Technical) Enterprise IT DevOps
- Software Defined Data Center Learnings
- Cloud Networking, End to End Solution Stacks – Hypervisor or Container Based
- Container Networking and the Shift to Microservices
- Scale and Performance in SDN Deployments
- Cloud Native App Development
- (Business & Architecture) Enterprise IT (CXO/IT Architects)
- Digital Transformation – Best Practices
- Bringing DevOps to NetOps – Best Practices
- SDN, and Open Networking Case Studies; ROI, TCO, Operations Improvements, etc.
- Intent-Based Networking
- NFV for Enterprise (vE-CPE, etc.)
- Security – Evolving attack surfaces, agile attack counter-measures, etc.
- Industrial IoT, and the Smart Enterprise, Networking’s role
Sunday, April 29. 2018
Message Stream Processing
I've been able to get my head around what Kafka by looking at it obliquely.
Mark Fletcher's blog entry is more PostgreSQL oriented but offers up some interesting ideas on sequencing messages between sub-systems.
5 tips for architecting fast data applications introduces some of the mechanisms large web sites handle the quantity of requests and responses.
2018/05/18 But it seems that Kafka can be difficult and complicated to setup and configure. It does have the redundancy and flexibility to undertake many difficult message passing scenarios. But is there something a bit easier to start with?
Maybe MQTT has an opportunity. LWN has An introduction to MQTT. MQTT is used extensively in the IOT world, which includes tooling supplied by Home Assistant.
I was somewhat leary of using MQTT, as there didn't appear to be robust, distributed solutions available. Well, there are:
- EMQ: "The Massively Scalable MQTT Broker for IoT and Mobile Applications" - based upon Erlang.
- VerneMQ - a MQTT publish/subscribe message broker which implements the OASIS industry standard MQTT protocol
To continue with this theme, Jan-Piet Mens has a blog article called How do your servers talk to you? where he discusses integrating some code bits he has, nagios/icinga, and mqtt to create alerts and messaging which can be processed via Pushover which is a PagerDuty alternative. Mens has another article exploring other messaging possibilities: Alerting or notifying on SSH logins.
Jonathan McDowell writing about MQTT and temperature sensors, and MQTT with automation.
2019/03/25 The Case for Database-First Pipelines talks about Kafka and Databases, in which the supposition that Kafka exists between everything is debunked. Sometimes you have to let the database be a database and issue the responses. Which, now that I've written that, simply means that the database is just another event generator, and Kafka is just the event passer, which seems pretty straightforward to me.
Articles Encountered
More items linked:
- The 50 Best Fantasy Books of the 21st Century (So Far)
- ZFS on Linux 0.7.8 Released to Deal with Possible Data Loss - something to be aware of
- Linux IPsec workshop - 2018, 26 - 28 March, Dresden -- a running summary and some links to discusions about IPSEC/XFRM/Linux. -- hardware offload, dscp re-order and drop, ...
- Common Applications Kept Enhanced (cake) qdisc - sch_cake targets the home router use case and is intended to squeeze the most bandwidth and latency out of even the slowest ISP links and routers, while presenting an API simple enough that even an ISP can configure it.
- LWN talks about a text based accounting system called . The comments section refer to another GPL oriented package called uzERP written in PHP.
- Policy Based Routing has arrived in Free Range Routing. The FRR commands map to native Linux route/rule/table constructs. Basic commands are documented.
- Packaging an out-of-tree module for Debian with DKMS by Vincent Bernat, 2018/03/07
- Route-based IPsec VPN on Linux with strongSwan by Vincent Bernat, 2017/09/13
- TweetDeck abuses my browser (blows up on memory), so I need a different way of doing this. Perhaps Deploying a "Native" TweetDeck App in Linux might be the solution. Another reference: mikebell/tweetdeck-desktop on github.
- In A blog about C, C++, Conan, Binaries and Devops, there are two references about running Boost on Android: Cross building Boost C++ libraries to Android with Conan and Android Studio project using Conan and C++ Boost libraries. Following these steps, it may pave the way for getting other libraries like OpenCV on to the Android.
- I kept a link to LXD, ZFS and bridged networking on Ubuntu 16.04 LTS+ for several reasons: it has a interesting tutorial on LXD and ZFS. Although I prefer Open vSwitch bridging over the native bridging. But more importantly, half way down in the article are some tuning parameters.
The tuning parameters suggested when running a multiple container system to prevent “too many open files” errors. In /etc/sysctl.conf, add:
fs.inotify.max_queued_events = 1048576 fs.inotify.max_user_instances = 1048576 fs.inotify.max_user_watches = 1048576
The following changes are suggested for /etc/security/limits.conf:
* soft nofile 100000 * hard nofile 100000
Notes on Resiliency - VRRP, AnyCast
This is another collection of random notes, this time, on how to build something on Linux somewhat resembling Cisco's Global Load Balancing capability, basically a continuation of my entry at Linux ifupdown2 VRRP.
Traditionally, one sets up VRRP using keepalived or the simpler vrrpd. This configuration is typically used when setting up (typically) two routers in an active/passive setup to act as a gateway for a network subnet. In essence, the two (or more) routers negotiate who will hold the gateway mac and ip address.
In other circumstances, it might be desired (and possible) to run active/active. This is a possibility when running containers on a host, and there are similar services running across the hosts. In this instance the same address can be assigned as a secondary address across multiple containers to load balance traffic.
And in even other cases, subnets may be stretched in a layer2 over layer3 encapsulated network across multiple hosts. And in this case, each host should be able to act as a gateway for the traffic local to it. It is this last example I am currently investigating.
Reynold's Blog has an entry called Configuring Cumulus Linux High Availability Layer 2 Network. The most interesting aspect of this post is reference to using 'address-virtual' commands when using ifupdown2 style /etc/network/interface structures:
address-virtual 00:00:5e:00:01:02 10.11.2.254/24
The ip and mac addresses are identical across interfaces sharing the gateway role. The mac address is a reserved range 00:00:5e:00:01:00 – 00:00:5e:00:01:ff for VRRP style operations. The ip address is the virtual ip address (VIP). This style of usage is explained more in Virtual Router Redundancy - VRR.
Or maybe I don't worry about this as Ethernet Virtual Private Network - EVPN has a section with asymmetric routing and symmetric routing which do not need vrrp style constructs.
Layer 3 routing on Cumulus Linux MLAG talks about VRR, the address-virtual, and FRR/route-map to obtain ECMP based load balancing. Now the question - how to get things to not need MLAG.
Tuesday, April 24. 2018
More Open vSwitch Commands ...
... from the mailing list:
and//You can verify that OVS supports deletion by cookie specification by checking these tests: 932: ofproto.at:1746 ofproto - del flows based on cookie 933: ofproto.at:1767 ofproto - del flows based on cookie mask // Run the tests using “make check” like this: sudo make check TESTSUITEFLAGS='932-933' -C _gcc //Look at what is being tested here: tests/ofproto.at
Maybe you want to check what Ryu is sending the switch http://www.openvswitch.org//support/dist-docs/ovs-ofctl.8.pdf “OpenFlow Switch Monitoring Commands: monitor switch [miss-len] [invalid_ttl] [watch:[spec...]] Connects to switch and prints to the console all OpenFlow messages received. Usually, switch should specify the name of a bridge in the ovs−vswitchd database.“ eg) ovs-ofctl monitor br0 65534 invalid_ttl --detach --no-chdir --pidfile 2> ofctl_monitor.log
Thursday, April 19. 2018
nftables: connection tracker helpers
From a mailing list entry:
Question:
using nft from nftables, I created some IP filter rules inside a partially virtualized (Linux Vserver, www.linux-vserver.org) machine. Almost all rules are working as desired, but rules that need connection tracking helpers, like ftp and tftp, do not . some ip packets are blocked though they should be allowed. As the same tftp rules - I am sure that I made no mistake - work on a real host, there is probably some requirement for these helpers to work correctly and that is not fulfilled inside a Vserver.
Answer:
In recent kernels no default assignments of helpers is done anymore, iptables users need to use -j CT target, nft users need to add a helper object:
nft add ct helper inet filter bar '{ type "ftp" protocol tcp; }' nft add rule inet filter output tcp dport 21 ct helper set "bar"The assignnment needs to be done in the direction that creates the connections that need the helper.
So for a local host (connecting to remote server), this needs to be output; for a server (expecting ftp connections), input.
For a gateway it can be in forward, or prerouting and output in case its needed everywhere (local and forwarded).
Also it makes sense to limit helper assignemnt to connections that need it (e.g. ip saddr 192.168/16 or somesuch).
With a later addendum:
As I do not have the required nftables and kernel versions, I reactivated default assignment withecho 1 > /proc/sys/net/netfilter/nf_conntrack_helperlike described at the bottom of connection tracking meta-information
Monday, April 16. 2018
Reading List
- Proper isolation of a Linux bridge talks about Linux bridges at the networking stack level. Very information and contains some isolation mechanisms I need to implement.
- alternative approach to rate limiting: rate limiting tcp requests for various services. has good technical discussion of various approaches (token bucket, window counters, sliding window log, sliding window counters) with a suggested purpose-built solution.
- Tcpdump: many examples of tcpdump queries
- 2018/05/18 - E-mail Cryptography
Bash Links
Wednesday, April 4. 2018
Migrating LXC Containers From One Machine To Another
For some machines with LXC containers, they have been running for a number of years. I want to take the easy way out and move the containers from one physical machine to another. At another time, I will rebuild the containers.
Since I am running BTRFS subvolumes for each container, I could be using BTRFS snapshot/send/receive commands to migrate/copy/replicate subvolumes. But before attempting that, I wanted to give the 'copy' a try. To do this properly, at the source, use the following -- with numeric-owner being a required paramenter -- command to collect the files:
tar --numeric-owner -czvf mycontainer.tar.gz /var/lib/lxc/my_container
At the destination, expand that file out:
tar --numeric-owner -xzvf mycontainer.tar.gz -C /var/lib/lxc/
The lxc users mailing list and Stack OverFlow were helpful.
Other stuff to do:
- Read up on CGroups in the Linux Programmer's Manual
In migrating from a very old version of LXC to a much newer version of LXC, I was getting errors. I needed to run a some debug to get a handle on errors:
lxc-start -n container -F --logpriority=DEBUG --logfile log
I had errors along the lines of:
Activating lvm and md swap...done. Checking file systems...Segmentation fault (core dumped) failed (code 139).
ServerFault had the solution: put "vsyscall=emulate" into /etc/default/grub, run 'update-grub' and reboot. Looks like I need to modernize my containers so I can eliminate this workaround, which may have some security considerations. There is a Debian Bug for this.
einstein home has a blog with some kernel references to the issue, in effect saying: "vsyscall is now disabled on latest linux distros". A lengthier LWN article at On vsyscalls and the vDSO. This works with kernel 4.14, my current version, but I see somewhere else that the workaround is entirely removed in kernel 4.15, at least in the Arch world. At bug 847154: "This breaks (e)glibc 2.13 and earlier".
Sunday, April 1. 2018
HP DL360 G6 ilo Configuration Utility: hponcfg
Some articles I need to go back to see what I can do with the iLO from the Linux Command line.
- iLO Configuration Procedure using hponcfg Utility: probably the most detailed of the list in terms of examples for reading and writing the config.
- Updating iLO Firmware Using Hponcfg and XML: a way to update the firmware, if there are firmware files available
- Scripting Toolkit for Windows - Using HPONCFG: has the first command I ran to get a basic XML configuration
HP DL360 G6 P410i with Debian Stretch
I have a couple old HP DL360 G6 servers running a several year old version of Debian Stretch. It became time to update them. Rather than fooling around with an upgrade, this is a re-install scenario. The servers have ILO2, with very lame keystroke ability. I had to use a combination of the ILO2 Remote Access in Internet Explorer as the viewer with an ssh login on another window to get keystrokes and menu operations going. I think it was a lot easier back with original Internet Explorer and Java. The Remote Mount of a CD also worked after a couple attempts at getting everything correct.
With the operating system installed, I used the Linux HP Smart Array Raid Controller article to determine the mechanism for installing HP proprietary tools. HPE (Hewlett Packard Enterprise) has a Software Delivery Repository. I am glad I hit the guy's article first, as I would not have known what to do without it.
Here is the server type determined via DMI:
dmidecode | grep -A3 '^System Information' System Information Manufacturer: HP Product Name: ProLiant DL360 G6 Version: Not Specified
PCI information shows the RAID Controller type:
# lspci -k|grep -i -A2 raid 03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01) Subsystem: Hewlett-Packard Company Smart Array P410i Kernel driver in use: hpsa
Which confirms kernel modules in use for my SAS drives:
# lsmod |grep hpsa hpsa 102400 2 scsi_transport_sas 45056 1 hpsa scsi_mod 253952 8 sd_mod,usb_storage,scsi_transport_sas,libata,hpsa,uas,sr_mod,sg
Since I am using Debian Stretch, here are some revised commands to get at some tools. This is the closest I could get to a set of descriptions.
# apt install curl # curl https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add - # echo -e "deb http://downloads.linux.hpe.com/SDR/repo/mcp/ stretch/current non-free" \ > /etc/apt/sources.list.d/hpe.list # apt update # apt install ssacli # apt install ssaducli # apt install hponcfg
ssacli provides access to the raid controller information. ssaducli is supposed to show wear rates, but it generated an empty report for me. And hponcfg deals with ILO management. More articles are needed for decoding that beast. Continue reading "HP DL360 G6 P410i with Debian Stretch" »
Debian Links I Lose
I have a couple older HP servers I acquired prior to understanding free firmware vs non-free. In the Debian world, when there are non-free network drivers, this can be an issue, as the non-free drivers are not included with the standard distribution downloads. Case in point, the bnx2 firmware. There are a ways to build your own distribution, but Debian, via an out of the way location, does provide an installable image: Unofficial non-free images including firmware packages
When trying out Debian Testing, here is the Debian Testing Installer page.