When using the BGP module in Free Range Routing, the 'network The draw back to advertising connected prefixes is that the prefix is advertised even a related interface is not 'up'. This could lead to a blackhole scenario.
A better way to handle the advertisements of connected prefixes is to use the 'redistribute connected' command.
Even with the use of this command, there may be scenarios (which I need to test at some point) where the prefix is advertised or withdrawn depending upon the link state. Free Range Routing has an additional command which could be used to ensure link state checking: 'bgp network import-check'.
There is more about the Linux state checking flags in the
Why Link-State Matters presentation from LinuxCon 2015.
In addition, the Free Range Routing developers have brought together some
relevant sysctl settings.
Monday, April 30. 2018
Linux Link State and Free Range Routing
Linux Drive Access
When a drive has lots of activity for seemingly no reason, what tools are available to troubleshoot? Here are a few possibilities.
Debian provides a tool called iotop, via the package iotop, which, by default, provides a 'top'-like experience. A different command line experience can be achieved through something like (which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used):
# iotop -o -b -d 10 Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND Total DISK READ : 0.00 B/s | Total DISK WRITE : 2041.86 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 1837.68 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 19967 be/4 root 0.00 B/s 2041.86 B/s 0.00 % 0.00 % conntrackd -C /etc/conntrackd/conntrackd.conf 14023 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % perl /usr/sbin/x2gocleansessions
More use cases are available at How to find which process is regularly writing to disk?
General statistics, which could be coupled with a 'wait', or in this case it refreshes itself with -k:
Continue reading "Linux Drive Access" »# iostat -xk 2 /dev/sdb Linux 4.14.0-0.bpo.3-amd64 (host01.ny1) 05/26/18 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.31 0.00 0.66 0.09 0.00 98.94 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 1.78 0.24 6.80 98.41 432.03 150.76 0.60 85.97 114.91 84.95 2.13 1.50
Sunday, April 29. 2018
Notes on Resiliency - VRRP, AnyCast
This is another collection of random notes, this time, on how to build something on Linux somewhat resembling Cisco's Global Load Balancing capability, basically a continuation of my entry at Linux ifupdown2 VRRP.
Traditionally, one sets up VRRP using keepalived or the simpler vrrpd. This configuration is typically used when setting up (typically) two routers in an active/passive setup to act as a gateway for a network subnet. In essence, the two (or more) routers negotiate who will hold the gateway mac and ip address.
In other circumstances, it might be desired (and possible) to run active/active. This is a possibility when running containers on a host, and there are similar services running across the hosts. In this instance the same address can be assigned as a secondary address across multiple containers to load balance traffic.
And in even other cases, subnets may be stretched in a layer2 over layer3 encapsulated network across multiple hosts. And in this case, each host should be able to act as a gateway for the traffic local to it. It is this last example I am currently investigating.
Reynold's Blog has an entry called Configuring Cumulus Linux High Availability Layer 2 Network. The most interesting aspect of this post is reference to using 'address-virtual' commands when using ifupdown2 style /etc/network/interface structures:
address-virtual 00:00:5e:00:01:02 10.11.2.254/24
The ip and mac addresses are identical across interfaces sharing the gateway role. The mac address is a reserved range 00:00:5e:00:01:00 – 00:00:5e:00:01:ff for VRRP style operations. The ip address is the virtual ip address (VIP). This style of usage is explained more in Virtual Router Redundancy - VRR.
Or maybe I don't worry about this as Ethernet Virtual Private Network - EVPN has a section with asymmetric routing and symmetric routing which do not need vrrp style constructs.
Layer 3 routing on Cumulus Linux MLAG talks about VRR, the address-virtual, and FRR/route-map to obtain ECMP based load balancing. Now the question - how to get things to not need MLAG.
Thursday, April 19. 2018
nftables: connection tracker helpers
From a mailing list entry:
Question:
using nft from nftables, I created some IP filter rules inside a partially virtualized (Linux Vserver, www.linux-vserver.org) machine. Almost all rules are working as desired, but rules that need connection tracking helpers, like ftp and tftp, do not . some ip packets are blocked though they should be allowed. As the same tftp rules - I am sure that I made no mistake - work on a real host, there is probably some requirement for these helpers to work correctly and that is not fulfilled inside a Vserver.
Answer:
In recent kernels no default assignments of helpers is done anymore, iptables users need to use -j CT target, nft users need to add a helper object:
nft add ct helper inet filter bar '{ type "ftp" protocol tcp; }' nft add rule inet filter output tcp dport 21 ct helper set "bar"The assignnment needs to be done in the direction that creates the connections that need the helper.
So for a local host (connecting to remote server), this needs to be output; for a server (expecting ftp connections), input.
For a gateway it can be in forward, or prerouting and output in case its needed everywhere (local and forwarded).
Also it makes sense to limit helper assignemnt to connections that need it (e.g. ip saddr 192.168/16 or somesuch).
With a later addendum:
As I do not have the required nftables and kernel versions, I reactivated default assignment withecho 1 > /proc/sys/net/netfilter/nf_conntrack_helperlike described at the bottom of connection tracking meta-information
Monday, April 16. 2018
Bash Links
Sunday, April 1. 2018
HP DL360 G6 P410i with Debian Stretch
I have a couple old HP DL360 G6 servers running a several year old version of Debian Stretch. It became time to update them. Rather than fooling around with an upgrade, this is a re-install scenario. The servers have ILO2, with very lame keystroke ability. I had to use a combination of the ILO2 Remote Access in Internet Explorer as the viewer with an ssh login on another window to get keystrokes and menu operations going. I think it was a lot easier back with original Internet Explorer and Java. The Remote Mount of a CD also worked after a couple attempts at getting everything correct.
With the operating system installed, I used the Linux HP Smart Array Raid Controller article to determine the mechanism for installing HP proprietary tools. HPE (Hewlett Packard Enterprise) has a Software Delivery Repository. I am glad I hit the guy's article first, as I would not have known what to do without it.
Here is the server type determined via DMI:
dmidecode | grep -A3 '^System Information' System Information Manufacturer: HP Product Name: ProLiant DL360 G6 Version: Not Specified
PCI information shows the RAID Controller type:
# lspci -k|grep -i -A2 raid 03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01) Subsystem: Hewlett-Packard Company Smart Array P410i Kernel driver in use: hpsa
Which confirms kernel modules in use for my SAS drives:
# lsmod |grep hpsa hpsa 102400 2 scsi_transport_sas 45056 1 hpsa scsi_mod 253952 8 sd_mod,usb_storage,scsi_transport_sas,libata,hpsa,uas,sr_mod,sg
Since I am using Debian Stretch, here are some revised commands to get at some tools. This is the closest I could get to a set of descriptions.
# apt install curl # curl https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add - # echo -e "deb http://downloads.linux.hpe.com/SDR/repo/mcp/ stretch/current non-free" \ > /etc/apt/sources.list.d/hpe.list # apt update # apt install ssacli # apt install ssaducli # apt install hponcfg
ssacli provides access to the raid controller information. ssaducli is supposed to show wear rates, but it generated an empty report for me. And hponcfg deals with ILO management. More articles are needed for decoding that beast. Continue reading "HP DL360 G6 P410i with Debian Stretch" »
Debian Links I Lose
I have a couple older HP servers I acquired prior to understanding free firmware vs non-free. In the Debian world, when there are non-free network drivers, this can be an issue, as the non-free drivers are not included with the standard distribution downloads. Case in point, the bnx2 firmware. There are a ways to build your own distribution, but Debian, via an out of the way location, does provide an installable image: Unofficial non-free images including firmware packages
When trying out Debian Testing, here is the Debian Testing Installer page.