Raymond P. Burkholder - Things I Do - Cisco

Troubleshooting ECMP/bundling issue (5-tuple black holing)

nospam@example.com (Raymond P. Burkholder) — Fri, 10 Mar 2017 10:29:47 +0000

From the cisco-nsp mailing list.

troubleshooting routing issues on paths external to our network that lead to blackholing of specific 5-tuple combinations here, very likely due to ECMP/Bundling issues (we are link is up/up and used for load-balancing, but cannot actually transmit or receive traffic, therefor dropping those packets on the floor).

Now, this happened a few times over the years, and I am wondering if you guys have any suggestions or tools that you use in those cases, other than tcpdump'ing at both ends, generating thousands of 5-tuple combinations and then analyzing them in wireshark.

Many people on this list are probably doing this already but with hardware devices like Ixia testers. You can use the control software for them to generate different flows in a pragmatic fashion. We haven’t the budget for them I’m afraid but I think we can test all we want with these pieces of software below.
You can check out Cisco’s TRex (https://trex-tgn.cisco.com/) or MoonGen (https://trex-tgn.cisco.com/). Both are built on DPDK so you need to install that as a prerequisite. These will let you generate large numbers of flows in a scriptable fashion and record the results.
I haven’t had time (story of my life!) but ideally I want to set up a new test server in our lab an get either/both of these installed so that each time we test a new device we can generate a range of traffic/flows to test the device forwards as desired, to test load balancing and hashing, testing ACLs, QoS etc.
At the minute we have a couple of low end devices and make do with the following open source tools to generate single packets or single flows for just basic speed testing or testing that traffic drops into a specific queue, matches an ACL etc:

Generating single customer packets: http://ostinato.org/ and http://packeth.sourceforge.net/packeth/Home.html
Layer 2 Ethernet/MPLS: https://github.com/jwbensley/Etherate
Layer3/4 IP/TCP/UDP: https://github.com/esnet/iperf Layer 2/3/4: http://pktgen.readthedocs.io/en/latest/
Specifically for testing ECMP have a look at this (I haven’t had a chance to play with it personally yet): https://github.com/facebook/UdpPinger

Also, after obtaining a list of affected and unaffected 5-tuples, any particular easy way to find out how this is getting hashed, so that we could find the likely number of bundle members (this could be very useful multiple interconnection and parties are involved).

You are probably going to need to dig into vendor specifics; what vendors are in play and the configs deployed (what load balancing / hashing options/knobs have been configured), then look at the hardware documentation with regards to what is supported by the hardware, does that match what is configured? If you test it empirically does it add up? The vendor documentation should say how the load-balancing is done.
You can roughly work out the hashing mechanism by say sending a fake flow from 10.0.0.1 to 10.0.0.2, proto TCP, src port 1, dst port 1. Then just increment one field by one, dst port == 2, then dst == 3 etc. Look as the traffic moves between links. If you keep incrementing you can brute force you way through an eventually you might see the same pattern of hashing results emerge.
Also some boxes have a command to test the hashing, example from a Cisco 4500X:
#show platform software etherchannel port-channel 1 map l4-port 1.1.1.1 100 2.2.2.2 200 | i is Te Map port for l4-port 1.1.1.1:100, 2.2.2.2:200 is Te1/1/16(Po1)

Another entry from the mailing list:

I'm looking at the "paris traceroute" toolsset right now, that looks like it could be the right tool for the job: lib paris trace route: aware of the multiple paths and can report on any single one of them accurately, as well as on all of them.

Cisco Spanning Tree PortFast with Bpdufilter / Bpduguard

nospam@example.com (Raymond P. Burkholder) — Mon, 12 Dec 2016 19:42:46 +0000

In trying to look for best practices for connecting various sorts of devices to a network, there are any number of combinations of portfast, bpduguard, and bpdufilter from which to choose.

Bpduguard checks for bpdu's entering the network when none are expected: a layer three device is normally connected, but then someone swaps in a switch which may create a loop connection.

Bpdufilter prevents bpdu's from exiting the network. This is used when someone the other side is looking for bpdu's, and you don't want them to see any, because you are certain not to create a loop.

What I want is to be able to bring up a port fast, and not have to go through the spanning-tree negotiation sequence, but I also want the protection that spanning-tree can offer, should strange configurations come into play.

The following table represents some scenarios with global bpduguard, global bpdufilter, interface level bpduguard, and interface level bpdufilter, all with portfast in place. This is a simple test on two adjacent ports on a switch with a cross over cable between the two ports.

interface bpduguard	interface bpdufilter	result
Global Bpduguard/Bpdufilter Enabled
disabled	disabled	blocking
n/a	n/a	err-disable
n/a	disabled	err-disable
disabled	enabled	crash
enabled	enabled	crash
enabled	disabled	err-disable
&nb;	enabled	crash
No Global Bpduguard/Bpdufilter
interface bpduguard	interface bpdufilter	result
disabled	disabled	blocking
disabled	enabled	crash
enabled	enabled	crash
enabled	disabled	err-disable

The final combination we came up with is to have global 'spanning-tree bpduguard' with interface local 'spanning-tree portfast'.

With 'spanning-tree bpduguard' at the interface level, the interface continuously generates bpdus. With 'spanning-tree bpduguard' at the global level, and not at the interface level, is that when the interface comes up, bpdus are generated for 20 seconds, then are no more are generated. This allows the interface to be checked for loops during the up transition, and then no more checks are needed as the interface is deemed to be good layer three interface for the duration.

For these 'spanning-tree portfast' ports, I have started to apply 'spanning-tree rootguard' as a matter of course.

Eric Leahy has a good reference for BPDU Guard, BPDU Filter, Root Guard, Loop Guard & UDLD settings.

TTCP: Test TCP

nospam@example.com (Raymond P. Burkholder) — Mon, 24 Nov 2008 21:56:00 +0000

A quick and simple tool for link bandwidth testing (aka Throughput Testing) is included in many flavours of Cisco's IOS. Although it is hidden and officially unsupported, it is documented and functional.

By running 'ttcp' from the command line in privileged mode on two different routers, one can test links between the routers.

Cisco documents the tool with Document 10340, Using Test TCP (TTCP) to Test Throughput.
Continue reading "TTCP: Test TCP"

Quality of Service Options on GRE Tunnel Interfaces

nospam@example.com (Raymond P. Burkholder) — Sat, 20 Oct 2007 16:28:26 +0000

Note to self, according to Cisco's document 10106, Quality of Service Options on GRE Tunnel Interfaces: when applying queuing to a tunnel interface, shaping must also be applied. The top level policy of a recommended hierarchical policy should be a shaping command, while the lower-level policies configure the queueing mechanisms.

Cisco Device Health

nospam@example.com (Raymond P. Burkholder) — Mon, 25 Jun 2007 18:26:59 +0000

In Cisco's book, Top Down Network Design, one useful show command is 'show buffers'. Some useful SNMP statistics include:

BusyPer. CPU busy percentage in the last 5-second period.
AvgBusy1. 1-minute exponentially decayed moving average of the CPU busy percentage.
AvgBusy5. 5-minute exponentially decayed moving average of the CPU busy percentage.
LocIfInputQueueDrops. The number of packets dropped because the input queue was full.
LocIfOutputQueueDrops. The number of packets dropped because the output queue was full.
LocIfInIgnored. The number of input packets ignored by the interface.
BufferElMiss. The number of buffer element misses. (You can also check misses for small, medium, big, large, and huge buffer pools.)
BufferFail. The number of buffer allocation failures.

I've been doing most of my snmp statistics gathering on 5 minutes intervals. On some interfaces, it may be of value to step that up to 1 minute intervals. Of course, if my total collection time is over 1 minute, I may have problems with that.

From the same book, is this interesting statistic about why Window's file transfers over WAN links can go only 'so fast'. SMB acts like a ping-pong protocol. It can only send up to 32KB before requiring an acknowledgement. So if the delay is 50 ms end to end, and ignoring client and server delays, a client can receive at most 32 KB every 100 milliseconds, or 320 KB per second. This means that the maximum throughput is 2.56 Mbps, at best.

Implementing a Wired 802.1X Network With Cisco and Microsoft

nospam@example.com (Raymond P. Burkholder) — Mon, 30 Apr 2007 19:38:16 +0000

Some companies will set up guest networks utilizing a parallel network configuration: separate switches or hubs, along with separate DSL/router internet connections. They will then designate certain ports in a conference room to be 'corporate connections' and certain ports to be 'guest network'. They then leave it up to the user to 'pick a port'.

This mechanism does indeed provide a separate path to the internet, but obviously, the weakness is an inability to prevent people from using inappropriate ports.

A better from of enforcement is provided through the implementation of an authentication/authorization protocol called 802.1X. This protocol works with wired as well as wireless networks. Various methods of operation are available. The simplest to to either enable or disable a switch port based upon receipt of appropriate credentials from the supplicant, which is the computer/user being connected to the network.
Continue reading "Implementing a Wired 802.1X Network With Cisco and Microsoft"

Running the Same Commands on Multiple Cisco Devices

nospam@example.com (Raymond P. Burkholder) — Fri, 20 Apr 2007 23:30:00 +0000

Older switches don't have the nifty 'interface range' command for applying the same configuration to multiple interfaces at the same time. On older style 48 port switches, it can be a boring task to update all interfaces with, say, a new vlan assignment.

I was thinking of doing an Expect script to perform the task. I tackled it once upon a time, and did come up with a working example, but it took a while to get used to the nuances of the Expect language.

Having more skills in Perl, and realizing that there is a CPAN add-on for Cisco devices, I recently did something up in Perl. The example below connects to a switch, and for each of 48 interfaces, it defaults it and then applies a new vlan.

By creating an array of devices, and if they have common usernames and passwords, or are authenticated through TACACS, the same commands could be applied to a range of devices in one easy run.

use strict;

use Net::Telnet::Cisco;

my $postDevice = 'bmsw08';
my $postCommand = 'sho inter status';

  my $session = Net::Telnet::Cisco->new(
    Host => $postDevice
    );
#  $session->login( '', 'password' );
  $session->login( 'username', 'password' );
  $session->enable('enable');

  my @output;
#  my @output = $session->cmd(String => $postCommand );
#  print @output;

   print $session->cmd( String => 'config t' );

   for ( my $i = 1; $i <= 48; $i++ ) {
     print $session->cmd( String => "inter f0/$i" );
     print $session->cmd( String => "default desc" );
     print $session->cmd( String => "swi acc vlan 103" );
   }

  $session->close();

There are two types of logins, one with a username and password, and one with just a password. An Enable is used in either case.

Cisco's Embedded Event Manager

nospam@example.com (Raymond P. Burkholder) — Thu, 05 Apr 2007 03:05:54 +0000

For a long time, certain of Cisco's routers had the ability to run tcl scripts, particularily routers with voip loads, in order to perform credit card billings for voip connection validations. To me, the tcl scripting was a black art and not well documented.

That has changed. The capability has now become officially and publically supported. It is now known as the Cisco Embedded Event Manager. Part of it is integrated into SLA functionality. This, for example, provides the ability to adjust routing and other configuration details on the fly depending upon reachability testing. That one example barely covers the surface of what can be done to dynamically and automatically alter a router's functioning parameters.

That, in itself, is a two edged sword. But with today's interconnected, converged, flexible networks, sometimes one needs a trick like that to make things happen.

Some URL's:

New Cisco Voip Wiki

nospam@example.com (Raymond P. Burkholder) — Sun, 18 Mar 2007 19:28:35 +0000

cisco-voip.org is community based self-help Wiki for users, buyers, installers, and troubleshooters of Cisco Voice products.

Quick and Simple MultiCast

nospam@example.com (Raymond P. Burkholder) — Thu, 25 Jan 2007 22:51:57 +0000

I worked with a customer who has a network with a server segment, a workstation segment, and a segment used for mass upgrades using Symantec's Ghost in MultiCast mode.

If multicasting is not turned on in a network, multicast packets will simply flood the whole network because the switches and routers have not been told how to deal with the packets, and they are forwarded by default.
Continue reading "Quick and Simple MultiCast"

Cisco References vol1

nospam@example.com (Raymond P. Burkholder) — Wed, 15 Nov 2006 01:12:29 +0000

Cisco has a number of reference documents that are very useful in day to day network consulting, but can be a real bear to find in a pinch.

The first is Cisco Unified Callmanager 4.1 TCP and UDP Port Usage. It goes through and identifies all the TCP and UDP ports in use by the various Callmanager services. Ports are grouped into the following categories:

Intracluster Ports Between CallManagers
Windows and Common Ports
Between CallManager and LDAP Directory
Web Requests from CCMAdmin or CCMUser to CallManager
Signalling, Media and Other Communications Between Phones and Callmanager
PC Behind the Phone to the Phone
Signalling, Media and Other Communications Between Gateways and Callmanager
Communications Between Applications and CallManager

The end of the document contains a number of links regarding PIX and IOS FW Inspection and Context Based Access Control.

When trying to select a Cisco device 'based on the numbers', you'll want to take a look at Cisco's Portable Product Sheets. The key sheets have to do with performance of the various switch models and router models. In addition, there are sheets relating to Port Adaptors, NM/WIC/VWIC compatibility, VPN performance, wireless comparisons, some info on GBICS, as some stuff on phones and voice density.

Cisco's TAC Tools page has such things as a DSP Calculator, IP Subnet Calculator, and a Voice Codec Bandwidth Calculator.

Configuring Wireless on Cisco 871W with SDM

nospam@example.com (Raymond P. Burkholder) — Mon, 06 Nov 2006 00:30:00 +0000

This was not a pleasant experience. No wonder I like CLI over GUI interfaces. With a command line, I get feedback as to what I'm doing wrong. With Cisco's SDM, it has some rudimentary user-interface checks before performing a post. On the other end, if you haven't put something into the UI correctly, at least for the wireless config pages, you don't even get a 'hey dummy' message, it just plain ignores you. No feedback, no hints, no nothing.

I simply wanted to get my wireless bit bridged to the vlan bit. The first step is to select the setting in SDM GUI to bridge the wireless with the wired. This will create the normal BVI interface. No problem there. After that, you go into the wireless configuration web pages. I simply wanted to get a WPA-PSK (Pre-Shared Key) into the unit. There is no obvious way to accomplish that task.
Continue reading "Configuring Wireless on Cisco 871W with SDM"