Raymond P. Burkholder - Things I Do - Network Monitoring

Essential Network Performance Metrics That Predict Problems

nospam@example.com (Raymond P. Burkholder) — Tue, 11 Nov 2025 02:29:46 +0000

The difference between reactive and proactive monitoring comes down to tracking the right network performance metrics. Important metrics that consistently predict issues before they impact users include:

Round-trip time variations: Subtle increases in RTT often precede more serious network congestion. Set baseline thresholds for different times of day and get alerted when patterns deviate
Buffer utilization patterns: Most admins ignore this until it's too late. Monitoring buffer usage across key network devices reveals impending bottlenecks before they trigger packet loss
Error rates on interfaces: Even small increases in CRC errors or interface discards can signal hardware issues days before actual failures occur
TCP retransmission rates: This metric reveals application performance issues that users might experience from data transmission rates before they report them
Network traffic symmetry: Unusual asymmetric traffic patterns often indicate security issues or misconfigured applications

Reference: Stop Reactive Network Troubleshooting: Monitor These 5 Metrics to Prevent Downtime

Cloud environments introduce unique challenges for network performance monitoring. Some standard metrics like latency and packet loss remain important, but you'll also want to focus on metrics specific to your cloud connectivity. For hybrid environments, these are good metrics:

Inter-region latency: Especially important for globally distributed applications
Connection establishment time: Often overlooked but crucial for microservice architectures
Throughput consistency: More important than raw bandwidth in many cloud scenarios
DNS resolution time: Can be a hidden bottleneck in cloud environments

Netflow with NFDUMP, NFSEN and Open vSwitch and What it Sees

nospam@example.com (Raymond P. Burkholder) — Mon, 28 Dec 2020 00:10:14 +0000

Since my NFDump/DFSen Upgrade post many moons ago, I am once again doing an install for analyzing Netflow, this time on a server running Open vSwitch (OVS).

NFDump is a netflow collector. NFSen analyzes the netflow entries captured by NFDump.

Installing NFDump is easy using Debian's package:

apt install nfdump
systemctl stop nfdump
systemctl disable nfdump

The last two lines stop and disable the default service. Service control is provided with NFSen.

From a visualization perspective, there are a number of possibilities:

NFSen Documentation is the main starting page for the code at SourceForge.
NFSen on Github is p-alik's unofficial updated version of the original repository - my preferred location as it has some primary bug fixes and stays with the times
mbolli / nfsen-ng - a modernized version of NFSen but doesn't appear to have the flexible query capability
robcowart / elastiflow - I saw a link to this from Netflow collection and visualization with Elastiflow - which may have not relationship to NFDump / NFSen but seems promising none-the-less. Installation notes are at Installing Elastiflow flow monitoring solution.

From an NFSen perspective, here are the steps I used on a Debian Bullseye installation:

apt install wget less vim
apt install php rrdtool  librrd-dev librrds-perl php-rrd
apt install libmailtools-perl
apt install libsocket6-perl
apt install git
cd /usr/src
git clone https://github.com/p-alik/nfsen.git
cd nfsen
cp etc/nfsen-dist.conf etc/nfsen.conf
vim etc/nfsen.conf
./install.pl ./etc/nfsen.conf
/usr/local/bin/nfsen start
diff nfsen-dist.conf nfsen.conf

These are the changes to the nfsen.conf file:
Continue reading "Netflow with NFDUMP, NFSEN and Open vSwitch and What it Sees"

DNS Benchmarking

nospam@example.com (Raymond P. Burkholder) — Thu, 27 Sep 2018 05:19:41 +0000

Some ideas from a mailing list:

quick and dirty:
jens@screen:~$ dig nanog.org @8.8.8.8 | grep "Query time"
;; Query time: 16 msec
jens@screen:~$ dig nanog.org @1.1.1.1 | grep "Query time"
;; Query time: 3 msec
Indeed. For instance, the delay depends wether the cache it hot or cold (measuring response time for an authoritative server is easier).

also could use ripe atlas ...
Which embeds clients for ICMP Echo, DNS, NTP, TLS, arbitrary TCP (with some hacks), and, with serious limitations, HTTP.

use dig -u to get microsecond resolution, e.g.
$ dig -u @131.111.8.42 nanog.org | grep time:
;; Query time: 611 usec

I recommend that eyeball networks don't run any external recursive server for optimal CDN performance. Yes, some CDNs support other methods, but not all. If not all do, then the requirement remains. See On Firefox Moving DNS to a Third Party

freeware GRC tool for benchmarking DNS performance

Check_MK with SmartMonTools

nospam@example.com (Raymond P. Burkholder) — Tue, 17 Oct 2017 00:40:52 +0000

A Check_MK plugin which is supplied as part of the standard install but not 'turned on' is the 'smart' agent. It is a plugin which resides on the target with hard-drives to be monitored. Temperatures and other useful statistics can be collected.

The plugin can be found in the share/check_mk/agents/plugins/ directory amongst a number of other useful plugins.

It needs to be copied to the /usr/lib/check_mk_agent/plugins/ directory on the target to be monitored.

In order to be useful, the 'smartmontools' package needs to be installed as a pre-requisite.

On one of my systems, I get output some output similar to:

SMART WDC_WD1003FZEX-00K3CA0_WD-WCC6Y3PN9XKC Stats
  OK - Powered on: 3529 hours, Power cycles: 51, Reallocated sectors: 0, 
  Reallocated events: 0 (was 0 during discovery: normalized value looks OK), 
  Spin retries: 0, Pending sectors: 0, UDMA CRC errors: 0
Temperature SMART WDC_WD1003FZEX-00K3CA0_WD-WCC6Y3PN9XKC
  OK - 30 °C

Values are charted as well.

lm-sensors notes

nospam@example.com (Raymond P. Burkholder) — Tue, 17 Oct 2017 00:26:10 +0000

To get at various hardware sensors like temperatures and voltages, it is typically necessary to install the 'lm-sensors' package. Once the package is installed, sensors can be auto-detected with:

sensors-detect --auto

A variation of that is:

yes | sensors-detect

At the end of the output, this will probably generate an output similar to:

To load everything that is needed, add this to /etc/modules:
#----cut here----
# Chip drivers
coretemp
#----cut here----

'modprobe coretemp' can be used to obtain instant gratification by loading the module interactively, instead of at the next reboot.

On a simple system, some values can be seen:

# sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)
temp2:        +29.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +50.0°C  (high = +105.0°C, crit = +105.0°C)
Core 0:        +49.0°C  (high = +105.0°C, crit = +105.0°C)
Core 1:        +48.0°C  (high = +105.0°C, crit = +105.0°C)

For obtaining an output suitable for scraping by a check_mk agent, a different format can be emitted:

# sensors -u
acpitz-virtual-0
Adapter: Virtual device
temp1:
  temp1_input: 27.800
  temp1_crit: 105.000
temp2:
  temp2_input: 29.800
  temp2_crit: 105.000

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:
  temp1_input: 49.000
  temp1_max: 105.000
  temp1_crit: 105.000
  temp1_crit_alarm: 0.000
Core 0:
  temp2_input: 48.000
  temp2_max: 105.000
  temp2_crit: 105.000
  temp2_crit_alarm: 0.000
Core 1:
  temp3_input: 48.000
  temp3_max: 105.000
  temp3_crit: 105.000
  temp3_crit_alarm: 0.000

While writing this article, a few interesting web sites were encountered:

Arch Linux' Sensors Page
Plugins for Check_MK
Check_MK plugin: lmsensors: a place to which I need to return and try installing his plugin for more information display in Check_MK.
Earlier Article I wrote doing something similar but different.

I use an app called 'psensor' for watching system temperatures in a chart form.

Installing OMD/Check_MK into a Debian Stretch LXC Container

nospam@example.com (Raymond P. Burkholder) — Tue, 17 Oct 2017 00:01:06 +0000

OMD is a collection of monitoring tools which include the Check_MK monitoring application. Check_MK is a series of Python scripts and agents designed to simplify the use of a Nagios based scanning engine.

I have a small server running Debian Buster. I have used LXC to build a container. I used the following command to build a container:

lxc-create -n omd  -f /var/lib/lxc/persist/omd/config -t debian -B btrfs -- -r stretch

I have a BTRFS filesystem mounted on /var/lib/lxc. the -B btrfs creates a BTRFS subvolume for the container, which allows snapshots and the like.

The config file looks something like:

# cat /var/lib/lxc/persist/omd/config 
# 20171015
# lxc-create -n omd  -f /var/lib/lxc/persist/omd/config -t debian -B btrfs -- -r stretch
lxc.utsname = omd
lxc.start.auto = 1
lxc.start.order = 5
lxc.start.delay = 1

lxc.network.type = veth
lxc.network.name = eth50
lxc.network.veth.pair = veth-omd-v50
lxc.network.hwaddr = 0a:00:40:50:00:xx
lxc.network.ipv4 = 10.0.10.201/24
lxc.network.ipv4.gateway = 10.0.10.254
lxc.network.script.up = /etc/lxc/ovsup.sh
lxc.network.script.down = /etc/lxc/ovsdn.sh
lxc.network.flags = up

I am using Open vSwitch to provide bridging capabilities to the container and its environs. This particular container, using the lxc.start.auto value, will auto-start when the host restarts.

Yesterday, I attempted to install the Buster version of OMD, but some package dependencies were stale (repository) , and since I didn't have time to troubleshoot that, I went with the stable Stretch version:

apt install nano bash-completion gnupg2 wget
wget --no-check-certificate https://labs.consol.de/repo/stable/RPM-GPG-KEY
apt-key add RPM-GPG-KEY
echo "deb http://labs.consol.de/repo/stable/debian stretch main" > /etc/apt/sources.list.d/labs-consol-stable.list
apt update
apt install omd

It is as easy as that.

To configure a monitoring instance:

omd setup
omd create site01
omd config site01
omd start site01
su - site01
nano etc/check_mk/main.mk

During the 'omd create site01' step, I select the Nagios 3 engine, and set the Check_MK interface as default.

The next experiment is to install the Raw Version and see how that differs.

Install Testing Version of Check_MK/OMD from Packages

nospam@example.com (Raymond P. Burkholder) — Mon, 24 Oct 2016 14:12:00 +0000

Nagios has been wrapped by check_mk (a python based pre-processor), which in turn, has been wrapped by OMD (Open Monitoring Distribution) to provide charting capabilities. The OMD web site installs an older version. This documentation shows how to install a slighter more recent testing version.

Modify /etc/apt/sources.list with (order may be important):

deb http://ftp.debian.org/debian/ stretch main
deb http://labs.consol.de/repo/testing/debian/ jessie main
deb http://labs.consol.de/repo/testing/debian/ stretch main

Will need to add the repository key:

wget --no-check-certificate https://labs.consol.de/repo/stable/RPM-GPG-KEY
apt-key add RPM-GPG-KEY

Update, search for, and install the package:

$ sudo apt-cache search omd
omd-1.31.20160718 - Open Source Monitoring Distribution, containing Nagios,
omd-daily - Open Source Monitoring Distribution
omd-labs-edition-daily - Open Source Monitoring Distribution
omd-2.11.20161019-labs-edition - OMD Labs Edition - Open Source Monitoring Distribution (Labs Edition),
$ sudo apt-get install omd-labs-edition-daily

Then a monitoring instance can be created:

sudo omd setup
sudo omd create site01
sudo omd config site01
sudo omd start site01
sudo su - site01

An agent can be installed to monitor the local machine:

ls -alt versions/default/share/check_mk/agents
sudo cp versions/default/share/check_mk/agents/check_mk_agent.linux /usr/local/bin
sudo cp versions/default/share/check_mk/agents/waitmax /usr/local/bin

Examine the output of the agent:

$ check_mk_agent.linux
< < < check_mk > > >
Version: 1.2.6p12
AgentOS: linux
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
< < < df > > >
udev           devtmpfs     1022472       0   1022472       0% /dev
... followed by lots more stuff ....

Login to the gui 'http://hostname/site01/check_mk/login.py' to add contacts and contact groups. Then update etc/check_mk/main.mk to have something similar to:

# Put your host names here
# all_hosts = [ 'localhost' ]
all_hosts = [
  'hostname|local|site01-checkmk|notifyNormal',
  ]

ipaddresses = {
  'hostname': '127.0.0.1',
  }

define_hostgroups = {
  'site01-checkmk': 'site01 - check_mk',
  }

host_groups = [
  ( 'site01-checkmk', ['site01-checkmk'], ALL_HOSTS ),
  ]

host_contactgroups = [
  ( 'email', ['notifyNormal'], ALL_HOSTS ),
  ]

datasource_programs = [
  ( '/usr/local/bin/check_mk_agent.linux', ['local'], ALL_HOSTS ),
  ]

Then inventory the host and reload the service, and monitoring should start:

cmk -II hostname
cmk -O

An over view of plugins available in check_mk: Catalog of check plugins.

Check_MK and OMD: project started in 2010 by Mathias Kettner, a distribution of several monitoring cores (Nagios, Icinga, Shinken) and addons

As an addendum, OMD can be built from the repository. See Compile from Source:

#> git clone http://git.mathias-kettner.de/omd.git/
#> cd omd
#> ./configure
#> make
#> make pack
#> make setup

Note: Use 'make deb' or 'make rpm' instead of 'make pack && make setup' to build real system packages. I also note that the git repository has only v1.2.6p12 included at the time of this writing (changelog on Oct 1/2015), when on check_mk download page, they have v1.2.6p16, 1.2.8p15 available with the RAW edition. The RAW edition might be superceding the OMD edition and might be the way to go in the future. I will need to try that through the above instructions during my next iteration.

Their main repository: Mathias Kettner - GIT repositories

path-loss.sh - Bash script to Detect Network Path Quality, provide status to Slack, alarm to PagerDuty

nospam@example.com (Raymond P. Burkholder) — Wed, 28 Sep 2016 16:23:12 +0000

Some organizations will run an internal network monitoring solution to detect significant network outages. This solution typically uses a cellular modem as a sort of out-of-band (OOB) alerting mechanism.

Rather than using that method for a particular requirement I had, I developed a bash script called path-loss on github which can be run in a VPS external to the network. In fact, I run one copy in an Amazon instance, and another copy in a Google instance. The scripts can thus check each other as well as the external network nodes under test.

Each node has a series of pings sent to it. If there is a loss of pings greater than some count, as defined in a script variable, then an mtr based traceroute will be generated for that node.

If any nodes show a loss of pings and have had an mtr generated during a measurement interval (based upon a crontab entry), then a log of all nodes and their results will be sent to a Slack channel of choice. A Slack bot token-id and channel name need to be supplied in script variables.

When all pings have returned to normal, another report to Slack is generated.

A 'member' indicator is used with select nodes, such that if all members are unreachable, then a PagerDuty alert is generated. There should be at least one non-member node. This allows another test: if all nodes (members and non-members) are unreachable, that typically means that there is a local network issue, so the event is logged to syslog and no Slack or PagerDuty events are generated.

OMD / Nagios Links

nospam@example.com (Raymond P. Burkholder) — Fri, 24 Jul 2015 17:48:39 +0000

Mix of VMware and Linux Monitoring And Management Tools

nospam@example.com (Raymond P. Burkholder) — Wed, 03 Dec 2014 17:28:28 +0000

Flings: Apps and tools built by VMware engineers that are intended to be played with and explored. One fling, called I/O Analyzer, is a virtual appliance designed to run workloads to help baseline performance capabilities of a system. Another one, called OS Optimization Tool can enable or disable services in a Windows system image template. Some services may not be necessary, if and not running, will reduce the load on the host.

Brendan Gregg (Netflix) designed a chart called Linux Performance Observability Tools. There is a slide show on that page describing tools for the various modules of a Linux system. Look for the Linux Performance Tools slide show. There are others interesting slide shows as well.

Something I need to review at some point in the future: Monitorix: An Open Source, Lightweight System Monitoring Tool For Linux

RVTools is a windows .NET 2.0 application which uses the VI SDK to display information about your virtual machines and ESX hosts.

VMware esxtop: similar to Linux top, but more expressive as it shows many performance indicators for a VMware host.

Update 2017/05/12: Brendan Gregg provided an example of using perf to see if a cpu really is busy with the command:


perf stat -a -- sleep 10

and look for the "insns per cycle" line.

Update 2017/11/18: An article on how to obtain and calculate some stats: Linux’s SRE Golden Signals

OpenTSDB - Open Time Series DataBase

nospam@example.com (Raymond P. Burkholder) — Sun, 16 Nov 2014 19:48:07 +0000

Just a few quick links for monitoring systems using OpenTSDB as a back end:

ticketmaster's metrilyx: web based dashboard engine allows for analyzing, cross cutting and viewing of time series data in a simple manner.
Bosun: StackExchange's advanced, open-source monitoring and alerting system

Upgrading Nfsen and Nfdump

nospam@example.com (Raymond P. Burkholder) — Wed, 09 May 2007 15:51:13 +0000

As an update to my two articles for installing nfdump and installing nfsen, here are a few corrections and a process for upgrading.
Continue reading "Upgrading Nfsen and Nfdump"

Drraw Installation

nospam@example.com (Raymond P. Burkholder) — Wed, 02 May 2007 23:30:00 +0000

Cricket and NetFlow create RRD files. Each has some basic utilities for drawing data found within the files. Drraw provides advanced capabilities for creating graphs, templates, and dashboards.
Continue reading "Drraw Installation"

Implementing Live URL Tracking

nospam@example.com (Raymond P. Burkholder) — Thu, 15 Mar 2007 17:16:25 +0000

To monitor browsing habits of company users, and to use the information for possible site blocking or traffic limiting with MasterShaper, URLSnarf along with a Perl script can be used to log specific items to a database and place the full capture results into a log file. The log file will be rotated with logrotate.

The Perl module Logfile::Access is key. On my machine it needed to be manually installed:

perl -MCPAN -eshell
  install Logfile::Access 
  # a missing file will be encountered
  exit
cd ~/.cpan/build/Logfile-Access-1.30/Access.pm
nano Access.pm 
  # change line 38 to '/etc/mime.types'
make 
make install

Installing Netflow Tool: nfsen

nospam@example.com (Raymond P. Burkholder) — Fri, 02 Mar 2007 22:20:22 +0000

nfsen is companion tool to nfdump. Where nfdump handles the capture and writing to disk of netflow records, nfsen takes the captured files and makes the data available through a web interface.
Continue reading "Installing Netflow Tool: nfsen"