Some organizations will run an internal network monitoring solution to detect significant network outages. This solution typically uses a cellular modem as a sort of out-of-band (OOB) alerting mechanism.
Rather than using that method for a particular requirement I had, I developed a bash script called path-loss on github which can be run in a VPS external to the network. In fact, I run one copy in an Amazon instance, and another copy in a Google instance. The scripts can thus check each other as well as the external network nodes under test.
Each node has a series of pings sent to it. If there is a loss of pings greater than some count, as defined in a script variable, then an mtr based traceroute will be generated for that node.
If any nodes show a loss of pings and have had an mtr generated during a measurement interval (based upon a crontab entry), then a log of all nodes and their results will be sent to a Slack channel of choice. A Slack bot token-id and channel name need to be supplied in script variables.
When all pings have returned to normal, another report to Slack is generated.
A 'member' indicator is used with select nodes, such that if all members are unreachable, then a PagerDuty alert is generated. There should be at least one non-member node. This allows another test: if all nodes (members and non-members) are unreachable, that typically means that there is a local network issue, so the event is logged to syslog and no Slack or PagerDuty events are generated.
