I run check_mk for monitoring some servers. Currently, the check_mk host uses ssh connections to acquire the data from the check_mk monitored host.
Journey into the SaltMine to keep nagios fed with check_mk shows some ways of not using ssh, but to use minion/master interactions to capture the data. I am leaning towards revisiting this by using SaltStack's inotify beacon to signal captured file changes, which then trigger events and orchestration to transfer the data from the minion/host to the check_mk/monitor. And I think it can be done in a way such that the salt master doesn't necessarily need to reside on the check_mk monitor. [as a note, the article shows some file locking mechanisms which might come in handy when I try to tackle this].
But, first, I wanted to prove the theory in a different scenario. This example uses three hosts:
- monitored host, which is running the salt-minion, and on to which the check_mk monitoring agent is to be installed,
- monitoring host, which is check_mk, and also has a salt-minion installed, and the
- salt-master, which controls the state and interactions between hosts
The monitoring host will use ssh to connect to the monitored host and access the agent. During the first ssh session, a manual intervention is typically required to confirm usage of the destination's public host key, which then goes into the ~/.ssh/known_hosts file. '-o StrictHostKeyChecking=no' could be used as a simple work-around, but is not very security conscious. Instead, I came up with a series of SaltStack events and states to get the monitored host's public key into the monitoring host's known_hosts file.
There are a number of key sets in use:
- When check_mk connects to an agent via ssh, it will typically use a local private key, and will require a shared public key in the monitored host's ~/.ssh/authorized_keys file. I use SaltStack states and pillars to distribute and install the public key, and make use of the "command="/usr/bin/check_mk_agent" option in the authorized_keys file
- Each host has a unique public/private key. SSH uses this to prevent man in the middle attacks, and to ensure the host hasn't changed. This blog entry is about getting this monitored host's public key into the monitoring host's known_hosts file.