I run check_mk for monitoring some servers. Currently, the check_mk host uses ssh connections to acquire the data from the check_mk monitored host.
Journey into the SaltMine to keep nagios fed with check_mk shows some ways of not using ssh, but to use minion/master interactions to capture the data. I am leaning towards revisiting this by using SaltStack's inotify beacon to signal captured file changes, which then trigger events and orchestration to transfer the data from the minion/host to the check_mk/monitor. And I think it can be done in a way such that the salt master doesn't necessarily need to reside on the check_mk monitor. [as a note, the article shows some file locking mechanisms which might come in handy when I try to tackle this].
But, first, I wanted to prove the theory in a different scenario. This example uses three hosts:
- monitored host, which is running the salt-minion, and on to which the check_mk monitoring agent is to be installed,
- monitoring host, which is check_mk, and also has a salt-minion installed, and the
- salt-master, which controls the state and interactions between hosts
The monitoring host will use ssh to connect to the monitored host and access the agent. During the first ssh session, a manual intervention is typically required to confirm usage of the destination's public host key, which then goes into the ~/.ssh/known_hosts file. '-o StrictHostKeyChecking=no' could be used as a simple work-around, but is not very security conscious. Instead, I came up with a series of SaltStack events and states to get the monitored host's public key into the monitoring host's known_hosts file.
There are a number of key sets in use:
- When check_mk connects to an agent via ssh, it will typically use a local private key, and will require a shared public key in the monitored host's ~/.ssh/authorized_keys file. I use SaltStack states and pillars to distribute and install the public key, and make use of the "command="/usr/bin/check_mk_agent" option in the authorized_keys file
- Each host has a unique public/private key. SSH uses this to prevent man in the middle attacks, and to ensure the host hasn't changed. This blog entry is about getting this monitored host's public key into the monitoring host's known_hosts file.
In etc/check_mk/main.mk, there is typically an entry like the following (I use SaltStack Jinja templates to fill in the variables). I also use SaltStack states to distribute the ssh agent public keys. The 'local' entry allows check_mk to monitor itself.
datasource_programs = [
( '/usr/bin/check_mk_agent', ['local'], ALL_HOSTS ),
( 'ssh -i /omd/sites/{{ cmk.sitename }}/.ssh/check_mk.prv -l root <IP>', ['ssh'], ALL_HOSTS ),
]
I use a SaltStack pydsl formatted state file to install the agent as a package, and then send an event with the host's public key. This event will then trigger another event to perform the discovery.
#!pydsl
#
# file: salt/omd/agent.sls
# author: Raymond Burkholder
#
# agents are found in
# /opt/omd/versions/default/share/check_mk/agents/
cmk = __pillar__['check_mk']
# values to determine the package
agent = cmk['agent']
linux = agent['linux']
stateInstallAgent = state( 'check_mk install agent' )
stateInstallAgent.pkg.installed(
sources = [
{ 'check-mk-agent': 'salt://_packages/debian/' + linux + '.deb' }
]
)
# prepare to install check_mk's public key
ssh = agent['ssh']
public_key = ssh['public_key']
stateSshAuthPresent = state( 'ssh auth check mk' )
stateSshAuthPresent.ssh_auth.present (
public_key['name'],
user = 'root',
enc = public_key['enc'],
comment = public_key['comment'],
options = ['command="/usr/bin/check_mk_agent"',]
).require( stateInstallAgent.pkg )
# now pickup the local host key
host_keys = __salt__['ssh.host_keys']()
host_key = host_keys[ 'ecdsa.pub' ]
key_type = ''
# send it in an event, and trigger an additional event when this one is done
stateEventRegisterKnownHost = state( 'event register known host' )
stateEventRegisterKnownHost.event.send(
'cmk/register/known_host',
data = {
'host_key': host_key,
'key_type': key_type,
'master': agent['master'],
'sitename': cmk['sitename'],
'events': {
'cmk/execute/inventory': {
'master': agent['master'],
'sitename': cmk['sitename'],
'minion': __grains__['id']
}
}
}
).require( stateSshAuthPresent.ssh_auth )
The reactor /etc/salt/master.d/reactor.conf configuration file has entries for these two events:
- cmk/register/known_host':
- /srv/reactor/event/cmk/register_known_host.sls
- 'cmk/execute/inventory':
/srv/reactor/event/cmk/execute_inventory.sls
SaltStack documentation recommends that as little time as possible is spent in the reactor subsystem, so the register_known_hosts.sls state simply passes the event on to the orchestrator:
# cat /srv/reactor/event/cmk/register_known_host.sls
#
# file: reactor/event/cmk/register_known_host.sls
# author: Raymond Burkholder
#
# fired by minion when cmk agent installed to
# trigger retrieval of minion's host key to be
# applied to cmk's known_hosts file
#
orchestrate_register_known_host:
runner.state.orchestrate:
- mods: orchestrate.cmk.register_known_host
- kwarg:
pillar:
event_tag: {{ tag }}
event_data: {{ data | json() }}
The orchestration file is another pydsl state file, which is executed on the salt-master. Pillar data is therefore very limited, and all necessary information elements need to come from the event:
# cat salt/orchestrate/cmk/register_known_host.sls
#!pydsl
#
# author: Raymond Burkholder
#
# fired by minion when cmk agent installed to
# trigger retrieval of minion's host key to be
# applied to cmk's known_hosts file
#
event_tag = __pillar__['event_tag']
event_data = __pillar__['event_data']
data = event_data[ 'data' ]
id = event_data[ 'id' ]
master = data[ 'master' ] #check_mk monitoring host
sitename = data[ 'sitename' ] #omd/check_mk site name
host_key = data[ 'host_key' ] #the monitored host's host key from the event
key_type = data[ 'key_type' ]
# this uses the message bus to execute the state on the minion,
# ie, the check_mk monitoring server
stateSetHostKey = state( 'set_known_hosts' )
stateSetHostKey.salt.state(
'set_known_host/ip',
tgt = master,
tgt_type = 'glob',
saltenv = 'base',
sls = 'omd.assign_known_host',
pillar = {
'hostname': id,
'user': sitename,
'key': host_key,
'enc': key_type
}
)
# then we can trigger the follow on discovery process on the check_mk server
# without having to worry about the known_hosts prompt
events = data[ 'events' ]
for event, values in events.items():
stateEvent = state( 'event_' + event )
stateEvent.salt.function(
name = 'event.send',
tgt = master,
tgt_type = 'glob',
arg = [ event ],
kwarg = {
'data': values
}
)
This is the omd.assign_known_host state, which is another pydsl state file to facilitate calling salt modules:
# cat salt/omd/assign_known_host.sls
#!pydsl
#
# author: Raymond Burkholder
#
# needs to be run on the OMD host, as it has the dig utility
# assigns known_host record by hostname and by ip address
#
enc_ = __pillar__['enc']
key_ = __pillar__['key']
user_ = __pillar__['user']
hostname_ = __pillar__['hostname']
# the minion id, which is a name, came in the event, but check_mk
# likes to use an ip address. So we need to lookup the address(es)
# which are assigned. known_host entries will be added for the name and
# for the addresses
ipv4_list = __salt__['dig.A']( hostname_ )
assert len(ipv4_list) > 0
# assign the hostname to the known_hosts file
stateAssignKnownHostByName = state( 'assign known host by name' )
stateAssignKnownHostByName.ssh_known_hosts.present(
hostname_,
user = user_,
key = key_,
enc = enc_
)
# and for each address, assign an entry
for item in ipv4_list:
stateAssignKnownHostByIpv4 = state( 'assign known host by ipv4 ' + item )
stateAssignKnownHostByIpv4.ssh_known_hosts.present(
item,
user = user_,
key = key_,
enc = enc_
)
The discovery process can now proceed with out intervention of confirming host keys.
I am thinking a future revision would be to use this event system to pass the check_mk agent data across the message bus from monitored server to monitoring server.
As a note to self: I want to revisit another entry: Virtually secure with openvpn, pillars, and salt, which might have some interesting ideas on automating/orchestrating tunnels and such.