Raymond P. Burkholder - Things I Do - Salt

SaltStack on Debian Bookworm

nospam@example.com (Raymond P. Burkholder) — Wed, 19 Jul 2023 01:09:49 +0000

I found out the hard way that SaltStack and Debian no longer place nice together. I had upgraded a Debian installation from Bullseye to Bookworm, along with the resident Salt Minion. When attempting to use the minion, it no longer starts up, due to various imports no longer working. Which was due to the salt-minion not being upgraded. The error message would started this odyssey:

salt ImportError: cannot import name 'Markup' from 'jinja2'

Taking a look at the Debian Developer Information for Salt, the last version started in 'unstable' was 3004.1 back in December of 2022. This is now almost 8 months later and little or no movement. There was some mention in a ticket somewhere that Salt release cycles don't cater to Debian stable release cycles. Not sure if that is a legitimate reason or not, but, well, for whatever reason, SaltStack management in Debian is no longer a simple no brainer.

However, after a little digging, there is a way to run SaltStack versions 3006 (current as of this writing). It is simple to install on Bullseye, but not easily done on Bookworm.

On Bullseye (as root, or implies sudo):

# cd ~ # apt remove salt-minion salt-master # apt install curl # curl -L https://bootstrap.saltstack.com -o install_salt.sh # sh install_salt.sh -M onedir

The '-M' installs the salt master at the same time (for machines running master). If you forget to do that, you'll need to diagnose and fix the systemctl mask error with the following:

# apt install file # file /etc/systemd/system/salt-master.service # rm /etc/systemd/system/salt-master.service # systemctl daemon-reload # sh install_salt.sh -M onedir

The 'sh install_salt.sh -M onedir' should show a symlink to /dev/nul, which the 'rm ...' will fix.

On Bookworm, the bootstrap isn't scheduled to work till beginning of 2024 sometime I think with Salt 3007 or 3008 -- more info in [FEATURE REQUEST] Add Salt support for Debian 12 #64223 .

In the meantime, I had to cheat a bit:

in /etc/debian_version, change 12.0 to 11.0
in /etc/apt/sources.list, change bookworm to bullseye
rm /etc/apt/sources.list.d/salt.list
run apt update
run the commands listed above for installing the one or both the salt services
restore /etc/debian_version and /etc/apt/sources.list to their original content

I'm sure there are more elegant ways of doing this, but this worked to fake the needed version 11 in the installation script and directory traversal requirements

Note, more info on the Salt Install/Bootstrap Process.

SaltStack Orchestration for SSH Known_Hosts File

nospam@example.com (Raymond P. Burkholder) — Sun, 31 Dec 2017 19:19:44 +0000

I run check_mk for monitoring some servers. Currently, the check_mk host uses ssh connections to acquire the data from the check_mk monitored host.

Journey into the SaltMine to keep nagios fed with check_mk shows some ways of not using ssh, but to use minion/master interactions to capture the data. I am leaning towards revisiting this by using SaltStack's inotify beacon to signal captured file changes, which then trigger events and orchestration to transfer the data from the minion/host to the check_mk/monitor. And I think it can be done in a way such that the salt master doesn't necessarily need to reside on the check_mk monitor. [as a note, the article shows some file locking mechanisms which might come in handy when I try to tackle this].

But, first, I wanted to prove the theory in a different scenario. This example uses three hosts:

monitored host, which is running the salt-minion, and on to which the check_mk monitoring agent is to be installed,
monitoring host, which is check_mk, and also has a salt-minion installed, and the
salt-master, which controls the state and interactions between hosts

The monitoring host will use ssh to connect to the monitored host and access the agent. During the first ssh session, a manual intervention is typically required to confirm usage of the destination's public host key, which then goes into the ~/.ssh/known_hosts file. '-o StrictHostKeyChecking=no' could be used as a simple work-around, but is not very security conscious. Instead, I came up with a series of SaltStack events and states to get the monitored host's public key into the monitoring host's known_hosts file.

There are a number of key sets in use:

When check_mk connects to an agent via ssh, it will typically use a local private key, and will require a shared public key in the monitored host's ~/.ssh/authorized_keys file. I use SaltStack states and pillars to distribute and install the public key, and make use of the "command="/usr/bin/check_mk_agent" option in the authorized_keys file
Each host has a unique public/private key. SSH uses this to prevent man in the middle attacks, and to ensure the host hasn't changed. This blog entry is about getting this monitored host's public key into the monitoring host's known_hosts file.

Continue reading "SaltStack Orchestration for SSH Known_Hosts File"

SaltStack Articles

nospam@example.com (Raymond P. Burkholder) — Mon, 07 Aug 2017 17:44:04 +0000

I had thought that the Reactor engine could use pillar states when running states for minions. It may or may not have worked in the future. For the present, it is confirmed as not working: Pillars not available to reactor states #42762. This refers to another issue which supposedly resolves everything in v2017.7.1: Only force saltenv/pillarenv to be a string when not None #42433.

While discovering this issue, I have been thinking about ways to maintain temporary variables somewhere. I first looked at Thorium, but it doesn't seem to have a very flexible two-step look up system, as I need to access variables based upon minion_id, and service type.

The consul pillar state seems to be used only on the master, but I am thinking something should be available on the minion, or accessible from the minion. I thought of the sqlite state for this, but might be a bit heavy duty.

While researching this issue, I came across other interesting Salt articles, which I book mark here, for possible future reference:

stretch repository is missing packages #42715: this is interesting because in order to load 2017.7.1, I would have to obtain it from the SaltStack repository (as debian doesn't have this yet), and since I run Stretch and Buster, this might cause some issues.
SaltStack-able Microservices Architecture: Beating the challenge with SaltStack, upstart, consul, consul-template, logstash, etc. An interesting article as it introduces the concept of a jinja macro.
Combine multiple pillar files under a single key: uses the 'multiple app needs logrotate entries' problem to introduce the concept of merge, os_family_map (which has another issue in github), and jinja 'do' statements.
Salt consul pillar list of dicts #34458: not necessarily directly relevant, but does have a link to the YAML v1.2 Specifications.
Using Consul as an External Pillar Source: a simple, detailed way of using Consul for pillar based data for common data as well as minion specific private data.
Two entries here from the same blogger: Create self-managing servers with Masterless Minions and Using salt-ssh to install Salt, with a bonus reference: Getting redundancy and scalability with multiple master servers"
Managing your computers with a Salt Master and Git.
Some notes on using SaltStack: centers the discussion around roles.
Using Consul as an External Pillar Source
SaltStack orchestration with Beacons and Reactors
Thorium and Salt API chapter extract from a book called "Mastering SaltStack".
Network Automation at Scale: Up and Running in 60 Minutes: an excellent tutorial on orchestration at scale, from a RIPE 74 presentation.
Salt-based secret storage solution - many of the chapters are incomplete, but this has content.

SaltStack - Automation with Salt - Interesting Pages

nospam@example.com (Raymond P. Burkholder) — Tue, 25 Oct 2016 12:29:24 +0000

I have been using Salt for a number of infrastructure automation projects. Still a lot to learn. For me, it is an excellent 'as built' documentation tool. Not to mention the time saving and repeatability of building, rebuilding, and supporting infrastructure.

Some links I use, and want to keep around:

rest_cherrypy: A REST API for Salt
kitchen-salt: A Test Kitchen Provisioner for Salt. The provider works by generating a salt-minion config, creating pillars based on attributes in .kitchen.yml and calling salt-call. This provisioner is tested with kitchen-docker against CentOS, Ubuntu, and Debian.
Salt Community Projects: The one currently at the top of the list: HubbleStack. "... on-demand profile-based auditing, real-time security event notifications, automated remediation, alerting and reporting"