I've been able to get my head around what Kafka by looking at it obliquely.
Mark Fletcher's blog entry is more PostgreSQL oriented but offers up some interesting ideas on sequencing messages between sub-systems.
5 tips for architecting fast data applications introduces some of the mechanisms large web sites handle the quantity of requests and responses.
2018/05/18 But it seems that Kafka can be difficult and complicated to setup and configure. It does have the redundancy and flexibility to undertake many difficult message passing scenarios. But is there something a bit easier to start with?
Maybe MQTT has an opportunity. LWN has An introduction to MQTT. MQTT is used extensively in the IOT world, which includes tooling supplied by Home Assistant.
I was somewhat leary of using MQTT, as there didn't appear to be robust, distributed solutions available. Well, there are:
- EMQ: "The Massively Scalable MQTT Broker for IoT and Mobile Applications" - based upon Erlang.
- VerneMQ - a MQTT publish/subscribe message broker which implements the OASIS industry standard MQTT protocol
To continue with this theme, Jan-Piet Mens has a blog article called How do your servers talk to you? where he discusses integrating some code bits he has, nagios/icinga, and mqtt to create alerts and messaging which can be processed via Pushover which is a PagerDuty alternative. Mens has another article exploring other messaging possibilities: Alerting or notifying on SSH logins.
Jonathan McDowell writing about MQTT and temperature sensors, and MQTT with automation.
2019/03/25 The Case for Database-First Pipelines talks about Kafka and Databases, in which the supposition that Kafka exists between everything is debunked. Sometimes you have to let the database be a database and issue the responses. Which, now that I've written that, simply means that the database is just another event generator, and Kafka is just the event passer, which seems pretty straightforward to me.