Raymond P. Burkholder - Things I Do

I picked up a zigbee controller and was going to attach it to me BeagleBoard. Zigbee2MQTT. Recent installs of Zigbee2MQTT require recent versions of node.js, which the latest BeagleBoard operating system can't quite yet supply.

I did have an espressobin sitting around. After reviewing it's capabilities, it looks feasible. But it needed an operating system update.

I needed to boot it up to see what flavour it is. A couple of startup notes:

• It requires more power than what the USB can provide, so even though there is a green light lit on the board when the USB cable is inserted, the power adapter must be connected as well.
• command to start a console: screen /dev/ttyUSB0 115200
• The startup text showed I had a version with 1G memory, 1000Mhz CPU, 800 Mhz memory
• The starting page with cryptic notes: Espressobin
• I first needed to reflash the boot loader. This required downloading from /dl/espressobin/u-boot with my boot file being flash-image-DDR3-1g_2cs_2-1000_800.bin and placing the file on a vfat SD
• booting up the espressobin, and starting the boot prompt, the boot flash could be installed with:
• bubt flash-image-DDR3-1g_2cs_2-1000_800.bin spi usb
• from my linux desktop, I downloaded Armbian Bookworm and imaged to an SD card with: sudo dd if=Downloads/Armbian_23.5.1_Espressobin_bookworm_current_5.15.113_minimal.img of=/dev/mmcblk0 bs=1M status=progress
• A reboot started Debian and entered in to the initial configuration process.
• a line in /etc/NetworkManager/conf.d/10-ignore-interfaces.conf needed to be commented out to be able to manage the interfaces (networking in this espressobin build may not be fully baked), and NetworkManager restarted

Espressobin Reference:

• Documentation
• Community

References

• systemd-networkd - from archlinux, with reference to systemd-nspawn for container networking. -- interesting note: "it is possible to run Docker containers inside an unprivileged systemd-nspawn container with cgroups v2 enabled". But from an old talk:
  • Creating containers with systemd-nspawn - It is targeted at "building, testing, debugging, and profiling", not at deployment. systemd-nspawn uses the same kernel APIs that the other two tools use, but is not a competitor to them because it is not targeted at running in a production environment.

Gone are the good 'ole days of using /etc/network/interfaces to manage basic networking stack configurations. There seems to be an explosion of alternate ways, each stepping on each other's toes: /etc/network/interfaces, NetworkManager, NetPlan, systemd-networking, ...

A problem I had the other day was where a new installation of an espressobin had NetworkManager installed, no NetPlan, and some stuff in /etc/network/interfaces. Unfortunately, since NetworkManager controls the dhcp-client, the /etc/network/interfaces interface was not obtaining an address.

The solution was to edit /etc/NetworkManager/conf.d/10-ignore-interfaces.conf, and comment the following line:

[keyfile] #unmanaged-devices=interface-name:eth*,interface-name:wan*,interface-name:lan*,interface-name:br*

After 'sudo service NetworkManager restart', this solves the error of NetworkManager not being able to manage strictly unmanaged interfaces, which for the espressobin, are lan0, lan1, wan.

Some commands for NetworkManager:

• nmcli
• nmcli device show
• nmcli connection show

File locations:

• /etc/NetworkManager/system-connections/ - interface configurations
• /usr/share/doc/network-manager/README.Debian - notes about managed/unmanaged devices

Some references:

• Debian NetworkManager - mostly desktop?
• Debian NetworkConfiguration - nostly server?
• NetworkManager homepage
• nmcli - command-line tool for controlling NetworkManager
• NetworkManager.conf - NetworkManager configuration file
• NetworkManager - archlinux view of NetworkManager, with a section on VPN connectivity based upon profiles

Visual Studio Code, by default, when using the clangd language server, will automatically insert headers files for types which may have already been declared.

To disable this, go into the configuration options for clangd, and in 'clangd:arguments' add:

--header-insertion=never

Source: Stop VSCode from adding redundant headers

It is always interesting looking at bug reports. They uncover interesting nooks and crannies of system tooling. In reading Debian Bug#1037194, there is a command for watching dbus:

$ sudo dbus-monitor --system

To keep locally generated systemd service files separate from distribution files (as an example):

$ sudo mkdir -p /usr/local/lib/systemd/system
$ sudo vim /usr/local/lib/systemd/system/bme680.service
$ sudo systemctl enable bme680
$ sudo systemctl daemon-reload
$ sudo systemctl start bme680

An example systemd service file in /etc/systemd/system might look like:

[Unit]
Description=BME680 Collector
Documentation=https://github.com/rburkholder/bme680
After=network.target

[Service]
Type=simple
User=debian
WorkingDirectory=/home/debian
Environment="ID=02" "LOCATION=top floor"
ExecStart=/home/debian/bme680/build/bme680-mqtt ${ID} ${LOCATION}
ExecReload=kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=5s
StandardOutput=null
# note change with logrotate implemented, or to a memory file

[Install]
WantedBy=multi-user.target

The example from ecowitt2mqtt got me started - is a small CLI/web server that can receive data from Fine Offset weather stations (and their numerous white-labeled counterparts, like Ecowitt and Ambient Weather), adjust that data in numerous ways, and send it on to one or more MQTT brokers.

Reference to:

• systemd.service — Service unit configuration
• systemd.exec — Execution environment configuration - has list of expanded exit codes

Use these sites to test whether a service is up from other locations:

• https://testlocal.ly
• https://www.locasnap.com
• https://www.uptrends.com/tools/uptime
• https://www.site24x7.com/check-website-availability.html

And use these sites to ping/mtr from other locations (keeping in mind, of course, that some hosts block ping/mtr replies even when working normally):

• https://dnstools.ws
• https://ping.pe
• https://perfops.net/mtr-from-world

• dnsmadeeasy
• ddns rout53
• www.changeip.com

Wherever possible, we use RFC 2136, to the primary authoritative name server. pfSense firewalls, for example, support RFC 2136 for Dynamic DNS (tracking the WAN IP when DHCP changes it) and for Acme Certificates (demonstrate control of the FQDN). RFC 2136 is also a nice method for internal devices to get Let's Encrypt certs.

But you're not going to find RFC 2136 support very common. You probably won't find it on many NVR units or cheap Netgear routers.

• email-security-scans.org
• Port25’s Authentication and Spam Assassin Tool
• verifier.port25.com - send an email to this address to check headers
• multirbl.valli.org - "The complete IP check for sending Mailservers"
• postmastery - Check the DNA of your email against important best practices using our free email deliverability test.
• email-security-scans - Public Email & DNS Testbed - Considering just DKIM tests, this one is the only tester which explicitly and clearly recognizes ed25519 signatures. For the rest, EmailAudit reported a pass, but then said "DKIM key size is 0 bits. More senders are starting to use 2048 bits."
• internet.nl - test website, email, connectivity
• Wander Science DKIM Test
• uriports - Test and optimize SPF, DMARC, DKIM, and MTA-STS
• Google Admin ToolBox - Check MX - maybe be broken in some aspects, take with grain of salt
• Easy DMARC - verify dns data
• dmarcian - DMARC resources

Note: seems ';' (semi-colon) in records may help with some records.

• send an e-mail to spf@tester.realsender.com
• check online the validation results (it will take a minute to appear) realsender

I've been using check_mk for quite some time. It has progressed substantially recently and documentation has improved dramatically. However, their pythonesque configuration management has gone the way of the dinosaur, which has made my saltstack configuration management somewhat obsolete.

I'll have to recreate it as a REST interface solution somehow.

In the meantime, in looking at what else is out there, a few links:

• OpenAPM - a site which shows the relationship between significant monitoring solutions - tools of interest: netdata, checkmk (doesn't really have many upstream/downstream ties), prometheus, rsyslog, collectd, jaeger, grafana
• promethius
• Jaeger - trace analysis
• collectd - gathers metrics from various sources
• grafana - feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB

One of my current projects is for home automation. Currently it is based upon domoticz. Rather than relying on it's rather limited interface for data analysis, there are a couple of alternatives I'm thinking:

• Using dashticz
• cobbling together a series of best of class tools: TimescaleDB for holding timeseries data, telegraf which can take mqtt data streams and transmit to TimescaleDB, Grafana for handling querying and visualization of data. All the ZWave, Zigbee and other sensor data can be brought together for summary and for automation.
• MQTT Consumer Telegraf Input Plugin
• Monitoring temperature and humidity with MQTT - and using prometheus/grafana for back-end.

Prometheus appears to be popular for metric collection and distribution. Some add-ons for system monitoring for review:

• MQTT2Prometheus - exporter translates from MQTT topics to prometheus metrics, written in Go. The exporter can only listen to one topic_path per instance.
• Monitor IoT Devices in The Things Network with Prometheus and Grafana - extensive discussion on MQTT, Prometheus and Grafana with a little CBOR thrown in.
• Since I use RabbitMQ to run my MQTT gateway, how appropo: RabbitMQ - Monitoring with Prometheus & Grafana
• nut_exporter - Network UPS Tools (NUT) Prometheus Exporter [referenced from Prometheus Exporters and integrations]
• Monitoring proxmox with prometheus and grafana - with a Proxmox specific dashboard

Random stuff:

• sengled - light socket colour light bulbs for zigbee, bluetooth or wifi (zwave soon)
• ZigBeeForDomoticz - Zigbee plugin for Domoticz. Allow to connect various zigbee controllers like Zigate but also Texas Instrument CC2531, CC13x2, CC26x2 ; Silicon-Labs; deConz based chipset to be connected to Domoticz [linked from Zigbee Device Compatibility Repository

From a debian-boot mailing list:

A couple more options may help: it's sometimes possible that you've extracted the USB stick while files were still writing.

sudo dd \
  if=/debian-11.7.0-amd64-netinst.iso \
  of=/dev/sda \
  bs=1M oflag=dsync status=progress

This forces a sync on each write to the usb so you don't lose data and also gives you a brief status output on the size written.

I had a very old proprietary ADT alarm system in my house. It was time for an upgrade. But the desire during the upgrade was that alarm sensor values could gbe read by my home automation solution. The value in this is that I wouldn't need to duplicate sensors between the two systems: door open/close, motion detection, smoke, ...

The two primary solutions I found in the local market were systems based upon:

• Qolsys IQ 4 Panel.
• DSC Neo Panel, an advancement over the DSC PowerSeries

Both of these systems rely on PowerG sensors as well as proprietary encrypted busses and communications. PowerG sensors do not seem to be available to consumers outside of the alarm industry. And these systems do not allow any sort of read-only or rw access with consumer equipment. Everything is totally locked down and the consumer locked out. Only industry heavy-weights like Control4 have been allowed third party access to these systems.

I had to resort to some old school thinking on this.

I am presently evaluating the DSC PowerSeries panels and keypads.

taligentx/dscKeybusInterface is a library which can run on ESP8266 microcontrollers to communicate with the DSC un-encrypted DSC Keybus. This provides an ability to arm/disarm the alarm system via home automation. Home automation can also read the alarm state to provide larger awareness of the event. ESP8266 into existing alarm DSC System is a large thread about this style of integration from a Home Assistant automation perspective.

DSCKeyBus ESPHOME custom component to interface to a DSC POWERSERIES alarm system took taligentx's code and added bells and whistles to be used with Home Assistant. ESPHome provides the glue between the microcontroller world and Home Assistant.

My goal here is to extract the ESPHome components so that I might integrate the ESP8266 with Domoticz, an alternate home automation solution. Let us see where this goes.

The first step is the installation:

$ python3 --version
Python 3.11.2
$ mkdir esphome
$ cd esphome
$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install esphome
Collecting esphome
  Using cached esphome-2023.4.4-py2.py3-none-any.whl (2.6 MB)
Collecting voluptuous==0.13.1
  Using cached voluptuous-0.13.1-py3-none-any.whl (29 kB)
 ....
$ esphome version
Version: 2023.4.4

hmm, interesting, it has installed esptool, tornado and platformio

'deactivate' can be used to exit the environment. 'source venv/bin/activate' to activate the environment and regain the (venv) prompt.

In the directory esphome, run the following to generate a bare bones configuration file:

$ esphome wizard test01.yaml
... a bunch of prompts ...
$ cat test01.yaml
esphome:
  name: test1
esp8266:
  board: esp12e
# Enable logging
logger:
# Enable Home Assistant API
api:
  password: "no"
ota:
  password: "no"
wifi:
  ssid: "an_ssid"
  password: "ssid_pass"
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Test1 Fallback Hotspot"
    password: "randomstuff"
captive_portal:

This can be installed with:

 esphome run test01.yaml

There is a dashboard available for monitoring connected devices. Use this to create .yaml devices in esphome/config, which can then be built and uploaded to the device manually

pip install tornado esptool
esphome dashboard config/

More instructions can be found at Getting Started with the ESPHome Command Line

During the build process, I see that PlatformIO is used. Some reference links:

• twitter: platformio_org
• github: platformio/platformio
• IDE for embedded development

Note: all this is for basic integration into home-assistant. I have to find the tutorial which shows how to build a native/simple esphome file with the basics.

VirtualKeypad-Web.ino requires some files to be loaded into the ESP8266's local file space in order to show a virtual keypad for the DSC PowerSeries style panels and keypads.

It suggests using ESP8266FS tools to create a file system and upload files.

However, this is a .jar file and is only compatible with the older 1.8 flavours of the arduino-ide. The flavour I have have running currently is 2.1.1.

In addition, the ESP8266FS is deprecated, and has been superseded by LittleFS file system API. The LittleFS tool also does not appear to function in the v2 IDE, as it also is a .jar file (Java related?)

The modern way appears to be a two step process:

• use the hidden mklittlefs to create a filesystem with resident files
• use ElegantOTA to upload the blob to the ESP8266's file system area (http://local/update)
• or use esptool to upload via the command line

In my Linux install of the Arduino IDE install, there is a tools directory called ~/.arduino15/packages/esp8266/tools/ which has several tools:

$ ls -al ~/.arduino15/packages/esp8266/tools/
total 0
drwxr-xr-x 1 rpb rpb 90 May  5 11:50 .
drwxr-xr-x 1 rpb rpb 26 May  1 21:03 ..
drwxr-xr-x 1 rpb rpb 42 May  1 21:03 mklittlefs
drwxr-xr-x 1 rpb rpb 42 May  1 21:03 mkspiffs
drwxr-xr-x 1 rpb rpb 22 May  1 21:03 python3
drwxr-xr-x 1 rpb rpb 42 May  1 21:03 xtensa-lx106-elf-gcc

Source for mklettlefs is found on earlephilhower/mklittlefs github

Anyway, a command line from mklittlefs might look like this for the required files. This creates a .bin (binary) file which is a representation of the the file system.

libraries/DSC_Keybus_Interface/examples/esp8266/VirtualKeypad-Web$ \
  ~/.arduino15/packages/esp8266/tools/mklittlefs/3.1.0-gcc10.3-e5f9fec/mklittlefs \
  -b 4096 -p 256 -s 294912  -c data image.bin
/favicon-16x16.png
/style.css
/favicon-32x32.png
/fonts/dsc_icons.ttf
/fonts/dsc_icons.eot
/fonts/dsc_icons.woff
/fonts/dsc_icons.svg
/index.html

This binary file needs to be copied over to the filesystem partition of the ESP8266. I encountered three ways to do this:

• Using add-ons with the v1.8 of the Arduino IDE. Since the current version is v2, those add-ons no longer work
• Installing ElegantOTA or incorporating library calls to do so (which then presents a web page from the ESP8266 for uploading)
• Using the command line tool esptool for copying the file over via the serial port (this document is mostly about this method)

FireBeetle-ESP32 and mklittlefs parameters was a resource for the parameters. That site also referenced a python tool called esptool. Maybe I won't need the ElegantOTA tool after all.

esptool source has documentation.

esptool can be installed with:

$ python3 --version
Python 3.11.2
$ mkdir esptool
$ cd esptool
$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install esptool
Collecting esptool
  Using cached esptool-4.5.1.tar.gz (252 kB)
  Preparing metadata (setup.py) ... done
$ source venv/bin/activate

Ensure no other application is running on the port (like the Arduino IDE), and perform a test:

$ esptool.py -p /dev/ttyUSB0 --chip esp8266 flash_id
esptool.py v4.5.1
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: c8:c9:a3:66:93:14
Uploading stub...
Running stub...
Stub running...
Manufacturer: 20
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

If there is something connected you may see an error like:

Serial port /dev/ttyUSB0
A fatal error occurred: Could not open /dev/ttyUSB0, the port doesn't exist

However, in looking at the source for VirtualKeypad-Web.ino, there is a line which states:

$ grep serveStatic VirtualKeypad-Web.ino
  server.serveStatic("/", SPIFFS, "/").setDefaultFile("index.html");

As a result, this requires the Spiffs file system. And this can be created with the tool at ~/.arduino15/packages/esp8266/tools/mkspiffs/3.1.0-gcc10.3-e5f9fec/mkspiffs:

$ ~/.arduino15/packages/esp8266/tools/mkspiffs/3.1.0-gcc10.3-e5f9fec/mkspiffs -b 4096 -p 256 -s 294912  -c data spiffs.bin
/favicon-16x16.png
/style.css
/favicon-32x32.png
/fonts/dsc_icons.ttf
/fonts/dsc_icons.eot
/fonts/dsc_icons.woff
/fonts/dsc_icons.svg
/index.html

The -s parameter 294912 is a number which is a multiple of 4096 and larger that then size of the files combined. The tool will complain if not enough space has been allocated.

This can them be uploaded to the ESP8266 with:

$ esptool.py --chip esp8266 --port /dev/ttyUSB0  \
  write_flash  --flash_mode dio \
  --flash_freq 80m --flash_size 4MB 0x200000 \
  ../libraries/DSC_Keybus_Interface/examples/esp8266/VirtualKeypad-Web/spiffs.bin
esptool.py v4.5.1
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: c8:c9:a3:66:93:14
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00200000 to 0x00247fff...
Compressed 294912 bytes to 34730...
Wrote 294912 bytes (34730 compressed) at 0x00200000 in 4.0 seconds (effective 590.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

My ESP8266 has 4M of flash, and in the arduino tools menu, the board is set to a 2M filesystem size, so the offset is 2MB: 0x200000

Extras:

• ESP8266 Arduino Core’s documentation
• espressif esp32 forums
• esptool.py docs - note the dropdown with hardware specific selections
• espressif technical documents
• ESP8266 NodeMCU OTA (Over-the-Air) Updates – AsyncElegantOTA using Arduino IDE with a websockets tutorial

I've repurposed the notes at Install Home Assistant OS with KVM on Ubuntu headless (CLI only) for a Debian Bullseye (v11) system.

Start by checking that virtualization is available (the second command counts the number of instances, and needs to be non-zero):

# lscpu|egrep '(Model|Virtualization)'
Model:                           158
Model name:                      Intel(R) Xeon(R) E-2176G CPU @ 3.70GHz
Virtualization:                  VT-x
# egrep -c '(vmx|svm)' /proc/cpuinfo
24

The virtualization packages can be installed, networking is not included as it is already configured:

apt install -y \
  qemu-kvm \
  qemu-utils  \
  libvirt-daemon-system \
  libvirt-clients \
  ovmf

Optional packages include:

• numad - User-level daemon that monitors NUMA topology and usage
• pm-utils - Utilities and scripts for power management

Confirm a successful installation:

 systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-04-26 00:10:10 MDT; 5min ago

I will have to return to this note at a later time, I decided to give domoticz a try. I was reading a reddit posting about the regular upgrade issues HASS has. So I think something a bit leaner might be more my style. Albeit, it may require more hands on.

And as it turns out, with Domoticz, there is a bit more hand-holding, and more manual installation, but I think it fits my style better. It has forced a deep examinatiohn of how Domoticz, ESPHome, the Arduino IDE, and the various Arduino boards work together.

This project presents an implementation and designing of safe, secure and smart home with enhanced levels of security features which uses IoT-based technology. We got our motivation for this project after learning about movement of west towards smart homes and designs. This galvanized us to engage in this work as we wanted for homeowners to have a greater control over their in-house environment while also promising more safety and security features for the denizen. This contrivance of smart-home archetype has been intended to assimilate many kinds of sensors, boards along with advanced IoT devices and programming languages all of which in conjunction validate control and monitoring prowess over discrete electronic items present in home.

This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in long-term dependencies. We explain close-to-identity weight matrix, long delays, leaky units, and echo state networks for solving this problem. Then, we introduce LSTM gates and cells, history and variants of LSTM, and Gated Recurrent Units (GRU). Finally, we introduce bidirectional RNN, bidirectional LSTM, and the Embeddings from Language Model (ELMo) network, for processing a sequence in both directions.

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

Population censuses are vital to public policy decision-making. They provide insight into human resources, demography, culture, and economic structure at local, regional, and national levels. However, such surveys are very expensive (especially for low and middle-income countries with high populations, such as India), time-consuming, and may also raise privacy concerns, depending upon the kinds of data collected.

In light of these issues, we introduce SynthPop++, a novel hybrid framework, which can combine data from multiple real-world surveys (with different, partially overlapping sets of attributes) to produce a real-scale synthetic population of humans. Critically, our population maintains family structures comprising individuals with demographic, socioeconomic, health, and geolocation attributes: this means that our ``fake'' people live in realistic locations, have realistic families, etc. Such data can be used for a variety of purposes: we explore one such use case, Agent-based modelling of infectious disease in India.

To gauge the quality of our synthetic population, we use both machine learning and statistical metrics. Our experimental results show that synthetic population can realistically simulate the population for various administrative units of India, producing real-scale, detailed data at the desired level of zoom -- from cities, to districts, to states, eventually combining to form a country-scale synthetic population.