Open Source

Request: Or people simple do not want to share own email, as it is not related for delivery, but required by the form. Thus they fill semi random email like string.

Response: That is what mailinator is for, or the temporary addresses offered by several browser plugins, or if you know they're going to use it to spam you, abuse@ their domain. nobody@example.com works as well, and 1-900-976-1212 for a "required" phone number.

• make sure the MTA's forward and reverse DNS match
• set up an SPF record, probably "v=spf1 mx ~all"
• manage DNS MX, DKIM, DMARC and SPF records (SPF, DKIM, and DMARC are a recommended order of implementation)
• set up DKIM signing for each domain you host, make the DKIM domain match the From: domain
• start slow and look at any bounces
• maybe collect DMARC stats but for a small volume MTA, not very interesting
• instead of struggling with Postfix, OpenDKIM, Dovecot and friends (and losing out on quite a few features), try looking at maddy
• arrange for a backup MTA
• manage reverse lookup records, including managing the uncertain chain of authority between the instance and the nearest SOA
• manage certificates associated with TLS for SMTP and IMAP
• manage DKIM certificate
• manage one's upstream to address PBL issues
• keep the MTA secure and free from DOS attack
• Use a static IPv4 address for your email system [counter point: If DNS is fully functional and IP addresses don't change too quickly and TTL is configured properly on DNS records, ... then why is a static IP address strictly required? ]
• Do not share this IPv4 address with user machines
• Do not host your email system 'in the cloud'
• Make sure that your IP address is not listed in the PBL
• Provide an MX record
• Provide meaningful and consistent reverse DNS
• Your system should say HELO (or EHLO) with its hostname
• Keep your software completely up-to-date
• Ensure that only authorised users can send email through your system.
• Limit outgoing email volumes
• Accept reports of problems with your systems
• Review the mail system logs on a regular basis
• Be reliable (viz have at least 4 9s availability)
• Don't be an open relay
• Don't create backscatter
• Maintain a good reputation
• users can probably have acceptable success with all but the large email providers if they simply have their MTA hello as the name that the ISP assigns to the connection presuming that the ISP has forward and reverse DNS configured therefor.

Debian with Automated Snapper Rollbacks is a short tutorial about setting up a Debian linux system with automated BTRFS snapshots of the system and easy rollback to previous auto-generated snapshots. Once it's setup, it'll automatically take pre/post snapshots when you run `apt` and you can boot them from grub.

• email-security-scans.org
• Port25’s Authentication and Spam Assassin Tool
• verifier.port25.com - send an email to this address to check headers
• multirbl.valli.org - "The complete IP check for sending Mailservers"
• postmastery - Check the DNA of your email against important best practices using our free email deliverability test.
• email-security-scans - Public Email & DNS Testbed - Considering just DKIM tests, this one is the only tester which explicitly and clearly recognizes ed25519 signatures. For the rest, EmailAudit reported a pass, but then said "DKIM key size is 0 bits. More senders are starting to use 2048 bits."
• internet.nl - test website, email, connectivity
• Wander Science DKIM Test
• uriports - Test and optimize SPF, DMARC, DKIM, and MTA-STS
• Google Admin ToolBox - Check MX - maybe be broken in some aspects, take with grain of salt
• Easy DMARC - verify dns data
• dmarcian - DMARC resources

Note: seems ';' (semi-colon) in records may help with some records.

• send an e-mail to spf@tester.realsender.com
• check online the validation results (it will take a minute to appear) realsender

I've been using check_mk for quite some time. It has progressed substantially recently and documentation has improved dramatically. However, their pythonesque configuration management has gone the way of the dinosaur, which has made my saltstack configuration management somewhat obsolete.

I'll have to recreate it as a REST interface solution somehow.

In the meantime, in looking at what else is out there, a few links:

• OpenAPM - a site which shows the relationship between significant monitoring solutions - tools of interest: netdata, checkmk (doesn't really have many upstream/downstream ties), prometheus, rsyslog, collectd, jaeger, grafana
• promethius
• Jaeger - trace analysis
• collectd - gathers metrics from various sources
• grafana - feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB

One of my current projects is for home automation. Currently it is based upon domoticz. Rather than relying on it's rather limited interface for data analysis, there are a couple of alternatives I'm thinking:

• Using dashticz
• cobbling together a series of best of class tools: TimescaleDB for holding timeseries data, telegraf which can take mqtt data streams and transmit to TimescaleDB, Grafana for handling querying and visualization of data. All the ZWave, Zigbee and other sensor data can be brought together for summary and for automation.
• MQTT Consumer Telegraf Input Plugin
• Monitoring temperature and humidity with MQTT - and using prometheus/grafana for back-end.

Prometheus appears to be popular for metric collection and distribution. Some add-ons for system monitoring for review:

• MQTT2Prometheus - exporter translates from MQTT topics to prometheus metrics, written in Go. The exporter can only listen to one topic_path per instance.
• Monitor IoT Devices in The Things Network with Prometheus and Grafana - extensive discussion on MQTT, Prometheus and Grafana with a little CBOR thrown in.
• Since I use RabbitMQ to run my MQTT gateway, how appropo: RabbitMQ - Monitoring with Prometheus & Grafana
• nut_exporter - Network UPS Tools (NUT) Prometheus Exporter [referenced from Prometheus Exporters and integrations]
• Monitoring proxmox with prometheus and grafana - with a Proxmox specific dashboard

Random stuff:

• sengled - light socket colour light bulbs for zigbee, bluetooth or wifi (zwave soon)
• ZigBeeForDomoticz - Zigbee plugin for Domoticz. Allow to connect various zigbee controllers like Zigate but also Texas Instrument CC2531, CC13x2, CC26x2 ; Silicon-Labs; deConz based chipset to be connected to Domoticz [linked from Zigbee Device Compatibility Repository

From an email list, the definition of email list-washing:

Trying to identify the addresses in a spam list that will get you in trouble, e.g., spam traps and people with a history of complaining, so that you can mail to what's left. And also, by implication, dealing with complaints by saying "we took you off the list" without looking at the overall list quality.

The technical methods for removing stale bouncing addresses from a legitimate list are somewhat similar but the motivation is not.

Extracts from a mailop email list:

• little known control panel, for creating such email/web servers: Aetolos - secure open source control panel for virtual hosting
• Roundcube is the default webmail client these days, I am not sure if its a "replacement" for gmail, but it is very efficient and works great on mobile devices
• Postfix+dovecot is the most common combination for running your own services, once combined they offer pretty much all the features most people need (SMTP/IMAP/POP3/etc)
• ClamAV anti-virus is said to require at least 4GB of ram to load its default data set.
• For a full featured modern package with reasonable spam resistance I can vouch for Mailu which requires Docker as a basis. Under the hood, this is postfix as MTA, dovecot for IMAP/POP3, rspamd as anti-spam solution (I think the SPF/DKIM/DMARC stuff is located there, too), Roundcube or RainLoop as webmail (both pretty usable, probably not as feature-rich as GMail), PostgreSQL for account persistence, and REDIS as memory cache mostly for the rspamd engine. Supports Letsencrypt out of the box, of course. It does take some effort to set up the basics right (Docker and configuration) but then runs very reliably and is a breeze to manage. Sadly, it does not integrate Mailman, so I had to do that manually, which is kind of a pain.
• If you have a reasonable insight into unix things, you could do worse than going with OpenBSD and an OpenSMTPD setup along the lines of what Aaron Poffenberger describes in his tutorial here: opensmtpd for the real world mail server tutorial (originally a BSDCan session I believe).
• I’ve been running similar setups for years (but with exim as the MTA mainly for inertia reasons), documented mainly in articles you can find via the first link in my signature. The most comprehensive is perhaps Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools , others will be tagged with keywords like spam, mail, spamd, smtp and so forth.
• Gmail's webmail interface is one of the worst possible. It is very inefficient to operate, counter-intuitive, hides many important information from the user etc., not mentioning that it is simply ugly. There are many much nicer and more user-friendly webmails, like for example already mentioned Roundcube.
• Here I found an overview of various webmail apps, with screenshots of some. You can pick something that suits you: Top 15 Self-hosted open-source free web-based email clients
• I've begun with OpenSMTP, but I've switched to Postfix as the primary developer has stopped actively working on it. I now run Postfix, Dovecot, Solr (for the search), and Rspamd, and it's working perfectly!
• By the time you go through all the hassle of managing spam filters and getting your IP warmed up, Fastmail at $50/yr/mailbox looks pretty attractive.
• If you can find someone who resells Tucows' white label e-mail, they have a pretty good product for about $10/mailbox/yr for 5GB, $20 for 10GB, $30 for 15GB.
• take a look at mailcow dockerized. It's all you need (mail server, sogo groupware, antispam/-virus, 2fa,...). Put it on a VPS or your NAS and be happy.

Some diagnostic tools to remember:

$ sudo upsc apc@localhost

battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.mfr.date: 2014/07/01
battery.runtime: 1260
battery.runtime.low: 120
battery.temperature: 22.5
battery.type: PbAc
battery.voltage: 27.7
battery.voltage.nominal: 24.0
....
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.delay.start: 30
ups.firmware: 653.19.D
ups.firmware.aux: 11.03
ups.load: 13.0
ups.mfr: American Power Conversion
ups.mfr.date: 2011/11/03
ups.model: Smart-UPS 1500
ups.productid: 0002
ups.serial: AS1145221386
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: -1
ups.timer.shutdown: -1
ups.timer.start: -1
ups.vendorid: 051d

$ sudo upscmd -l apc@localhost

Instant commands supported on UPS [apc]:

beeper.disable - Disable the UPS beeper
beeper.enable - Enable the UPS beeper
beeper.mute - Temporarily mute the UPS beeper
beeper.off - Obsolete (use beeper.disable or beeper.mute)
beeper.on - Obsolete (use beeper.enable)
load.off - Turn off the load immediately
load.off.delay - Turn off the load with a delay (seconds)
load.on - Turn on the load immediately
load.on.delay - Turn on the load with a delay (seconds)
shutdown.reboot - Shut down the load briefly while rebooting the UPS
shutdown.return - Turn off the load and return when power is back
shutdown.stayoff - Turn off the load and remain off
shutdown.stop - Stop a shutdown in progress
test.battery.start.deep - Start a deep battery test
test.battery.start.quick - Start a quick battery test
test.battery.stop - Stop the battery test
test.panel.start - Start testing the UPS panel
test.panel.stop - Stop a UPS panel test

A list of possible variables: C.1. Variables

To find out what variables are in use by which UPS, visit Devices Dumps Library

UPS monitoring with NUT, Nagios and Check_MK

Some recent suggestions I've encountered:

• SpamAssassin KAM Rules - Just a note that the much better way to use the rules is with the KAM Channel, see KAM Channel
• Sane Security Signatures

Rspamd is not a dead project and it can do all the stuff OpenDKIM+OpenDMARC+OpenARC can do (yes, even ARC signing and DMARC reporting). I have even added support of OpenDKIM style signing tables [1].

So far, Rspamd supports ed25519 and dual-signing out of the box. It also supports Hashicorp Vault to store (and rotate!) DKIM secrets [2]. In terms of DKIM signing there is also support of fast signing (with all content scan features disabled), so it is highly unlikely that it's performance can be beaten by any other signing tool in this mode [3].

• Use of Signing Table
• DKIM signing using Vault
• Optimizing Signing Only Mode

Rspamd (embedded in a Mailu installation) for low-maintenance operation. That packages includes several mechanisms to score messages and handle them according to score intervals. This does a relatively good job but isn't really able to detect persistent spammers using their own IP ranges.

Are you aware of its reputation module? In conjunction with selector system you can build reputation on wide range of parameters without touching Lua. Beside its IP reputation i use sender's eSLD, IPnet & ASN reputation with score limited to not be enough for rejection.

If spammer uses not related IP, ASN and/or senders (snowshoe), you still can use its bayes and/or fuzzy checks. But basically the reputation module with RBL scoring does good job and in last year i do not remember any false negative mail and small amount of false positives, especially from gmail, these false positives are mostly not due reputation, but due SPF/DKIM/DMARC ignorance of our eshop systems.

I have small python script, which gets reputation data from rspam's redis DB and fill dynamic blocklist for exim, to block persistent spammers at MTA level (RCPT stage).

Anyway, I document what I do in a blog post: Zimbra AntiSpam Best Practices

The post describes how we rely on several layers of protection (including those provided by Postfix, which I know you are not using), as well as our adjustments to Amavis and SpamAssassin. I hope there are some techniques in the blog post that will be helpful to you.

For Let's Encrypt certificates, I use these awesome scripts, they are written as bash shell scripts and they are infinitely better than the official certbot tool, they can be used without a web server, by using DNS API integration. They are highly recommended:

• acme.sh
• dehydrated

For the first one, it no longer defaults to LE, it uses zerossl, because its rights are now owned by them. You can re-enable LE as default though by:

acme.sh --set-default-ca --server letsencrypt

I use Bruce Guenter's mailfront for my incoming mail so I had to add SNI support. It wasn't very hard. [when using SNI to differentiate between certificates]

if one abuses any of 25|465|587 they are blocked on all of 25/465/587 with a lengthy filter time

if one abuses any of 110|143|993|995 they are blocked on all of 110/143/993/995 with a slightly shorter filter time

Some articles I've been collecting for observability / distributed tracing:

• How to maximize span ingestion while limiting writes per second to Scylla with Jaeger - Jaeger primarily supports two backends: Cassandra and Elasticsearch. Here at Grafana Labs we use Scylla, an open source Cassandra-compatible backend. In this post we’ll look at how we run Scylla at scale and share some techniques to reduce load while ingesting even more spans. We’ll also share some internal metrics about Jaeger load and Scylla backend performance.
• The Cost of Containerization for Your Scylla - There is, however, a performance payoff for the operational convenience of using containers. This is to be expected because of the extra layer of abstraction (the container itself), relaxation on resource isolation, and increased context switches.
• Hacking your way to Observability — Part 3 - A distributed tracing quick start with Jaeger and OpenTelemetry

2021/09/26 - Two different links from two different lists talking about the same thing but differently:

• Five years evolution of open-source distributed tracing - provides temporal context for Hawkular, Zipkin, OpenTracing, Jaeger, OpenTelemetry, Hypertrace, Grafana Tempo and SigNoz
• Observability: The 5-Year Retrospective - how to find answers to unknown-unknowns.

2021/10/28 Hacker News - OpenTelemetry

• OpenTelemetry was OpenCensus, and before that it was OpenTracing
• OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.
• OpenMetrics relates to the metrics part of OpenTelemetry, where the latter is currently in alpha/experimental phase. OpenTelemetry has a statement on being interoperable with OpenMetrics for both accepting, forwarding and generating in the OpenMetrics format [0]. So they are two separate CNCF projects where OpenMetrics has overlap with a part of OpenTelemetry.

phoronix had an item on Software-Defined Radio Benchmarks, Other Updates This Week in terms of, since SDR is becoming popular and compute intensive, phoronix has added bencharks to their workload test tools.

Some interesting links originating from here:

• Open Benchmarking - LuaRadio - is a lightweight software-defined radio (SDR) framework built atop LuaJIT. LuaRadio provides a suite of source, sink, and processing blocks, with a simple API for defining flow graphs, running flow graphs, creating blocks, and creating data types.
• Open Benchmarking - gnuradio - is a free & open-source software development toolkit that provides signal processing blocks to implement software radios.
• Open Benchmarking - srsLTE - Open-source LTE software radio suite developed by Software Radio Systems (SRS).
• Open Benchmarking - liquidsdr - is home to liquid-dsp, a free and open-source signal processing library for software-defined radios written in C. Its purpose is to provide a set of extensible DSP modules that do not rely on external dependencies or cumbersome frameworks.

Somewhat associated: Noise Mapping with KSQL, a Raspberry Pi and a Software-Defined Radio - ADS-B capture, RTL2832U radio

zram, formerly called compcache, is a Linux kernel module for creating a compressed block device in RAM, i.e. a RAM disk, but with on-the-fly “disk” compression. Debian has a init system file for it.

I'm one of those who use zsmalloc as a module - mainly because I use zram as a compressing general purpose block device, not as a swap device. I create zram0, mkfs, mount, checkout and compile code, once done - umount, rmmod. This reduces the number of writes to SSD. Some people use tmpfs, but zram device(-s) can be much larger in size. That's a niche use case and I'm not against the patch.

Running out of RAM in Ubuntu? Enable ZRAM -

When using some relatively inexpensive parts to build a test three server Ceph solution, it seems that drive controllers and SSD drive combinations are finicky. Specifically, it seems that a Dell H310 controller may have issues with some or all Samsung SSD drives.

At Slow writes PERC H700 and Samsung 850 PRO SSD, a Dell response indicates:

The Samsung 850 Pro SSDs are not validated or certified to work with Dell controllers and as such there is a communication mismatch between the drives and the controller at the firmware level. As a result, you are bound to realize unexpected poor Read and Write performance regardless of controller cache settings.

It said that the Samsung 840 drive will work with the controllers. With Broadcom taking over LSI, they've removed old links, so documents are hard to find now. In trying to follow a document trail:

• Dell PowerEdge RAID Controller (PERC) H310, H710, H710P, and H810User’s Guide on page 9 says the H310 has an LSI2008 chipset. I'm looking at the H310 as it provides for direct passthrough to the drives with out RAID. The card has no caching.
• LSI SAS 2008 RAID Controller/ HBA Information - An H310 is similar to the LSI9211-8i
• Check Interoperability and Compatibility

To use the H310 for non-cache passthrough, it is recommended to flash the card in IT mode. I'll try this once I obtain the card. In the meantime, some possibly relevant links to the drivers and process:

• Disk Controller features and Queue Depth? - from 2014, discusses queue depth and diagnostic commands.
• Flashing the H310 Mono Mini to IT mode has a link to probably the best link to a flashing tutorial. But is complicated for flashing the mini H310. From the notes: the R610 is an 11th generation server, and when the mini H310 is reflashed to IT mode, the server probably won't reboot.
• Crossflashing the Dell PERC H200 and H310 to the LSI 9211-8i (from year 2018) implies that the H310 is the equivalent of an LSI 9211-8i. Also, a Dell H200 and an H310 must be similar from a firmware perspective.
• Flashing IT Firmware to the LSI SAS9211-8i HBA - 2012 - flash to a pci card
• How to crossflash PERC H310 to IT mode LSI 9211-8i firmware (HBA for FreeNAS, UnRAID) - an article from 2017 with better, more native instructions
• Crossflash Dell PERC H310 to LSI 9211-8i IT Mode Using Legacy (DOS) and UEFI Method (HBA Firmware + BIOS)
• PERC H310 - LSI 9211-8i - $50 - notes on purchase, install, and upgrade of card.
• Problems Flashing Dell Perc H310 - discusses H310 in PCI slot instead
• How-to Flash Dell Perc H310 with IT Firmware To Change Queue Depth from 25 to 600
• Updated: SAS HBA crossflashing or flashing to IT mode, Dell Perc H200 and H310 from 2016 where is is suggested that "Integrated, Mini or Mini Mono Perc H310 do NOT try to crossflash with these steps".
• DELL PERC H310/H710/H710P/H810 Controllers driver version 5.1.112.64,A00 - a starting point for the search of appropriate drivers
• LINUX PERCCLI Utility For All Dell HBA/PERC Controllers - and the starting point for command line utilities
• LSI MegaRAID SAS - some notes on the CLI utilities
• Doing battle with a Dell R620 and Ubuntu - has some remarks on H310 testing.
• H310 and Fan Noise
• Low random read/write on samsung 840 250gb ssd - spectre bios upgrades can destroy random read/write times

Drive Information:

• Don't do it: consumer-grade solid-state drives (SSD) in Storage Spaces Direct
• Ceph: how to test if your SSD is suitable as a journal device?
• High-performance cluster storage for IOPS-intensive workloads

Cluster test:

rados bench -p vm_storage 10 write -b 4M -t 16"

2022/04/13 - challenges

In a Dell R620, with an LSI SAS card, it took a while to find a utility to communicate with the card. Over the fold are some results: Continue reading "Dell R610 with H310 Drive Controller" »

There is a tool called testssl.sh - a free command line tool which checks a server's service on any port for the support of TLS/SSL ciphers, protocols as well as recent cryptographic flaws and more. Very extensive.

$ ./testssl.sh --starttls smtp myrelay.example.org:25
$ ./testssl.sh www.example.org

Or run some simpler tests:

$ openssl s_client -cipher 'DHE' -starttls smtp -crlf -connect myrelay.example.org:25

An older blog entry: SWAKS - Swiss Army Knife SMTP, the all-purpose SMTP transaction tester

2020/01/25 Send an email to

auth-results@verifier.port25.com

2020/01/25 Visit mail-tester.com for another flavour of email tester for spamminess.

2022/06/11 Authentication Checker

2022/06/11 mx toolbox

2024/01/26 about my email

• ZFS
• OpenZFS Developer Summit 2019
• Storage Configurator - from the conference
• iX Systems - storage vendor
• ZFS: Fun with ZFS – is compression and deduplication useful for my data and how much memory do I need for zfs dedup?
• How-To: Using ZFS Encryption at Rest in OpenZFS (ZFS on Linux, ZFS on FreeBSD, …)
• Performance tuning - OpenZFS