Technology

There are some useful pieces of information from a recent twitter thread on 'figuring out how systems are functioning'.

• "Three Pillars" implies some sort of definition. But "metrics, logs and traces" are a random grab bag! Two primitive data storage types (one of which should never be used anymore) and one visualization layer (for third type which isn't named!) -- Charity Majors
• I think the "Three Pillars of Observability" is an odd way to look a things. Pillars, by their nature are related only by what they hold up. Metrics nor tracing, should be pillars. All events should be traced and have metrics attached to them. -- Michael Wilde
• There are no three pillars of observability -- Charity Majors
• You want to know why observability is playing catch up? Because the EVENT is the execution path of your code through the system. It is the correct lens through which to understand your systems from the inside out. Without it, you are toast -- Charity Majors
• The event is the connective tissue for metrics, logs and tracing, because you can derive the first two from events and you can visualize the third from events. But it doesn't go in reverse: you can never work backwards from metrics, logs, and/or traces to get your event back -- Charity Majors
• Context is EVERYTHING in debugging, yet people have been happily discarding it all irretrievably at write time because they were so stuck in the mindset of metrics, logs and traces. This shit matters, man. How we frame it in our brain informs what we build going forward.-- Charity Majors
• A system is observable iff you can determine the behaviour of the system based on its outputs -- Greg Poirier
• Monitoring is the action of observing and checking the behaviior and outputs of a system and its components over time -- Greg Poirier

What hasn't been mentioned is 'high cardinality is king' and helps with context (if not, is the context).

2021/08/24 - Going off on a tangent: Observability: A New Theory Based on the Group of Invariance:

[Reviewed by Bill Satzer, on 10/31/2020 ]

Observability is a critical concept in control theory. Loosely speaking, the state of a system is observable if knowledge of its inputs and outputs during a given time interval permit determination of that state. For linear time-invariant systems, there is a simple criterion for observability: that the observability matrix (conceptually straightforward and easy to compute) should have full rank. For more general systems with all the inputs known there are similar criteria. But for nonlinear systems with unknown inputs, the problem is a good deal harder. This monograph offers a new approach to observability in nonlinear systems, one that was developed by the author.

The primary problem that the theory of observability studies is determining whether an input-output system provides the information necessary to estimate the state. Such information comes from sensors that provide measurements of system inputs and outputs. One way to formulate this problem quantitatively is to ask whether it is possible to determine the initial state of a system given system inputs and outputs over a given time interval.

I've been thinking about installing solar panels. In my area, Alberta, if I want to sell back to the power company, they will only allow me to connect to power grid about as much as I'd consume on average. That puts a dent on building bigger capacity. They say if I could create a co-op of nearby neighbors, I'd be able to increase my capacity.

So, in keeping with the thought of having extra capacity on hand, how can that energy be utilized. A little bit of searching indicates that hydrogen energy is in the forefront of many minds.

Currently, most hydrogen is produced by using hydrocarbons as a source. It seems that solar produced hydrogen isn't quite as efficient as desired.

Once I have a few moments, I need to further my research with the following starting points:

• RH2—The ultimate decarbonizer - RH2 - renewable hydrogen. Storage and transportation seems to be a concern. How does solar produced hydrogen fit in with the fuel cell concept? say, for powering vehicles? There is a note that "hydrogen can also be transported in natural gas pipelines, with blends of up to 20% hydrogen".
• Canada is one of the largest hydrogen producers in the world. - via The Canadian Hydrogen and Fuel Cell Association, indicates quite a bit of industry interest.
• SunHydrogen - has developed a breakthrough technology to make renewable hydrogen using sunlight and any source of water.
• Solar-to-Hydrogen Conversion Efficiency - papers via ScienceDirect describing hydrogen generation via electricity
• Low-cost direct solar-to-hydrogen ambitions see the light - from pv magazine: The road to cheap hydrogen production is riddled with potholes and energy losses. Researchers in Australia have demonstrated rethinking solar technology and skipping electrolysers could hold great promise for reaching the hydrogen holy grail. (2020/06/22)
• Canada prepares to unleash residential solar and create a green hydrogen economy—a convo with CanREA CEO Robert Hornung - An article at PVBuzz about the state of energy storage in Canada
• Micro-generation solutions - via Atco
• Canadian hydrogen and fuel cells industry - Federal Initiatives
• Office of Energy Efficiency & Renewable Energy - 2016/07/19 - current price at the pump for hydrogen gas is between $13-$16/kilogram (kg) – 1 kg of hydrogen contains roughly the same energy density as 1 gallon of gasoline. But, because a fuel cell is more than twice as efficient as an internal combustion engine, you basically need half the amount of hydrogen to go twice as far
• Canada: Micro-Generation In Alberta – Watt It's All About - 2020/11/23 - Microgeneration under the Alberta regulatory scheme means the generation of electricity from renewable or alternative energy sources for one's own use, in the amount of 5 MW or less. Another link.
• A Closer Look at Canada’s Hydrogen Future - from energyrates.ca
• Renewable energy could provide 95% of electricity in Canada by 2050 - 2021/07/22 - The small city of Summerside, Prince Edward Island in Canada installed a smart grid and five nodes in each home. They offered residents an incentive to switch from oil furnaces to electric, heat-storing units that can store up to three days of heat. With just 314 furnaces and smart water heaters installed and connected they flattened the curve on wind energy, enabling them to use all of the wind energy they were producing.
• -

• Zonemaster
• dnsviz
• EDNS Compliance Tester
• MX Toolbox - if you are not familiar with "dig" and the applicable RFCs
• dnssec-analyzer.verisignlabs.com - mainly about DNSSEC
• dnschecker.org/domain-health-checker.php - " ... of the eight warnings that it issued--and that the reporter thought needed attention--they were either incorrect, irrelevant, inscrutable, or simply wrong ..." - so be careful what you use
• emailaudit.com - checks SPF, DKIM, alignment, and DMARC.

Something I didn't know: if you add a '+' to the end of a bitly or tinyurl url, it will show what the actual url is and when it was created.

But twitter's t.co doesn't work that way.

Someone said Google's Translate will do something similar. So that might be an alternative.

wheregoes.com will also perform the checking. Ironically, from the tweet it goes through a t.co link, which does do the lookup.

I'm looking for a do-all home system, something to store camera video, maybe act as media center, and host some files. This maybe over-kill, but this is what I came up with so far

I did some searching through Reddit's Small Form Factor PC, I picked up a few ideas. Most pointed idea was the link for PCPartPicker to look for Mini-ITX boards with ECC memory. I first looked at the Asus P11C-I, but it didn't have any HDMI or Display Ports, which force the use of a Graphics card. But I did like their CPU Listing site for selecting CPUs.

There was the ASRock Rack Motherboard C236 WSI, but it doesn't appear to have a location for an M.2 storage card.

But the selector didn't show what I was looking for. In a reddit post I came across GigaByte C246N-WU2 which suits my needs better.

• GIGABYTE C246N-WU2 LGA 1151 (300 Series) Intel C246 SATA 6Gb/s Mini ITX Intel Motherboard
• Intel Xeon E-2176G Coffee Lake 3.70 GHz LGA-1151 80W CM8068403380018 Server Processor Intel UHD Graphics P630 - maybe overkill, and 80W, more than the recommended 65W for the case
• Crucial 16GB DDR4 SDRAM ECC Unbuffered DDR4 2666 (PC4 21300) Server Memory CT16G4WFD8266
• Fractal Design Node 202 case. Some discussion on fans - may need a custom 3D duct for short CPU fan. 202 designs. more fan stuff
• Noctua NH-L9i fan (but may need a printed 3D duct)
• Western Digital WD BLACK SN750 NVMe M.2 2280 1TB PCI-Express 3.0 x4 64-layer 3D NAND Internal Solid State Drive (SSD)
• CORSAIR SF Series SF450 450W 80 PLUS GOLD Active PFC SFX SFX12V Micro ATX Full Modular Power Supply
• Total: CDN $2000

As a reference, there is also SFF.Network for small form factor systems.

Chassis
• Premium Builds has some MiniITX case suggestions.
• Supermicro Superchassis CSE-721TQ-250B Mini-Tower with 250W Power Supply - seems to have good airflow
• SilverStone but their airflow doesn't look good

• BitZuma - blog and link to bitcoin book

• The Ultimate List of SANS Cheat Sheets - General IT Security, Digital Forensics and Incident Response, Penetration Testing, All Around Defender Primers (Linux, PowerShell, Network)
• Blueprint: MEF-SDN/NFV Certification Exam - more for the references to SDN sources
• spam resource
• MS SNDS
• infrapedia
• Canadian Numbering
• NANPA
• small mailserver bcp
• mta-sts
• data247 - paid service

• My Hard Drive Died - primary goal is to provide you with the data recovery knowledge and tools you need – whether it be our free videos and content, or our structured training (seated classes, distance learning or specialized).
• Resource Public Key Infrastructure (RPKI), the community-driven technology based on open standards that is aimed at making Internet routing more secure

Building an enterprise datacenter is easy. When building multiple datacenters, and trying to share workloads, that is an animal of a totally different colour.

The goal is to have a separation of services into availability zones (which in itself, has multiple definitions).

From a very high level perspective, I looked at VMware NSX, Microsoft SDN, OpenStack, and ProxMox to see how they function.

VMware NSX

First up is VMware NSX. NSX comes in two flavours: a) NSX-V, better known as NSX-vSphere, which uses VxLAN as the encapsulation layer, and b) NSX-T, better known as NSX Transformers, which uses Geneve as the encapsulation layer. VMware appears to be no longer adding services to NSX-V, and is only performing bug fixes. VMware is instead putting the development effort into NSX-T, which has ties into a broader range of platforms such as KVM, as well as ESXi/vSphere. Since NSX-V is, for all intents and purposes, being deprecated, I will focus on NSX-T. The key technology document supplied by VMware appears to be DOC-39405 - NSX-T 2.4 Multisite - a document which goes into great detail on how to build a multi-site NSX solution.

Based upon that document, it is difficult to design a two-semi-autonomous-datacenter solution. Basically, the lack of ‘local egress’ on stretched vlans without massive, all-or-nothing scripted gateway migration reduces perceived flexibility of their solution. That is, if you want to have a controller instance at each site, licensing costs are driven up by about double. And there are limitations to WAN round trip times if you want to split the controller across sites, which gets into split brain problems and resiliency. As another observation, VMware seems to be going to more of the MS PowerShell for automation. (and leaving Python in the dust?)

One more item is Brief History of VMware NSX saying "Active-active multi-site deployment is a joke and works almost as well as stretched data center fabric control plane - when you lose the inter-site link in an active-active setup, one of the sites shuts down".

With VMware no longer allowing kernel level insertions, there are very few effective avenues for providing third party routing/encapsulation engines. Cisco's ACI is said to have a plug-in for vSphere, and for insertion into each virtualization guest (which is a bit grotesque, I think). [2019/11/16 - Update - Cisco programs directly against the VMware API now for the things they need to do]

Microsoft Hyper-V

So... moving on to Microsoft Hyper-V with the (Software Defined Networking) SDN add-on. Most of the docs, as of this writing, appear to reference Windows 2016, when Windows 2019 is 'out there'. And in those docs, the underlay/overlay does not appear to handle IPv6!! And in a similar manner to VMware, if two separate datacenters are required, a lot of SCCM and controller licensing is probably going to have to be doubled up. And I'm not really sure how two sites will connect or inter-operate.

Another interesting fact is that the Microsoft SDN solution appears to use the Open vSwitch engine under the hood. But it appears as though they have disabled or hidden all the interesting command line and troubleshooting tools for Open vSwitch. More embrace, extend, and extinguish philosophy?

The documentation didn't provide any indication how the Microsoft VxLAN encapsulation would interact with, say Cisco Nexus switch VxLAN VTEP provisioning, which would allow third party devices access to the various VxLAN domains.

And as a final observation, Microsoft SDN does use the VxLAN encapsulation capability. Cisco is said to have an ACI shim to fit in there to help with the network side provisioning aspects.

OpenStack

One open source possibility I've been examining, and I have been examining this one in more depth: OpenStack. No licensing fees. Automated via Python. Runs VxLAN as the encapsulation layer. Knows how to work with Open vSwitch and keep the related engine diagnostic commands visible. It can also integrate with Free Range Routing for the BGP-EVPN side of things. Cisco has a tool called Cisco Nexus Fabric OpenStack Enabler for integrating OpenStack into a Cisco Nexus VxLAN enabled solution. With a standard multi-site deployment, OpenStack works with a single instance of Horizon and Keystone. I will need to experiment with that to understand the ramifications for Availability Zones.

ProxMox

Last on my list of candidates, is ProxMox which is a complete open-source platform for enterprise virtualization. It is designed mostly as a single site solution. But having access to the underlying networking, and with a simple to use API, engineering the appropriate integration with the network and with other sites is straight-forward. There is easy access to Free Range Routing, Open vSwitch, LXC, KVM, iproute2, ifupdown2, and related toolsets.

• Grey Noise - At GreyNoise, we tell you all of the things not to be worried about. Think of us as "anti-threat intelligence".
• FTTH Parts List - fusion splicers, light meter, fault locator, enclosures
• Stack OverFlow JSON Hijacking
• Haacked JSON Hijacking
• TrustWave JSON Hijacking
• WebAssembly’s post-MVP future: A cartoon skill tree - from Mozilla

• 6-Digit NTP Synchronized Network Clock with IEEE 802.3af PoE - LED digital wall clocks that synchronize over an Ethernet network to an accurate time reference. [with a starting point of A Guide To GPS NTP Servers For Network Time Synchronization]
• GPS NTP Network Time Server (TM1000A) - The TimeMachines GPS NTP Network Time Server is a simple to use GPS based time server that supplies accurate time for all computers and time keeping devices on the network.
• Nothing to see, but something to read: Network Synchronization: Public NTP Servers vs. GPS NTP Servers

Or not see:

• LeoNTP Network Master Clock / Time Server is a Stratum 1 Master Clock / Time server with GPS synchronised clock source designed from the ground up with the sole purpose of providing a cost effective but highly accurate and extremely high performing networked time server.

• crowd sourced tower map
• FCC Listings - Sizable towers need to be registered with the FAA

• Get-Console: "world's smallest device server", very inexpensive as well
• OpenGear: a bit more sophisticated, higher in price as well

I found that pictures I captured and edited were printing somewhat dark. Time to investigate how to get a consistent photographic workflow. I use darktable for photo management and preliminary colour management. So, some links I have to come back to:

• DisplayCAL - Open Source Display Calibration and Characterization powered by ArgyllCMS
• Loading/Applying profiles in Xfce in the displaycal forums.
• Display color management in darktable
• My Free and Open Source Photography Workflow - darktable, gimp, Geeqie, ArgyllCMS
• X-Rite i1Display Pro and ColorChecker Passport Bundle at B&H Photo Video

• Download Windows 7, 8.1 or 10 ISO Images Direct From Microsoft
• Go in a scratch VM: When a linux machine starts, first some low-level magic happens to mount the root file system, and load and run the kernel. Once the kernel is ready it hands control to user-space by running /sbin/init as process ID 1. So, what happens if we replace /sbin/init with a statically linked Go binary that just prints “Hello World!” and then sleeps a lot?
• Developing CoreDNS backends with gRPC: CoreDNS normally provides serving DNS zone data through files, however it also allows you to proxy to a backend endpoint via gRPC.
• Practical IoT Cryptography on the Espressif ESP8266: This article will focus on applying AES encryption and hash authorization functions to the MQTT protocol using the popular ESP8266 chip running NodeMCU firmware.
• General Data Protection Regulation - Here you can find the official PDF of the Regulation (EU) 2016/679 (General Data Protection Regulation) in the current version of the OJ L 119, 04.05.2016; cor. OJ L 127, 23.5.2018 as a neatly arranged website.
• Multi-tier load-balancing with Linux
• Lets Encrypt - a highly detailed mechanism for generating keys for and obtaining Let's Encrypt certificates is described using a minimal LE tooling set.
• APT for package self-builders
• BTRFS and SE Linux - problems with systems running SE Linux on BTRFS losing the XATTRs used for storing the SE Linux file labels after a power outage.
• What makes a great leader, explained in eight counterintuitive charts
• Historical inventory of collaborative editors