Technology

• My Hard Drive Died - primary goal is to provide you with the data recovery knowledge and tools you need – whether it be our free videos and content, or our structured training (seated classes, distance learning or specialized).
• Resource Public Key Infrastructure (RPKI), the community-driven technology based on open standards that is aimed at making Internet routing more secure

Building an enterprise datacenter is easy. When building multiple datacenters, and trying to share workloads, that is an animal of a totally different colour.

The goal is to have a separation of services into availability zones (which in itself, has multiple definitions).

From a very high level perspective, I looked at VMware NSX, Microsoft SDN, OpenStack, and ProxMox to see how they function.

VMware NSX

First up is VMware NSX. NSX comes in two flavours: a) NSX-V, better known as NSX-vSphere, which uses VxLAN as the encapsulation layer, and b) NSX-T, better known as NSX Transformers, which uses Geneve as the encapsulation layer. VMware appears to be no longer adding services to NSX-V, and is only performing bug fixes. VMware is instead putting the development effort into NSX-T, which has ties into a broader range of platforms such as KVM, as well as ESXi/vSphere. Since NSX-V is, for all intents and purposes, being deprecated, I will focus on NSX-T. The key technology document supplied by VMware appears to be DOC-39405 - NSX-T 2.4 Multisite - a document which goes into great detail on how to build a multi-site NSX solution.

Based upon that document, it is difficult to design a two-semi-autonomous-datacenter solution. Basically, the lack of ‘local egress’ on stretched vlans without massive, all-or-nothing scripted gateway migration reduces perceived flexibility of their solution. That is, if you want to have a controller instance at each site, licensing costs are driven up by about double. And there are limitations to WAN round trip times if you want to split the controller across sites, which gets into split brain problems and resiliency. As another observation, VMware seems to be going to more of the MS PowerShell for automation. (and leaving Python in the dust?)

One more item is Brief History of VMware NSX saying "Active-active multi-site deployment is a joke and works almost as well as stretched data center fabric control plane - when you lose the inter-site link in an active-active setup, one of the sites shuts down".

With VMware no longer allowing kernel level insertions, there are very few effective avenues for providing third party routing/encapsulation engines. Cisco's ACI is said to have a plug-in for vSphere, and for insertion into each virtualization guest (which is a bit grotesque, I think). [2019/11/16 - Update - Cisco programs directly against the VMware API now for the things they need to do]

Microsoft Hyper-V

So... moving on to Microsoft Hyper-V with the (Software Defined Networking) SDN add-on. Most of the docs, as of this writing, appear to reference Windows 2016, when Windows 2019 is 'out there'. And in those docs, the underlay/overlay does not appear to handle IPv6!! And in a similar manner to VMware, if two separate datacenters are required, a lot of SCCM and controller licensing is probably going to have to be doubled up. And I'm not really sure how two sites will connect or inter-operate.

Another interesting fact is that the Microsoft SDN solution appears to use the Open vSwitch engine under the hood. But it appears as though they have disabled or hidden all the interesting command line and troubleshooting tools for Open vSwitch. More embrace, extend, and extinguish philosophy?

The documentation didn't provide any indication how the Microsoft VxLAN encapsulation would interact with, say Cisco Nexus switch VxLAN VTEP provisioning, which would allow third party devices access to the various VxLAN domains.

And as a final observation, Microsoft SDN does use the VxLAN encapsulation capability. Cisco is said to have an ACI shim to fit in there to help with the network side provisioning aspects.

OpenStack

One open source possibility I've been examining, and I have been examining this one in more depth: OpenStack. No licensing fees. Automated via Python. Runs VxLAN as the encapsulation layer. Knows how to work with Open vSwitch and keep the related engine diagnostic commands visible. It can also integrate with Free Range Routing for the BGP-EVPN side of things. Cisco has a tool called Cisco Nexus Fabric OpenStack Enabler for integrating OpenStack into a Cisco Nexus VxLAN enabled solution. With a standard multi-site deployment, OpenStack works with a single instance of Horizon and Keystone. I will need to experiment with that to understand the ramifications for Availability Zones.

ProxMox

Last on my list of candidates, is ProxMox which is a complete open-source platform for enterprise virtualization. It is designed mostly as a single site solution. But having access to the underlying networking, and with a simple to use API, engineering the appropriate integration with the network and with other sites is straight-forward. There is easy access to Free Range Routing, Open vSwitch, LXC, KVM, iproute2, ifupdown2, and related toolsets.

• Grey Noise - At GreyNoise, we tell you all of the things not to be worried about. Think of us as "anti-threat intelligence".
• FTTH Parts List - fusion splicers, light meter, fault locator, enclosures
• Stack OverFlow JSON Hijacking
• Haacked JSON Hijacking
• TrustWave JSON Hijacking
• WebAssembly’s post-MVP future: A cartoon skill tree - from Mozilla

• 6-Digit NTP Synchronized Network Clock with IEEE 802.3af PoE - LED digital wall clocks that synchronize over an Ethernet network to an accurate time reference. [with a starting point of A Guide To GPS NTP Servers For Network Time Synchronization]
• GPS NTP Network Time Server (TM1000A) - The TimeMachines GPS NTP Network Time Server is a simple to use GPS based time server that supplies accurate time for all computers and time keeping devices on the network.
• Nothing to see, but something to read: Network Synchronization: Public NTP Servers vs. GPS NTP Servers

Or not see:

• LeoNTP Network Master Clock / Time Server is a Stratum 1 Master Clock / Time server with GPS synchronised clock source designed from the ground up with the sole purpose of providing a cost effective but highly accurate and extremely high performing networked time server.

• crowd sourced tower map
• FCC Listings - Sizable towers need to be registered with the FAA

• Get-Console: "world's smallest device server", very inexpensive as well
• OpenGear: a bit more sophisticated, higher in price as well

I found that pictures I captured and edited were printing somewhat dark. Time to investigate how to get a consistent photographic workflow. I use darktable for photo management and preliminary colour management. So, some links I have to come back to:

• DisplayCAL - Open Source Display Calibration and Characterization powered by ArgyllCMS
• Loading/Applying profiles in Xfce in the displaycal forums.
• Display color management in darktable
• My Free and Open Source Photography Workflow - darktable, gimp, Geeqie, ArgyllCMS
• X-Rite i1Display Pro and ColorChecker Passport Bundle at B&H Photo Video

• Download Windows 7, 8.1 or 10 ISO Images Direct From Microsoft
• Go in a scratch VM: When a linux machine starts, first some low-level magic happens to mount the root file system, and load and run the kernel. Once the kernel is ready it hands control to user-space by running /sbin/init as process ID 1. So, what happens if we replace /sbin/init with a statically linked Go binary that just prints “Hello World!” and then sleeps a lot?
• Developing CoreDNS backends with gRPC: CoreDNS normally provides serving DNS zone data through files, however it also allows you to proxy to a backend endpoint via gRPC.
• Practical IoT Cryptography on the Espressif ESP8266: This article will focus on applying AES encryption and hash authorization functions to the MQTT protocol using the popular ESP8266 chip running NodeMCU firmware.
• General Data Protection Regulation - Here you can find the official PDF of the Regulation (EU) 2016/679 (General Data Protection Regulation) in the current version of the OJ L 119, 04.05.2016; cor. OJ L 127, 23.5.2018 as a neatly arranged website.
• Multi-tier load-balancing with Linux
• Lets Encrypt - a highly detailed mechanism for generating keys for and obtaining Let's Encrypt certificates is described using a minimal LE tooling set.
• APT for package self-builders
• BTRFS and SE Linux - problems with systems running SE Linux on BTRFS losing the XATTRs used for storing the SE Linux file labels after a power outage.
• What makes a great leader, explained in eight counterintuitive charts
• Historical inventory of collaborative editors

It is an interesting conjunction of events today: reading an article on sum-of-squares problems for optimal solution finding and reading an article about an indecisive car. It makes me think about what sort of algorithms, efficient or not, advanced or not, are being used in the technology to which we entrust our lives.

"Why Do They Keep Doing It?" Tesla On Autopilot Crashes Into Parked Laguna Beach Cop Car references multiple automated car accidents. One in particular is interesting:

Yesterday, video was posted showing what is allegedly a Tesla in Autopilot mode having a bout of "confusion" when the road being traveled changes from a straight road past a section on a highway that offers an exit by the road dividing. The video appears to show the car unable to determine which lane to stay in and nearly hitting the center median that divides the roadway from the exit.

In reading about minimizing polynomials in A Classical Math Problem Gets Pulled Into the Modern World, the author uses an example of an autonomous car and how it must make decisions for obstacle avoidance. The article implies that the algorithm referenced might be of value to autonomous vehicle programmers, drones, or, .. whatever.

But more strongly, the article does make a strong point that autonomous vehicles and drones are in 'continuous collision avoidance' mode, and that the solutions vary dynamically and quickly depending upon input quantity and variation. Everything works together: obstacle detection, identification, classification, and avoidance all work together. Not to mention the other parameters such as location awareness, resolution, and destination requirements, amongst others. Algorithmic decision making becomes a difficult problem to solve.

And if a vehicle becomes indecisive, what happens if the three laws of robotics get mixed in?

Based upon the Call for Proposals for ONS Europe 2018 to be held in Amsterdam in September 2018, this is what the industry appears to be thinking about:

Networking Latest Trends
• Disaggregation of the Data Plane – Programmable Open Hardware including Silicon & White Boxes + Open Forwarding Innovations/Interfaces
• The Latest in Standards and Open Source Interworking to Drive Industry Forward
• Artificial Intelligence, Machine Learning, and Deep Learning Applied to Networking
• SD-WAN, IIOT, Data Insights, Business Intelligence
• Cloud Native, Kubernetes, and Network Automation
• Containers, Microservices, and the Service Mesh

Networking Business and Operations
• High Availability, Scalability, Security in Network Platforms
• Next Generation Automated Architecture (Enterprise, Carriers, Cloud)
• Business Case & ROI on Transformation, Next Gen Services Based on SDN/NFV
• Lessons Learned from Migrating/Integrating Legacy PNF/VNF/OSS/BSS to Automated Systems
• Migration of Data Center to Virtualized Software Defined Lessons and Roadmaps
• The Road to 5G deployment; POCs, Pilots, Partners, and Roll-out

(Technical) Service Provider & Cloud Networking
• Policy, Closed Loop Automation, and Zero Touch
• Edge Computing; Requirements, Tools, and Emerging Solutions
• NFV Security
• Building High Availability in SDN/NFV Solutions
• 5G Application Use Cases; What Comes First?
• The Shift to Microservice-based VNFs
• Multi-Cloud, and the Benefits it Brings

(Business & Architecture) Service Provider & Cloud Networking
• SDN/NFV Learnings to Date, ROI, TCO, Agility Gains, etc.
• Business Case and Strategy for 5G Workloads; IoT, VR/AR, Connected Cars, etc.
• Software Defined Packet Optical
• Mobile Edge Computing/4G Video/CDN
• Planning the Migration – Legacy Workload Migration, and Integration of New Systems with Existing Carrier OSS/BSS/FCAPS Systems

(Technical) Enterprise IT DevOps
• Software Defined Data Center Learnings
• Cloud Networking, End to End Solution Stacks – Hypervisor or Container Based
• Container Networking and the Shift to Microservices
• Scale and Performance in SDN Deployments
• Cloud Native App Development

(Business & Architecture) Enterprise IT (CXO/IT Architects)
• Digital Transformation – Best Practices
• Bringing DevOps to NetOps – Best Practices
• SDN, and Open Networking Case Studies; ROI, TCO, Operations Improvements, etc.
• Intent-Based Networking
• NFV for Enterprise (vE-CPE, etc.)
• Security – Evolving attack surfaces, agile attack counter-measures, etc.
• Industrial IoT, and the Smart Enterprise, Networking’s role

• Proper isolation of a Linux bridge talks about Linux bridges at the networking stack level. Very information and contains some isolation mechanisms I need to implement.
• alternative approach to rate limiting: rate limiting tcp requests for various services. has good technical discussion of various approaches (token bucket, window counters, sliding window log, sliding window counters) with a suggested purpose-built solution.
• Tcpdump: many examples of tcpdump queries
• 2018/05/18 - E-mail Cryptography

Some articles I need to go back to see what I can do with the iLO from the Linux Command line.

• iLO Configuration Procedure using hponcfg Utility: probably the most detailed of the list in terms of examples for reading and writing the config.
• Updating iLO Firmware Using Hponcfg and XML: a way to update the firmware, if there are firmware files available
• Scripting Toolkit for Windows - Using HPONCFG: has the first command I ran to get a basic XML configuration

# lspci -knn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:5910] (rev 05)
        Subsystem: CLEVO/KAPOK Computer Device [1558:65a1]
00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 05)
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Sunrise Point-H USB 3.0 xHCI Controller [1558:65a1]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Sunrise Point-H Thermal subsystem [1558:65a1]
        Kernel driver in use: intel_pch_thermal
        Kernel modules: intel_pch_thermal
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA Controller [AHCI mode] [8086:a103] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Sunrise Point-H SATA Controller [AHCI mode] [1558:65a1]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:1c.6 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #7 [8086:a116] (rev f1)
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a152] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Sunrise Point-H LPC Controller [1558:65a1]
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Sunrise Point-H PMC [1558:65a1]
00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a171] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Device [1558:65a2]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
        Subsystem: CLEVO/KAPOK Computer Sunrise Point-H SMBus [1558:65a1]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104M [GeForce GTX 1070] [10de:1be1] (rev a1)
        Subsystem: CLEVO/KAPOK Computer GP104M [GeForce GTX 1070] [1558:65a4]
        Kernel driver in use: nvidia
        Kernel modules: nvidia
6d:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTL8411B PCI Express Card Reader [10ec:5287] (rev 01)
        Subsystem: CLEVO/KAPOK Computer RTL8411B PCI Express Card Reader [1558:65a1]
        Kernel driver in use: rtsx_pci
        Kernel modules: rtsx_pci
6d:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 12)
        Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1558:65a1]
        Kernel driver in use: r8169
        Kernel modules: r8169
6e:00.0 Network controller [0280]: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter [168c:003e] (rev 32)
        Subsystem: Bigfoot Networks, Inc. QCA6174 802.11ac Wireless Network Adapter [1a56:1535]
        Kernel driver in use: ath10k_pci
        Kernel modules: ath10k_pci
6f:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804]
        Subsystem: Samsung Electronics Co Ltd Device [144d:a801]
        Kernel driver in use: nvme
        Kernel modules: nvme

Bonus section, identifying a computer:

$ sudo grep . /sys/devices/virtual/dmi/id/{bios,board,product}*
/sys/devices/virtual/dmi/id/bios_date:03/09/2017
/sys/devices/virtual/dmi/id/bios_vendor:American Megatrends Inc.
/sys/devices/virtual/dmi/id/bios_version:1.05.04RLS2
/sys/devices/virtual/dmi/id/board_asset_tag:Tag 12345
/sys/devices/virtual/dmi/id/board_name:P65_67HSHP
/sys/devices/virtual/dmi/id/board_serial:Not Applicable
/sys/devices/virtual/dmi/id/board_vendor:Notebook
/sys/devices/virtual/dmi/id/board_version:Not Applicable
/sys/devices/virtual/dmi/id/product_family:Not Applicable
/sys/devices/virtual/dmi/id/product_name:P65_67HSHP
/sys/devices/virtual/dmi/id/product_serial:Not Applicable
/sys/devices/virtual/dmi/id/product_uuid:435BFA80-DBC6-0000-0000-000000000000
/sys/devices/virtual/dmi/id/product_version:Not Applicable

Bonus section, #2, using dmi information (ignore the grep for all the raw info):

# dmidecode | grep -A3 '^System Information'
System Information
        Manufacturer: Notebook
        Product Name: P65_67HSHP
        Version: Not Applicable

It is a very rare occasion where I hear about someone doing things completely different from the main stream. Something in which a series of esoteric or raw computing science concepts are brought together to create a cohesive whole.

Today's episode starts with Adventures in the Rho Calculus. And onwards to thinking beyond programming languages with applied category theory, context, persistent memory, and single-level memory.

Backing up a bit, an interesting quote from the first:

Essentially Goldman built a distributed transaction log of everything their global corporate machine did — an enterprise-wide blockchain of sorts. But writing software for this sort of system is not trivial and certainly not something that fits with traditional Silicon Valley thinking. It’s only when multiple banks started trying to figure out how to co-mingle these transactions for traceability and regulatory purposes did the topic of blockchain-esque designs start in earnest. SecDB was software built for risk management — with the primary risk being the software system itself. This was a hot topic after the 2008 crash and probably not coincidence that Craig Wright’s research was in this same area and Satoshi Nakamoto emerged about the same time.

With a note to the paper on Rho Calculus

It is surprising how many of today's techniques hearken back to Lisp (WebAssembly) and Multics.

Unbundle Enterprise Capabilities introduces and refines concepts of early binding (static analysis compilers like C++) and late binding (iterative tools like Python). The article is one of several which incorporate CodeBuilders, a code/data analysis platform. A passing reference is made to SecDB (securities DB) built by Goldman, in which they recorded all trades and related information, which helped them survive the 2008 financial meltdown.

In the article Collaboration… Inception style, we get this interesting quote:

Developers must work through outdated file-based methodologies and deal with jarring edit-run processes (with only limited hot loading capabilities). While that may have been acceptable decades ago, Singh and Higgins proved there are better alternatives.

... which would tend to indicate that there are better ways of creating solutions that typically encountered.

Another article, AI Automation of Software Development, gets heavy:

... monadic structures to retain and manipulate computing state remain nebulous and are a long way from becoming easy to use or first-class structures (at least outside Haskell). NYC was an early adopter of a functional optimization technique called memoization that stores and tracks interim model state. The analogy in general computing (and getting back to the cone of specialization) is something called ‘function currying’. Wall Street found an interesting way to persist and manage graphs of curried functions.

A random reference elsewhere: Protocol Labs -- originators of Interplanetary File System.

• Reading privileged memory with a side-channel: the original source of the investigation, followed up with discussion
• Spectre and Meltdown: Vulnerabilities Status Page
• MeltDown
• Spectre
• Actions Required to Mitigate Speculative Side-Channel Attack Techniques: from the Chromium Projects
• Intel Issues Updates to Protect Systems from Security Exploits
• More details about mitigations for the CPU Speculative Execution issue
• Mitigations landing for new class of timing attack
• Vulnerability of Speculative Processors to Cache Timing Side-Channel Mechanism: (arm.com)
• Intel Responds to Security Research Findings: (intel.com)
• Linux page table isolation is not needed on AMD processors: (lkml.org)
• LLVM patch to fix half of Spectre attack (llvm.org)
• Spectre example code: with many people proving it out
• Michael Schwarz: list of attack publications
• CPU hardware vulnerable to side-channel attacks: (cert.org)
• Retpoline: a software construct for preventing branch-target-injection
• raphaelsc/Am-I-affected-by-Meltdown: Proof-of-concept / Exploit / checks whether system is affected by Variant 3: rogue data cache load (CVE-2017-5754), a.k.a MELTDOWN
• Detecting Spectre And Meltdown Using Hardware Performance Counters
• Testing Intel Core Processor Platforms and a Variety of Workloads: Intel Blog Initial Performance Data Results for Client Systems

Debian has a [DSA 4078-1] linux security update:

This specific attack has been named Meltdown and is addressed in the Linux kernel for the Intel x86-64 architecture by a patch set named Kernel Page Table Isolation, enforcing a near complete separation of the kernel and userspace address maps and preventing the attack. This solution might have a performance impact, and can be disabled at boot time by passing `pti=off' to the kernel command line.

2018/01/16: Meltdown/Spectre mitigation for 4.15 and beyond (LWN)

2018/01/21

• Loose threads about Spectre mitigation from Steinar H. Gunderson
• SMEP emulation in PTI

2018/01/23:

• Hacker News references lkml #1 and lkml #2.
• Do not enable KPTI for fixed Intel processors references [PATCH v2 0/5] Basic Speculation Control feature support

2019/07/24

• A Global Switch To Kill Linux's CPU Spectre/Meltdown Workarounds?
• commands and evaulation

2019/12/23 - Linux kernel gets another option to disable Spectre mitigations