Entries by Raymond P. Burkholder

When needing to rebuild a personal linux machine, some suggested commands to collect state:

dpkg --get-selections > ~/mint/Package.list
apt-key exportall > ~/mint/Repo.keys
crontab -u wgking -l > ~/mint/crontab.backup

For some odd reason, my Bluetooth based mouse stopped working on my Debian Linux workstation. Going into the Bluetooth control panel in Settings indicated that no Bluetooth Adapter could be found.

In scouring the help groups, the solution is to go into BIOS, disable Bluetooth, reboot, restart into BIOS, and enable Bluetooth. In the end, this revived my Bluetooth adapters and devices.

One handy command I found to restart into the BIOS settings:

# systemctl reboot --firmware-setup

Building an enterprise datacenter is easy. When building multiple datacenters, and trying to share workloads, that is an animal of a totally different colour.

The goal is to have a separation of services into availability zones (which in itself, has multiple definitions).

From a very high level perspective, I looked at VMware NSX, Microsoft SDN, and OpenStack to see how they function.

First up is VMware NSX. NSX comes in two flavours: a) NSX-V, better known as NSX-vSphere, which uses VxLAN as the encapsulation layer, and b) NSX-T, better known as NSX Transformers, which uses Geneve as the encapsulation layer. VMware appears to be no longer adding services to NSX-V, and is only performing bug fixes. VMware is instead putting the development effort into NSX-T, which has ties into a broader range of platforms such as KVM, as well as ESXi/vSphere. Since NSX-V is, for all intents and purposes, being deprecated, I will focus on NSX-T. The key technology document supplied by VMware appears to be DOC-39405 - NSX-T 2.4 Multisite.

Based upon that document, it is difficult to design a two-semi-autonomous-datacenter solution. Basically, the lack of ‘local egress’ on stretched vlans without massive, all-or-nothing scripted gateway migration reduces perceived flexibility of their solution. That is, if you want to have a controller instance at each site, which in itself, drives up licensing costs, by about double. And there are limitations to WAN round trip times if you want to split the controller across sites, which gets into split brain problems and resiliency. VMware seems to be going to more of the MS PowerShell for automation. (and leaving Python in the dust?)

One more item is Brief History of VMware NSX saying "Active-active multi-site deployment is a joke and works almost as well as stretched data center fabric control plane - when you lose the inter-site link in an active-active setup, one of the sites shuts down".

With VMware no longer allowing kernel level insertions, there are very few effective avenues for providing third party routing/encapsulation engines. Cisco's ACI is said to have a plug-in for vSphere, and for insertion into each virtualization guest (which is a bit grotesque, I think). - 20191116 - Cisco programs directly against the VMware API now for the things they need to do.

So... moving on to Microsoft Hyper-V with the SDN add-on. Most of the docs, as of this writing, appear to reference Windows 2016, when Windows 2019 is 'out there'. And in those docs, the underlay/overlay does not appear to handle IPv6!! And in a similar manner to VMware, if two separate datacenters are required, a lot of SCCM and controller licensing is probably going to have to be doubled up. And I'm not really sure how two sites will connect or inter-operate. Another interesting fact is that the Microsoft SDN solution appears to use the Open vSwitch engine under the hood. But it appears as though they have disabled or hidden all the interesting command line and troubleshooting tools. More embrace, extend, and extinguish philosophy. And as a final observation, Microsoft SDN does use the VxLAN encapsulation capability. Cisco is said to have an ACI shim to fit in there to help with the network side provisioning aspects.

One more possibility I've been examining, and have been examining this one in more depth: openstack. No licensing fees. Automated via Python. Runs VxLAN as the encapsulation layer. Knows how to work with Open vSwitch and keep the related engine diagnostic commands visible. It can also integrate with Free Range Routing for the EVPN side of things. [TODO: find the Cisco link which talks about using LLDP and Cisco Data Center Network Manager (DCNM) to communicate with OpenStack to provision VxLAN end points]. With a standard multi-site deployment, OpenStack works with a single instance of Horizon and Keystone. I will need to experiment with that to understand the ramifications for Availability Zones.

• ZFS
• OpenZFS Developer Summit 2019
• Storage Configurator - from the conference
• iX Systems - storage vendor
• ZFS: Fun with ZFS – is compression and deduplication useful for my data and how much memory do I need for zfs dedup?
• How-To: Using ZFS Encryption at Rest in OpenZFS (ZFS on Linux, ZFS on FreeBSD, …)
• Performance tuning - OpenZFS

At one point in time, removing 'quiet' and placing 'text' on the grub/linux command line would boot in to console mode, the current way to do that is the remove 'quiet' and add the following to the linux kernel command line:

systemctl set-default multi-user.target

I need to keep this around and see what it does to the boot up log listing (based upon Force Debian console into text mode, it can be used when ILO complains of "Monitor is in graphics mode or an unsupported text mode.":

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset"

Another useful function is the following to disable graphical terminal in the grub boot file (use update-grub after any changes):

GRUB_TERMINAL=console

To build a wpa_supplicant configuration file:

# wpa_passphrase testap >> /etc/wpa_supplicant/testap.conf
  a_password
# cat /etc/wpa_supplicant/testap.conf
  # reading passphrase from stdin
  network={
          ssid="testap"
          #psk="a_password"
          psk=1b97a5c02076ac80739263fb33db49c7c50d3199202872e06097e7baf305f35e
  }

Additional info at How to connect to a WPA/WPA2 WiFi network using Linux command line.

To limit the amount of memory supplied to each container, place something similar to the following in the container configuration file:

lxc.cgroup.memory.limit_in_bytes = 5120M

The lxcfs daemon/service/package should be installed, and should be seen to be mounted on /var/lib/lxcfs

Prior to setting the memory limit, unfettered memory useage can be determined with

cat /sys/fs/cgroup/memory/lxc//memory.max_usage_in_bytes

When inside the container, something like 'free -m' should show the memory available to the container.

# free -m
              total        used        free      shared  buff/cache   available
Mem:          16029        5121        5890         444        5017       10046
Swap:          9535           0        9535

Some choice extractions from Configuration Management is an AntiPattern:

.... you’ve got an operations team that bottlenecks all changes in production. They’re the only ones with commit access to the configuration management repository, and they have to make all changes in production. This is the anti-DevOps way of doing things.

The other option is that you expect developers to run configuration management on the clusters that they are responsible for. This is the much more DevOps way of doing things, but it suffers from a different problem. Now you have to teach all your engineers the DSL of your configuration management software of choice, and depending on how you deploy your code to production, every developer now has the power to take down your whole system with a poorly written configuration change.

... you just have a separate configuration management branch for every cluster you run in production, and restrict developers to running code on only the systems they manage ...

configuration management promises that you’ll know the complete state of your infrastructure, but it never works that way

The alternative is immutable infrastructure. If you’re running in the cloud, this means baking your AMI with your application code already on it. See, imagine a world where you finish writing the code, you push it into git, and it gets built into an RPM or Debian package. That package is installed on a Base-AMI that was carefully crafted by your security and performance engineers, and then an AMI, with your software installed, along with its dependencies is pushed out

Start with a base, or foundation image — this is either your handcrafted, optimized image or just the default image from an upstream vendor. ... Your base image should have the latest security updates as well as any base infrastructure packages that are run platform-wide. Things like your monitoring packages, or your service discovery. Now, you could use a configuration management tool to build your base image, but you could also use OS packages and a little python to install and configure your base image.

This enables continuous deployments because new code just goes through your pipeline and you don’t have to deal with old versions of libraries or dependencies or configuration that might have been left around.

Moving to immutable allows you to take advantage of some cool release strategies. The first one is a rolling release, where nodes are replaced one at a time. This can be handy when you have state on your cluster that needs to be preserved.

My favorite release strategy is the Blue/Green push. You have 100 nodes, you start up 100 new nodes, and then you move traffic to the new nodes. If you have issues, you can quickly move traffic back to the old nodes. After a reasonable window, say 1–3 hours, you shutdown the old nodes.

Inspiration from Linux Find Out What Process Are Using Swap Space:

/proc/meminfo – This file reports statistics about memory usage on the system. It is used by free to report the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel. You can also use free, vmstat and other tools to find out the same information.

/proc/${PID}/smaps, /proc/${PID}/status, and /proc/${PID}/stat : Use these files to find information about memory, pages and swap used by each process using its PID.

From Top, something like:

KiB Mem : 16414448 total,  2233536 free, 10902532 used,  3278380 buff/cache
KiB Swap:  9764860 total,  3628544 free,  6136316 used.  4156564 avail Mem

Where is the swap space going? The following will list in descending sorted order:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

By using 'f' in Top, the SWAP field can be added to the display. LXC containers can also be added to the display. Then the listing can be sorted by SWAP to determine highest to lowest swap usage, and in what container, if any, is/are the offending process(es). As a note: SWAP approximately is VIRT - RES (total virtual memory - resident memory) -- with various notes abounding about the truthfulness of this.

I can't remember the source of this, probably some sort of Debian bug entry:

FT4232H - Hi-Speed Quad USB UART IC is FTDI’s 5th generation of USB devices. The FT4232H is a USB 2.0 Hi-Speed (480Mb/s) to UART IC. It has the capability of being configured in a variety of industry standard serial or parallel interfaces.

From Debian Bug report logs - #940105:

There were some reports over the last weeks from users on linux-btrfs which suffered from catastrophic btrfs corruption.

The bug which is apparently a regression introduced in 5.2 has now been found[0] an a patch is available[1].

Since it's unclear how long it will take to be part of a stable release and when Debian will pick this up in unstable, please consider to cherry-pick the patch.

• [0] lore.kernel.org
• [1] patchwork.kernel.org

My next stop in IDE evaluation was the Sublime Text editor. It showed much promise. But no CMake related integration. That was a short evaluation.

Jet Brains CLion seems to have all the bases covered. It knows CMake and can pull the include directories for building the symbol relationship database.

The drawback is that it is resource intensive. By using some of the Performance Tuning Tips, particularly increasing heap memory and disabling some inspections, things improved. Given enough time, it will still consume all allocated memory and will need to be restarted. The startup and initial scanning indicates a stack overflow, seemingly from a stack recursion issue, so there are some bugs to fix.

So far, given that, this is still the best of the bunch.

If I was on Windows, I'd probably return to Visual Studio. But that isn't available natively on Linux, only some VSCode thingy which is only an editor. Or maybe it has morphed into something more tangible lately. Oddly, it has Debug menu, but no Build menu, and no obvious mention of CMake.

Given some time, I'm thinking of following up on a combination of:

• Vim or NeoVim
• You Complete Me - a code-completion engine for Vim
• cquery - C/C++ language server supporting multi-million line code base, powered by libclang - cross references, completion, diagnostics, semantic highlighting and more
• NERDTree for viewing the file system in a tree view
• Tagbar for viewing the functions and classes in a tree view for the curent file.
• ctrlp.vim for fuzzy file search
• vim-cpp-enhanced-highlight highlights things commonly found in C++ programs such as STL functions and classes
• ag.vim for fuzzy content search, I usually still prefer using ag from the commandline though
• vim-fugutive for common git operations such as blaming and logging
• ultisnips for commonly used boilerplate
• Vim bindings for rtags - client/server indexer for c/c++ (not ctags, which doesn't use a compiler back end)

From reddit:

Spacemacs/SpaceVim with ycmd for auto completion and error checking, with rtags (or perhaps, soon, cquery) for indexing. All libclang based, all highly performant and responsive. Outside of C++ specific stuff, it has the best vim emulation, the best buffer/window/workspace management, the best git extension, and the best textual search/manipulation (powered by things like helm and helm-ag) of any editor I've used.

I've recently come to enjoy gdb + tmux + voltron for debugging with this workflow as well.

Ten years ago, I had written a blog entry about KDevelop. There was some sort of short coming, and moved on from there. I think my next environment was NetBeans.

Netbeans worked for me for quite a while. But when it was taken over as an Apache project, the C++ side of things have dwindled away. Only through an obscure programmer's build was I able to get it to build C++17 code.

That forced me to cycle back into Eclipse to edit and build my C++ apps. It has good integration with the CMake build process. That worked well. For a while. Then I started building some Linux kernel attached apps. Those apps require including various kernel libraries.

The pain point here is that there are a number of special include paths, and I could find no way to get those paths into the environment so I could open include paths and name references. By using a CMake build process, the C++ configuration items for those paths are all hidden/removed (as the Eclipse make system isn't used).

Coming full circle, KDevelop has evolved. I have evolved. And we seem to be meeting at the same point. KDevelop 'knows' how to pull the needed information from the CMake build process. My first half hour of use has provided being able to review #includes and the associated symbol references. So far so good.

The disadvantage of KDevelop is in window management. I can split windows, but to move a tab from split to split, doesn't seem to be possible. And on program restart, window/tab arrangement is not restored. And there are some symbols definitions to which it should be able to jump, but doesn't. KDevelop is not as good as it could be.

I think I could have worked with KDevelop, however, it couldn't seem to build a complete symbol table reference. That or I didn't twist the correct knobs. Going from file to file, following symbol referencesis how I typically come up to speed with code.

So... onward to something else.

There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors

Meanwhile, against this geopolitical backdrop we return from the summer to continue to see global markets which now aggressively buy equities for their yield and bonds for capital appreciation: which is a brilliant strategy until rates have been cut to new negative terminal lows and stock multiple expansion is at new highs: at which point the music stops and someone is left holding seriously over-priced and under-delivering cans. And nobody makes any returns at all – which will be a real big bang.

by Michael Every of Rabobank