Raymond P. Burkholder - Things I Do

Some choice extractions from Configuration Management is an AntiPattern:

.... you’ve got an operations team that bottlenecks all changes in production. They’re the only ones with commit access to the configuration management repository, and they have to make all changes in production. This is the anti-DevOps way of doing things.

The other option is that you expect developers to run configuration management on the clusters that they are responsible for. This is the much more DevOps way of doing things, but it suffers from a different problem. Now you have to teach all your engineers the DSL of your configuration management software of choice, and depending on how you deploy your code to production, every developer now has the power to take down your whole system with a poorly written configuration change.

... you just have a separate configuration management branch for every cluster you run in production, and restrict developers to running code on only the systems they manage ...

configuration management promises that you’ll know the complete state of your infrastructure, but it never works that way

The alternative is immutable infrastructure. If you’re running in the cloud, this means baking your AMI with your application code already on it. See, imagine a world where you finish writing the code, you push it into git, and it gets built into an RPM or Debian package. That package is installed on a Base-AMI that was carefully crafted by your security and performance engineers, and then an AMI, with your software installed, along with its dependencies is pushed out

Start with a base, or foundation image — this is either your handcrafted, optimized image or just the default image from an upstream vendor. ... Your base image should have the latest security updates as well as any base infrastructure packages that are run platform-wide. Things like your monitoring packages, or your service discovery. Now, you could use a configuration management tool to build your base image, but you could also use OS packages and a little python to install and configure your base image.

This enables continuous deployments because new code just goes through your pipeline and you don’t have to deal with old versions of libraries or dependencies or configuration that might have been left around.

Moving to immutable allows you to take advantage of some cool release strategies. The first one is a rolling release, where nodes are replaced one at a time. This can be handy when you have state on your cluster that needs to be preserved.

My favorite release strategy is the Blue/Green push. You have 100 nodes, you start up 100 new nodes, and then you move traffic to the new nodes. If you have issues, you can quickly move traffic back to the old nodes. After a reasonable window, say 1–3 hours, you shutdown the old nodes.

Inspiration from Linux Find Out What Process Are Using Swap Space:

/proc/meminfo – This file reports statistics about memory usage on the system. It is used by free to report the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel. You can also use free, vmstat and other tools to find out the same information.

/proc/${PID}/smaps, /proc/${PID}/status, and /proc/${PID}/stat : Use these files to find information about memory, pages and swap used by each process using its PID.

From Top, something like:

KiB Mem : 16414448 total,  2233536 free, 10902532 used,  3278380 buff/cache
KiB Swap:  9764860 total,  3628544 free,  6136316 used.  4156564 avail Mem

Where is the swap space going? The following will list in descending sorted order:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

By using 'f' in Top, the SWAP field can be added to the display. LXC containers can also be added to the display. Then the listing can be sorted by SWAP to determine highest to lowest swap usage, and in what container, if any, is/are the offending process(es). As a note: SWAP approximately is VIRT - RES (total virtual memory - resident memory) -- with various notes abounding about the truthfulness of this.

I can't remember the source of this, probably some sort of Debian bug entry:

FT4232H - Hi-Speed Quad USB UART IC is FTDI’s 5th generation of USB devices. The FT4232H is a USB 2.0 Hi-Speed (480Mb/s) to UART IC. It has the capability of being configured in a variety of industry standard serial or parallel interfaces.

From Debian Bug report logs - #940105:

There were some reports over the last weeks from users on linux-btrfs which suffered from catastrophic btrfs corruption.

The bug which is apparently a regression introduced in 5.2 has now been found[0] an a patch is available[1].

Since it's unclear how long it will take to be part of a stable release and when Debian will pick this up in unstable, please consider to cherry-pick the patch.

• [0] lore.kernel.org
• [1] patchwork.kernel.org

My next stop in IDE evaluation was the Sublime Text editor. It showed much promise. But no CMake related integration. That was a short evaluation.

Jet Brains CLion seems to have all the bases covered. It knows CMake and can pull the include directories for building the symbol relationship database.

The drawback is that it is resource intensive. By using some of the Performance Tuning Tips, particularly increasing heap memory and disabling some inspections, things improved. Given enough time, it will still consume all allocated memory and will need to be restarted. The startup and initial scanning indicates a stack overflow, seemingly from a stack recursion issue, so there are some bugs to fix.

So far, given that, this is still the best of the bunch.

If I was on Windows, I'd probably return to Visual Studio. But that isn't available natively on Linux, only some VSCode thingy which is only an editor. Or maybe it has morphed into something more tangible lately. Oddly, it has Debug menu, but no Build menu, and no obvious mention of CMake.

Given some time, I'm thinking of following up on a combination of:

• Vim or NeoVim
• You Complete Me - a code-completion engine for Vim
• cquery - C/C++ language server supporting multi-million line code base, powered by libclang - cross references, completion, diagnostics, semantic highlighting and more
• NERDTree for viewing the file system in a tree view
• Tagbar for viewing the functions and classes in a tree view for the curent file.
• ctrlp.vim for fuzzy file search
• vim-cpp-enhanced-highlight highlights things commonly found in C++ programs such as STL functions and classes
• ag.vim for fuzzy content search, I usually still prefer using ag from the commandline though
• vim-fugutive for common git operations such as blaming and logging
• ultisnips for commonly used boilerplate
• Vim bindings for rtags - client/server indexer for c/c++ (not ctags, which doesn't use a compiler back end)

From reddit:

Spacemacs/SpaceVim with ycmd for auto completion and error checking, with rtags (or perhaps, soon, cquery) for indexing. All libclang based, all highly performant and responsive. Outside of C++ specific stuff, it has the best vim emulation, the best buffer/window/workspace management, the best git extension, and the best textual search/manipulation (powered by things like helm and helm-ag) of any editor I've used.

I've recently come to enjoy gdb + tmux + voltron for debugging with this workflow as well.

2019/11/26 - ccls - C/C++/ObjC language server supporting cross references, hierarchies, completion and semantic highlighting - appears to be the successor to cquery.

2019/11/26 - reddit comments suggest to watch for clangd, which implements the Language Server Protocol, which is used to add smart features to your editor: code completion, compile errors, go-to-definition and more. getting started with clangd indicates it works with YouCompleteMe for VIM, Visual Studio Code, CMake-based projects, and others. clangd in clion shows how to enable clangd in CLion. A tutorial Using LSP & clangd in Vim.

Ten years ago, I had written a blog entry about KDevelop. There was some sort of short coming, and moved on from there. I think my next environment was NetBeans.

Netbeans worked for me for quite a while. But when it was taken over as an Apache project, the C++ side of things have dwindled away. Only through an obscure programmer's build was I able to get it to build C++17 code.

That forced me to cycle back into Eclipse to edit and build my C++ apps. It has good integration with the CMake build process. That worked well. For a while. Then I started building some Linux kernel attached apps. Those apps require including various kernel libraries.

The pain point here is that there are a number of special include paths, and I could find no way to get those paths into the environment so I could open include paths and name references. By using a CMake build process, the C++ configuration items for those paths are all hidden/removed (as the Eclipse make system isn't used).

Coming full circle, KDevelop has evolved. I have evolved. And we seem to be meeting at the same point. KDevelop 'knows' how to pull the needed information from the CMake build process. My first half hour of use has provided being able to review #includes and the associated symbol references. So far so good.

The disadvantage of KDevelop is in window management. I can split windows, but to move a tab from split to split, doesn't seem to be possible. And on program restart, window/tab arrangement is not restored. And there are some symbols definitions to which it should be able to jump, but doesn't. KDevelop is not as good as it could be.

I think I could have worked with KDevelop, however, it couldn't seem to build a complete symbol table reference. That or I didn't twist the correct knobs. Going from file to file, following symbol referencesis how I typically come up to speed with code.

So... onward to something else.

There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors

Meanwhile, against this geopolitical backdrop we return from the summer to continue to see global markets which now aggressively buy equities for their yield and bonds for capital appreciation: which is a brilliant strategy until rates have been cut to new negative terminal lows and stock multiple expansion is at new highs: at which point the music stops and someone is left holding seriously over-priced and under-delivering cans. And nobody makes any returns at all – which will be a real big bang.

by Michael Every of Rabobank

$ systemctl list-unit-files --state=enabled --no-pager
UNIT FILE                                  STATE  
cups.path                                  enabled
accounts-daemon.service                    enabled
anacron.service                            enabled
apparmor.service                           enabled
autovt@.service                            enabled
avahi-daemon.service                       enabled
binfmt-support.service                     enabled
blk-availability.service                   enabled
bluetooth.service                          enabled
console-setup.service                      enabled
cron.service                               enabled
cups-browsed.service                       enabled
cups.service                               enabled
dbus-fi.w1.wpa_supplicant1.service         enabled
dbus-org.bluez.service                     enabled
dbus-org.freedesktop.Avahi.service         enabled
....
logrotate.timer                            enabled
man-db.timer                               enabled

74 unit files listed.

As a continuation of a previous article on using KVM/Qemu from the command line, here are some simple instructions for installing Microsoft Windows 10 in a KVM/Qemu Virtualization environment.

To start, a Windows 10 installation ISO will be required from, for example: Windows 10 ISO. Also, to improve the response of Windows 10 in a KVM/Qemu environment, and to be able to read the local drives, the latest binary virtio drivers are required.

# qemu-img create -f raw images/win10.img 25g
# qemu-system-x86_64 \
  -drive id=disk0,format=raw,file=images/win10.img,if=virtio  \
  -drive file=/var/local/downloads/Win10_1803_English_x64.iso,index=1,media=cdrom  \
  -drive file=/var/local/downloads/virtio-win-0.1.171.iso,index=2,media=cdrom \
  -enable-kvm \
  -m 4096 \
  -cpu host \
  -smp cores=2,threads=4 \
  -name win10 \
  -net nic,model=virtio \
  -net tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-win10-v90 \
  -rtc base=localtime,clock=host \
  -usb -device usb-tablet \
  -vga virtio \
  -no-reboot \
  -boot d 

The above is used for the initial install. When Windows wants to reboot, remove the last two lines. For this example, this is a continuation of the previous environment where it uses an Open vSwitch script to attach the tap device to the OVS bridge via a vlan.

Network drivers will need to be installed from the virtio cd in directory NetKVM/w10/amd64.

Other interesting links:

• Windows 10 Virtualization with KVM - mentions the installation of the network card driver
• Installing Windows 10 on KVM (with Fedora Core) uses a virt-install mechanism but mentions Spice and QXL drivers for the display
• High KVM/QEMU CPU utilization when Windows 10 guest is idle talks about some performance tools for optimizing cpu utilization. When using the GUI managers, some extra options are enabled, which I have not enabled in my example above.
• Running Windows 10 on Linux using KVM with VGA Passthrough which is a very interesting article about how to configure pass-through devices in order to supply a guest with direct access to certain hardware devices.

Some of the perf commands from before:

perf kvm --host top -p `pidof qemu-system-x86_64`
perf stat -e 'kvm:*' -a -- sleep 1
perf kvm --host stat live

A python program which explains the helpers, which, by default, pulls info from linux-4.19.39/include/uapi/linux/bpf.h:

./linux-4.19.39/scripts/bpf_helpers_doc.py

A refined clang invocation to build an ebpf kernel load:

clang \
  -I /usr/include/x86_64-linux-gnu \
  -I linux-4.19.39 \
  -I /usr/src/linux-headers-4.19.0-5-common/include \
  -O2 -target bpf  -c sock_kern.c -o sock_kern.o

A refined clang invocation of the userland loader, which should run the samples in linux-4.19.39/samples/bpf/:

SRC=linux-4.19.39 \
LIB=${SRC}/tools/lib/bpf/libbpf.a \
clang \
  -I ${SRC}/tools/lib/bpf \
  -I ${SRC}/tools/lib \
  -I ${SRC}/tools/include \
  -I ${SRC}/tools/perf \
  -I ${SRC}/samples \
  -I linux-5.2.7/tools/include \
  -O2 -Wall -g \
  -o sock_user \
  ${LIB} -lelf  \
  sock_user.c ${SRC}/samples/bpf/bpf_load.c

On Debian, rather than loading the libbpf package it can be build from the kernel source to obtain libbpf.a:

# cd linux-4.19.39/tools/lib/bpf
# make

Auto-detecting system features:
...                        libelf: [ on  ]
...                           bpf: [ on  ]

  HOSTCC   fixdep.o
  HOSTLD   fixdep-in.o
  LINK     fixdep
  CC       libbpf.o
  CC       bpf.o
  CC       nlattr.o
  CC       btf.o
  CC       libbpf_errno.o
  CC       str_error.o
  LD       libbpf-in.o
  LINK     libbpf.a
  LINK     libbpf.so

It should be possible to do the same with linux-4.19.38/tools/bpf/bpftool, but fails on a missing bfd.h. But does work on linux-5.2.7/tools/bpf/bpftool.

'bpftool feature' will provide the features available in the current kernel. bpftool also has a bash completion file.

Some other bpftool invocations:

/bpftool feature
./bpftool map show
sudo apt install jq
./bpftool prog show --json id 52 | jq -c '[.id, .type, .loaded_at]'
# makes some things faster:
echo 1 > /proc/sys/net/core/bpf_jit_enable

To enable / disable at kernel boot time, place the options on the kernel CMDLINE in /etc/default/grub and then 'update-grub2':

apparmor=1 selinux=1 audit=1

Appropriate packages may need to be in place.

• AppArmor: How To Use

Some notes on manual operation of KVM/Qemu with open vswitch.

/etc/qemu-ifup:

#!/bin/bash
# logger shows that $1 is only supplied
logger "values = $1 $2 $3 $4 $5"
# a default bridge is set
BRIDGE=ovsbr0
# port name is similar to:  tap-deb1-v90 [last digits are a vlan id]
VLAN=`echo $1 | awk '/tap-[A-Za-z0-9]+-v[0-9]+/ {split($0,A,"-"); print substr(A[3],2)}'`
logger "vlan = $VLAN"
ovs-vsctl --may-exist add-br $BRIDGE
if [[ "$VLAN" != "" ]]; then
  ovs-vsctl --may-exist add-port $BRIDGE $1 tag=$VLAN
else
  ovs-vsctl --may-exist add-port $BRIDGE $1
  fi
ip link set dev $1 up

/etc/qemu-ifdown:

#!/bin/bash
# bridge needs to match that of the ifup script
BRIDGE=ovsbr0
ovs-vsctl --if-exists del-port $BRIDGE $1

Create a raw disk image of 8GB. QCow has compression/encryption overhead which isn't necessary in some instances. QEMU/Images references other formats, and describes how to use the loop interface to mount a raw image to view it.

qemu-img create -f raw images/deb1.img 8g

Initial build from a cd and prevent a reboot:

qemu-system-x86_64 \
  -drive format=raw,file=images/deb1.img  \
  -cdrom /home/rpb/Downloads/debian-testing-amd64-netinst.iso  \
  -no-reboot \
  -boot d \
  -m 1024 \
  -cpu kvm64 \
  -name deb1 \
  -no-reboot \
  -enable-kvm \
  -net nic,model=virtio \
  -net tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-deb1-v90

Subsequent operations can be run without the -cdrom and -no-reboot operations. The ifname will be the interface name as show in 'ip link' and the last two digits are the vlan to be used by the qemu ifup/ifdown scripts.

For disk io optimization in a virtual machine, it is suggested by Use elevator=noop For Linux Virtual Machines and KVM convert qcow2 disk images to raw disk images for performance to use raw disk images and to set the scheduler to none/noop:

# cat /sys/block/sda/queue/scheduler
[mq-deadline] none
# echo none > /sys/block/sda/queue/scheduler
# cat /sys/block/sda/queue/scheduler
[none] mq-deadline

Putting 'elevator=none' on the kernel command line via '/etc/default/grub' doesn't seem to work for me. Perhaps there is a newer flavour available.

A suggested command to test in qcow2, raw and host mode (generating 1.1GB of data):

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

'-ctrl-grab' changes to right-control instead of Ctrl-Alt.

Image file examination commands:

# qemu-img info images/buckeye1.img
image: images/buckeye1.img
file format: raw
virtual size: 8.0G (8589934592 bytes)
disk size: 1.9G

# file images/buckeye1.img
images/buckeye1.img: DOS/MBR boot sector

# fdisk -lu images/buckeye1.img
Disk images/buckeye1.img: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6e9340d7

Device               Boot    Start      End  Sectors  Size Id Type
images/buckeye1.img1 *        2048 14680063 14678016    7G 83 Linux
images/buckeye1.img2      14682110 16775167  2093058 1022M  5 Extended
images/buckeye1.img5      14682112 16775167  2093056 1022M 82 Linux swap / Solaris

Standard Performance Evaluation Corporation: buried in there somewhere should be suggested configurations for guests and hosts.

Block layer statistics in /sys/block//stat: a note for future monitoring

When upgrading kernels, I have an issue with lack of space in /boot, so I need to manually remove files, and the rebuild them. In the latest case, I managed to forget to run 'update-grub' which rebuilds the grub menu.

Upon reboot, the kernel aborted with an error along the lines of 'could not mount device(0,0). This was because there was a missing 'initrd .... ' line in the grub menu.

It didn't solve the problem, but I did learn a new sequence of commands. After booting with a live cd:

sudo mount /dev/nvme0n1p4 /mnt
sudo mount /dev/nvme0n1p2 /mnt/boot
sudo mount --bind /proc /mnt/proc
sudo mount --bind /dev /mnt/dev
sudo mount --bind /sys /mnt/sys
sudo mount --bind /dev/pts /mnt/dev/pts
chroot /mnt

The first mount mounts the root file system (I needed to look in /etc/fstab on the target to find that), and the second mounts my separate /boot partition. The other three mounts mount operating system virtual file systems.

The chroot then provides an environment within the target file system for performing management functions without having actually booted into the drive. This being the fundamental of containers.

Under certain circumstances, after changes, the following may be required:

update-grub2

Q: Why do programmers confuse Halloween and Christmas?
A: Because OCT 31 == DEC 25.