Entries by Raymond P. Burkholder

On 06/14/2018 09:22 PM, someone wrote:
> So I have to ask, why is it advantageous to put this in a container 
> rather than just run it directly
> on the container's host? 

Most any host now-a-days has quite a bit of horse power to run services. All those services could be run natively all in one namespace on the same host, or ...

I tend to gravitate towards running services individually in LXC containers. This creates a bit more overhead than running chroot style environments, but less than running full fledged kvm style virtualization for each service.

I typically automate the provisioning and the spool up of the container and its service. This makes it easy to rebuild/update/upgrade/load-balance services individually and enmasse across hosts.

By running BGP within each container, BGP can be used to advertise the loopback address of the service. I go one step further: for certain services I will anycast some addresses into bgp. This provides an easy way to load balance and provide resiliency of like service instances across hosts,

Therefore, by running BGP within the container, and on the host, routes can be distributed across a network with all the policies available within the bgp protocol. I use Free Range Routing, which is a fork of Quagga, to do this. I use the eBGP variant (RFC 7938) for the hosts and containers , which allows for the elimination of the extra overhead of OSPF or similar internal gateway protocol.

Stepping away a bit, this means that BGP is used in a tiered scenario. There is the regular eBGP with the public ASN for handling DFZ-style public traffic. For internal traffic, private eBGP ASNs are used for routing traffic between and within hosts and containers.

With recent improvements to Free Range Routing and the Linux Kernel, various combinations of MPLS, VxLAN, EVPN, and VRF configurations can be used to further segment and compartmentalize traffic within a host, and between containers. It is now very easy to run vlan-less between hosts through various easy to configure encapsulation mechanisms. To be explicit, this relies on and contributes to a resilient layer 3 network between hosts, and eliminates the bothersome layer 2 redundancy headaches with spanning tree and such.

That was a very long winded way to say: keep a very basic host configuration running a minimal set of functional services, and re-factor the functionality and split it across multiple containers to provide easy access to and maintenance of individual services like dns, smtp, database, dashboards, public routing, private routing, firewalling, monitoring, management, ...

There is a higher up-front configuration cost, but over the longer term, if configured via automation tools like Salt or similar, maintenance and security is improved.

It does require a different level of sophistication with operational staff.

Other BGP routing Daemons:

• exabgp - python based
• gobgp - go based
• bird - c based
• bagpipe - python based, in openstack project

Other mailing list comments:

Our use case was both on exporting service IPs as well as receiving routes from ToRs. Exa is more geared towards the former than the latter. Rather then working on getting imports and route installation through Exa, we found it simpler with BIRD exporting the service IP from it bound to a loopback to run local healthchecks on the nodes and then have them yank the service IP from the loopback on failing healthchecks in order to stop exporting.

The intent of the original post was vague. Like a lot of people, I would not run a full BGP router in a container. Now, if the purpose is to inject or learn a handful of routes in order to do limited host routing, I can see the need. A route-server or a looking glass in a container would be fine, or something to perform analysis on the routing table, but not anything that has to route actual traffic.

I use ExaBGP to inject routes, perfect tool for that. If routes have to be received (not my use case) it makes more sense, as stated by previous posts, to use Quagga or BIRD. Which one is better : easy : if you like Cisco better, use Quagga. If you like Juniper better, use BIRD

BIRD looking glass looks very good

Using 5 Commands for Checking Memory Usage in Linux as a starting point, here is a summary of commands:

• top - use '<' and '>' to select different columns to sort
• free - to report in MB and to sum the columns: free -mt
• vmstat - 'vmstat -s' to provide labelled statistics, 'vmstat 2 5' - run five times at 2 second intervals, 'vmstat -d' for disk statistics
• dmidecode - 'dmidecode -t' to find types, 'dmidecode -t memory' shows memory modules, 'dmidecode -t memory|grep -i size' shows slots and how they are filled
• cat /proc/meminfo - 'egrep --color 'Mem|Cache|Swap' /proc/meminfo' to select and colorize entries

The reason to use these commands is my monitoring system says shared memory usage is increasing. The following command should show something, but doesn't:

# ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

# cat /proc/sysvipc/shm
       key      shmid perms                  size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime                   rss                  swap

# ipcs -a

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

# ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18014398442373116
min seg size (bytes) = 1

Maybe it doesn't matter. From top:

The amount of shared memory used by a task. It simply reflects memory that could be potentially shared with other processes.

A look behind the scenes on high frequency trading data:

• From 2012: TradingPhysics Historical TotalView-ITCH Files - a one post blog with some examination of a post mortem order book.
• QuantInsti’s Blog on Algo Trading and Quantitative Finance
• No BS Day Trading - training materials
• Idiots Guide to High Frequency Trading - Mark Cuban, some philosophy

• Stripe - is the best software platform for running an internet business. We handle billions of dollars every year for forward-thinking businesses around the world.
• adyen - A single payments platform to accept payments anywhere, on any device
• wirize.com - is a ready-made checkout payment built on Stripe. Collect one-time payments and sell your digital services in minutes (no code or website required).
• ElioPay - By connecting to Stripe, PayPal and PayBear you can accept credit and debit cards in 135+ currencies, PayPal and 7 cryptocurrencies including Bitcoin, Litecoin and Ethereum (with more to come). [We don't actually process the payments as we connect to Stripe, PayPal and Paybear]
• How to Accept Payments Online If You Can’t Use Stripe? -- two payment systems that could solve this problem: Gumroad and Paddle.

Payment Gateway Comparison: Guide to Ecommerce Payments compares PayPal, Stripe, Square, BrainTree, WePay, Authorize.net, 2CheckOut, MangoPay, WorldPay, Dwolla.

I recently picked up and browsed through a book called "Quench" by Dana Cohen and Gina Bria. They discuss how the hydration role of water can be optimized. Bottom line, it isn't about consuming large amounts of water. Rather, it is about how best to deliver water to the body's cells, even if delivered in less quantity.

A long while ago, I had cut down on my intake of table salt during meals. Excess salt tended to change the taste of food, and I was interested in tasting food based upon the original preparation. Cutting back on table salt is considered a good thing. But the body still needs electrolytes. Instead of consuming table salt during meals (where table salt typically is processed), the book recommends using a pinch of natural salt in water in the morning.

Natural salt could be selected from:

• Sea Salt
• Celtic Sea Salt
• Rock Salt
• Himalyan Salt

A morning hydrating drink could be:

• 8 to 12 oz. of plain or coconut water
• ping of natural salt
• squirt (0.5 to 1.0 oz) of lemon or lime juice
• teaspoon of honey or maple syrup

In the book, they create a recipe called the Tara Fix:

• 12 oz water
• 1 or 2 teaspoons of ground chea seeds (omega-3s)
• 4 ox kombucha
• 1 pinch of sea or rock salt

The book is loaded with information and background on the how's and why's of how this works.

Common houseplants that do double-duty. A little potted greenery can do wonders to brighten up your living space. But as Arielle Ray’s animated video explains, houseplants can also clear the air of harmful chemicals. The top three plants for surface area to chemical removal, according to a 1989 NASA study? The Gerbera daisy, snake plant, and English ivy.

• Download Windows 7, 8.1 or 10 ISO Images Direct From Microsoft
• Go in a scratch VM: When a linux machine starts, first some low-level magic happens to mount the root file system, and load and run the kernel. Once the kernel is ready it hands control to user-space by running /sbin/init as process ID 1. So, what happens if we replace /sbin/init with a statically linked Go binary that just prints “Hello World!” and then sleeps a lot?
• Developing CoreDNS backends with gRPC: CoreDNS normally provides serving DNS zone data through files, however it also allows you to proxy to a backend endpoint via gRPC.
• Practical IoT Cryptography on the Espressif ESP8266: This article will focus on applying AES encryption and hash authorization functions to the MQTT protocol using the popular ESP8266 chip running NodeMCU firmware.
• General Data Protection Regulation - Here you can find the official PDF of the Regulation (EU) 2016/679 (General Data Protection Regulation) in the current version of the OJ L 119, 04.05.2016; cor. OJ L 127, 23.5.2018 as a neatly arranged website.
• Multi-tier load-balancing with Linux
• Lets Encrypt - a highly detailed mechanism for generating keys for and obtaining Let's Encrypt certificates is described using a minimal LE tooling set.
• APT for package self-builders
• BTRFS and SE Linux - problems with systems running SE Linux on BTRFS losing the XATTRs used for storing the SE Linux file labels after a power outage.
• What makes a great leader, explained in eight counterintuitive charts
• Historical inventory of collaborative editors

• rwestphal/frr-bgp-l3vpn.yml - example for buiding an MPLS/L3VPN based EVPN. Note that the 'sysctl net.mpls.conf.XXX.input' flags should be set.
• "you can refer to the original man page of the daemon (it was ported from OpenBSD): https://man.openbsd.org/ldpd.conf.5 FRR's ldpd supports all the same configuration options, the CLI commands are different only in format."
• Default ARP Settings in Cumulus Linux - detailed discussion of ARP settings in Debian, with reference to load balancing interactions.
• FRR Docs
• FRR Dev Docs

• ROS with a wiki: ROS (Robot Operating System) provides libraries and tools to help software developers create robot applications. It provides hardware abstraction, device drivers, libraries, visualizers, message-passing, package management, and more. ROS is licensed under an open source, BSD license.
• FreeRTOS is a real-time operating system kernel for embedded devices that has been ported to 35 microcontrollers. It is distributed under the MIT License. It can even be built to run Xilinx Soft MicroBlaze FPGA based Real Time solution.

It is an interesting conjunction of events today: reading an article on sum-of-squares problems for optimal solution finding and reading an article about an indecisive car. It makes me think about what sort of algorithms, efficient or not, advanced or not, are being used in the technology to which we entrust our lives.

"Why Do They Keep Doing It?" Tesla On Autopilot Crashes Into Parked Laguna Beach Cop Car references multiple automated car accidents. One in particular is interesting:

Yesterday, video was posted showing what is allegedly a Tesla in Autopilot mode having a bout of "confusion" when the road being traveled changes from a straight road past a section on a highway that offers an exit by the road dividing. The video appears to show the car unable to determine which lane to stay in and nearly hitting the center median that divides the roadway from the exit.

In reading about minimizing polynomials in A Classical Math Problem Gets Pulled Into the Modern World, the author uses an example of an autonomous car and how it must make decisions for obstacle avoidance. The article implies that the algorithm referenced might be of value to autonomous vehicle programmers, drones, or, .. whatever.

But more strongly, the article does make a strong point that autonomous vehicles and drones are in 'continuous collision avoidance' mode, and that the solutions vary dynamically and quickly depending upon input quantity and variation. Everything works together: obstacle detection, identification, classification, and avoidance all work together. Not to mention the other parameters such as location awareness, resolution, and destination requirements, amongst others. Algorithmic decision making becomes a difficult problem to solve.

And if a vehicle becomes indecisive, what happens if the three laws of robotics get mixed in?

I've been building an Android app off and on. It uses NDK to delivery audio mixing and synthesis capabilities. There is functionality I would like to add which exists in third party libraries such as Boost and libsodium. There are a bunch of pieces necessary to make it all work: tooling, directory locations, options, ...

To start, Boost has quite a number of symbols, and even with improvements in Android Studio, which as I write this, is at v3.1.2, Android Studio will consume much CPU for many minutes while it builds symbol tables.

To help make this go a bit faster, it would probably be good to have the project installed on fast drives like SSD or NVME drives. In addition, if you have memory to spare, providing memory to Android Studio benefits the process. These are changes I made to my bin/studio.vmoptions file:

-Xms1024m
-Xmx4096m
-XX:ReservedCodeCacheSize=1024m

Some say the above doesn't work. An alternate location is in "Help" -> "Edit custom VM options" , which will create studio64.vmoptions, and put something similar in place:

-Xms1024m
-Xmx4096m
-XX:MaxPermSize=700m
-XX:ReservedCodeCacheSize=480m
-XX:+UseCompressedOops

Another suggested setting from Speed Up Gradle Build In Android Studio is "File" -> "Settings" -> "Gradle", and select "Offline Work".

Running 'invalidate caches' from the main File menu helped trial some changes in configuration. With three sets of boost include files (a set of files per each of the three architectures, it took about 12minutes for the Symbol Build to happen. I think for Boost, I will just use a single Include directory and list accordingly in the CMakeLists.txt file, rather than one for each architecture.

Changing anything in build.gradle, causes a total resync of everything, including rebuilding the Symbols. Uggh.

Reading in Reduce the APK size , I wonder if these Symbol Build issues have to with classes.dex file, as mentioned under the heading "Avoid enumerations". [in looking, it seems to be about MB in size, so that may not be the issue]

I'm not sure if it is really required, but Android Studio: Background task running indefinitely suggested putting the following into /etc/sysctl.conf file and restarting with 'sudo sysctl -p':

# Increase the limit for Android Studio
fs.inotify.max_user_watches = 500000

For the really desperate, more suggestions are found at Speed up your slow Android Studio.

When I had just one architecture, x86, the initial symbol build took a number of minutes. When I bumped it up to three architectures, 'x86', 'arm64-v8a', 'armeabi-v7a', building symbols seemed to take for ever. I had to add additional changes. In gradle.properties (in Gradle Scripts via Project window), I added:

org.gradle.daemon=true

I've scripted much of what I've learned, with the latest version available in github - rburkholder/android-ndk-build-libraries. Since the script is designed to run from within the libsodium directory and from within the boost directory, you will need to change into the directory of the library you wish to build, then run an absolute or relative path to run the script. BASE and PROJECT will need to be defined. You will probably want to read through the script first to understand the implied directory locations for the library, the build tools, the target android project, and the options available for the command line. Only some of the Boost libraries are built in this script. Additional ones can be easily added.

No matter what settings I applied to the boost shared library build, I was unable to remove the boost version codes at the end of the filename. There must be an option, but I havn't spotted it yet. So I build that library as a static library and link it it. Even changing 'version_type=linux' to 'version_type=none' in libtool.m4 in section 'linux*)' did not work. Even the patch suggested at NDK: Support for prebuild libs with full sonames didn't seem to work. But then I got to thinking, that it would be better to run with statically linked libraries. Since multiple architectures are packaged together, multiple architectures of the libraries will tend to bloat the package, where static libraries are reduced down to only the parts required. But having said that, for reference sake, there may be a solution to the Boost issue via an article at Build boost without version number suffixes where they suggest commenting out some or all of the following in boostcpp.jam:

    if $(type) = SHARED_LIB &&
      ! [ $(property-set).get  ] in windows cygwin darwin aix &&
      ! [ $(property-set).get  ] in pgi
    {
        result = $(result).$(BOOST_VERSION)  ;
    }

One of the next libraries to build will be ffmpeg, with a good write up at How to integrate pre-built libraries in case of the FFmpeg.

Via a suggestion at How to build Boost 1.64.0 libraries (static, shared) for ARM, MIPS, x86, +64-bit variants (using clang), it is possible to view the file type of the built libraries:

$ ar -t libboost_thread.a 
thread.o
once.o
future.o
$ ar -x libboost_thread.a  
$ ls -alt
total 1352
drwxr-sr-x 1 rpb rpb    498 May 29 16:34 .
-rw-r--r-- 1 rpb rpb  12080 May 29 16:34 future.o
-rw-r--r-- 1 rpb rpb   4184 May 29 16:34 once.o
-rw-r--r-- 1 rpb rpb 308220 May 29 16:34 thread.o
-rw-r--r-- 1 rpb rpb 359568 May 29 16:32 libboost_thread.a
$ file future.o
future.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

A possible optimization for the boost libraries is to run 'ranlib' on them prior to copying over to the final project. This code would need to be modified to work with the directory structures and destinations in the script, but is recorded here for posterity. From the man page, ranlib "enerates an index to the contents of an archive and stores it in the archive".

echo "Running ranlib on libraries..."
libs=$(find "bin.v2/libs/" -name '*.a')
for lib in $libs; do
  "$toolchain/bin/arm-linux-androideabi-ranlib" "$lib"
done

Some miscellaneous references for working with CMake and NDK:

• Getting Started with the NDK -
• Standalone Toolchains - the example lower in the page forms the basis of my script
• Android NDK Native APIs - Table 1 has an excellent summary of key native API support introduced by Android version, with the remainder of the page describing the libraries at each API step.
• ABI Management - primary ABIs are armeabi, armeabi-v7a, arm64-v8a, x86, x86_64.
• Configure CMake - the basics of what goes into the CMakeLists.txt file
• Configure Build Variants - what goes into the gradle.build file
• Link Gradle to Your Native Library -
• CMake tutorial -
• CMake Reference Documentation -

• Telegeography.com
• Hurricane Electric
• Cablemap
• Submarine Cable Map

When upgrading Windows 10 on a small HP Stream with limited drive space, I get an error code 0xC190020E.

Here are some steps to possibly get past this obstacle:

• Bring up task manager (ctrl-alt-del). Kill the windows update application.
• Bring up task manager (ctrl-alt-del). Switch over to the services tab. Stop the 'wuauserv' service.
• In control panel - programs and services - delete the windows update program
• Using file manager, delete c:\windows\SoftwareDistribution
• Using file manager, delete c:\Windows10Upgrade
• Right click the trash bin and empty the contents
• Connect a large USB drive, 10G or more, identify drive letter (probably d:)
• Start cmd.exe in administrator mode, run these commands:
  • mkdir d:\microsoft
  • mkdir d:\microsoft\updates
  • mkdir d:\microsoft\upgrades
  • mklink /j c:\windows\SoftwareDistribution d:\microsoft\updates
  • mklink /j c:\Windows10Upgrade d:\microsoft\upgrades
• start microsoft update
• it should start to use the USB drive for files. you will need at least 8G free on the c: drive for the process.
• hopefully the upgrade process works for you, it did for me

tmux is a terminal multiplexer includes most features of screen. tmux.github.io is the main reference. There is an online reference book. A Quick and Easy Guide to tmux helped me get started. More shortcuts.

Some commands (all prefixed with control-b):

• % - side by side pane
• " - top over bottom pane
• arrow keys - jump pane to pane
• ctrl-d - close a pane
• d - detach
• D - detach with choice
• ? - available commands list
• z - pane to full screen, again to shrink
• , - rename window
• { - move pane left
• } - move pane right
• q - show pane numbers
• ; - last pane

Rsesize Pane
• :resize-pane -<DURL> <x> [ Down, Up, Right, Left ] [relative count]
• :resize-pane -t <pane> -y <val> [pane number] [absolute height]
• :resize-pane -t <pane> -x <val> [pane number] [absolute width]

windows (virtual desktop)
• c - new window
• p - previous window
• n - next window
• number - window number from status bar
• l - last window

Session management
• tmux ls - running session list
• tmux attach -t 0
• tmux new -s sessionname
• tmux rename-session -t 0 sessionname
• tmux attach -t sessionname

Something from 24 Things I’ve Learned As A Software Developer Living On Ubuntu For A Weekish:

sudo modprobe v4l2loopback && \
  sudo ffmpeg -f x11grab -r 15 -s 1920x1080 -i :0.0+0,623 \
  -vcodec rawvideo -threads 0 -pix_fmt yuv420p -f v4l2 /dev/video0

What does this do? It uses ffmpeg to capture a display and output the stream as a v412loopback (easily installed from apt) to a file at /dev/video0, which happens to be the file-system’s representation of the webcam. v412loopback uses the same codec as the webcam, so by running this single line of code, I effectively turn my webcam into a screen-sharing tool, with or without support from WebEx. Couple this with a world-class, open-source broadcasting application like OBS (I still haven’t completely figured this out, but I’m onto something; ), and you have a more powerful screen-sharing tool than anything WebEx could offer anyways.