Entries by Raymond P. Burkholder

NAT exists in IPv6:

• IPv6-to-IPv6 Network Prefix Translation exists (RFC 6296, June 2011).

NATing between IPv6 and IPv4:

• Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers (RFC 6146, April 2011)
• Mapping of Address and Port using Translation (MAP-T) (RFC 7599, July 2015)

Delivering a virtual IPv4 link over IPv6:

• Dual-Stack Lite Broadband Deployments Following IPv4 Exhaustion (RFC 6333, August 2011)
• 464XLAT: Combination of Stateful and Stateless Translation (RFC 6877, April 2013)
• Mapping of Address and Port with Encapsulation (MAP-E) (RFC 7597, July 2015)

As for firewalls see:

• Recommended Simple Security Capabilities in Customer Premises Equipment (CPE) for Providing Residential IPv6 Internet Service (RFC 6092, January 2011)
• Thousands of ISP’s are using these RFCs to deliver working IPv4 and IPv6 to their customers today. Often the customer doesn’t even know they are using them.

If you don’t want to do a forklift upgrade deploy routers which support these RFCs as the old ones die and slowly migrate to a IPv6-only IPV4AAS model. Nobody has ever said that forklift upgrades where required.

Seen 2021/09/14 mailing list

Some articles I've been collecting for observability / distributed tracing:

• How to maximize span ingestion while limiting writes per second to Scylla with Jaeger - Jaeger primarily supports two backends: Cassandra and Elasticsearch. Here at Grafana Labs we use Scylla, an open source Cassandra-compatible backend. In this post we’ll look at how we run Scylla at scale and share some techniques to reduce load while ingesting even more spans. We’ll also share some internal metrics about Jaeger load and Scylla backend performance.
• The Cost of Containerization for Your Scylla - There is, however, a performance payoff for the operational convenience of using containers. This is to be expected because of the extra layer of abstraction (the container itself), relaxation on resource isolation, and increased context switches.
• Hacking your way to Observability — Part 3 - A distributed tracing quick start with Jaeger and OpenTelemetry

There are some useful pieces of information from a recent twitter thread on 'figuring out how systems are functioning'.

• "Three Pillars" implies some sort of definition. But "metrics, logs and traces" are a random grab bag! Two primitive data storage types (one of which should never be used anymore) and one visualization layer (for third type which isn't named!) -- Charity Majors
• I think the "Three Pillars of Observability" is an odd way to look a things. Pillars, by their nature are related only by what they hold up. Metrics nor tracing, should be pillars. All events should be traced and have metrics attached to them. -- Michael Wilde
• There are no three pillars of observability -- Charity Majors
• You want to know why observability is playing catch up? Because the EVENT is the execution path of your code through the system. It is the correct lens through which to understand your systems from the inside out. Without it, you are toast -- Charity Majors
• The event is the connective tissue for metrics, logs and tracing, because you can derive the first two from events and you can visualize the third from events. But it doesn't go in reverse: you can never work backwards from metrics, logs, and/or traces to get your event back -- Charity Majors
• Context is EVERYTHING in debugging, yet people have been happily discarding it all irretrievably at write time because they were so stuck in the mindset of metrics, logs and traces. This shit matters, man. How we frame it in our brain informs what we build going forward.-- Charity Majors
• A system is observable iff you can determine the behaviour of the system based on its outputs -- Greg Poirier
• Monitoring is the action of observing and checking the behaviior and outputs of a system and its components over time -- Greg Poirier

What hasn't been mentioned is 'high cardinality is king' and helps with context (if not, is the context).

2021/08/24 - Going off on a tangent: Observability: A New Theory Based on the Group of Invariance:

[Reviewed by Bill Satzer, on 10/31/2020 ]

Observability is a critical concept in control theory. Loosely speaking, the state of a system is observable if knowledge of its inputs and outputs during a given time interval permit determination of that state. For linear time-invariant systems, there is a simple criterion for observability: that the observability matrix (conceptually straightforward and easy to compute) should have full rank. For more general systems with all the inputs known there are similar criteria. But for nonlinear systems with unknown inputs, the problem is a good deal harder. This monograph offers a new approach to observability in nonlinear systems, one that was developed by the author.

The primary problem that the theory of observability studies is determining whether an input-output system provides the information necessary to estimate the state. Such information comes from sensors that provide measurements of system inputs and outputs. One way to formulate this problem quantitatively is to ask whether it is possible to determine the initial state of a system given system inputs and outputs over a given time interval.

How to get stats at all of the different points in the stack to track down reasons for dropped packets:

• ethtool -S for h/w and driver
• tc -s for drops by the qdisc
• /proc/net/softnet_stat for drops at the backlog layer
• netstat -s for network and transport layer
• ip -s

I've been thinking about installing solar panels. In my area, Alberta, if I want to sell back to the power company, they will only allow me to connect to power grid about as much as I'd consume on average. That puts a dent on building bigger capacity. They say if I could create a co-op of nearby neighbors, I'd be able to increase my capacity.

So, in keeping with the thought of having extra capacity on hand, how can that energy be utilized. A little bit of searching indicates that hydrogen energy is in the forefront of many minds.

Currently, most hydrogen is produced by using hydrocarbons as a source. It seems that solar produced hydrogen isn't quite as efficient as desired.

Once I have a few moments, I need to further my research with the following starting points:

• RH2—The ultimate decarbonizer - RH2 - renewable hydrogen. Storage and transportation seems to be a concern. How does solar produced hydrogen fit in with the fuel cell concept? say, for powering vehicles? There is a note that "hydrogen can also be transported in natural gas pipelines, with blends of up to 20% hydrogen".
• Canada is one of the largest hydrogen producers in the world. - via The Canadian Hydrogen and Fuel Cell Association, indicates quite a bit of industry interest.
• SunHydrogen - has developed a breakthrough technology to make renewable hydrogen using sunlight and any source of water.
• Solar-to-Hydrogen Conversion Efficiency - papers via ScienceDirect describing hydrogen generation via electricity
• Low-cost direct solar-to-hydrogen ambitions see the light - from pv magazine: The road to cheap hydrogen production is riddled with potholes and energy losses. Researchers in Australia have demonstrated rethinking solar technology and skipping electrolysers could hold great promise for reaching the hydrogen holy grail. (2020/06/22)
• Canada prepares to unleash residential solar and create a green hydrogen economy—a convo with CanREA CEO Robert Hornung - An article at PVBuzz about the state of energy storage in Canada
• Micro-generation solutions - via Atco
• Canadian hydrogen and fuel cells industry - Federal Initiatives
• Office of Energy Efficiency & Renewable Energy - 2016/07/19 - current price at the pump for hydrogen gas is between $13-$16/kilogram (kg) – 1 kg of hydrogen contains roughly the same energy density as 1 gallon of gasoline. But, because a fuel cell is more than twice as efficient as an internal combustion engine, you basically need half the amount of hydrogen to go twice as far
• Canada: Micro-Generation In Alberta – Watt It's All About - 2020/11/23 - Microgeneration under the Alberta regulatory scheme means the generation of electricity from renewable or alternative energy sources for one's own use, in the amount of 5 MW or less. Another link.
• A Closer Look at Canada’s Hydrogen Future - from energyrates.ca
• Renewable energy could provide 95% of electricity in Canada by 2050 - 2021/07/22 - The small city of Summerside, Prince Edward Island in Canada installed a smart grid and five nodes in each home. They offered residents an incentive to switch from oil furnaces to electric, heat-storing units that can store up to three days of heat. With just 314 furnaces and smart water heaters installed and connected they flattened the curve on wind energy, enabling them to use all of the wind energy they were producing.
• -

• update-help
• get-help -name about*
• get-variable
• get-command -verb
• get-command -noun
• $psversiontable.psversion
• $null - can be assigned to and assigned from
• $var.gettype().name
• get-member -inputobject $variable
• $var = @('one','two') - array variable
• $var = @{ test1=1; test2='here' } - hash table
• get-content -path $filepath - can be used to pipe entries (piped as objects)
• get-help -name get-service -full - for details and examples

Using Linux Commands Cheat Sheet 2021, I found a few commands I did not know:

• last reboot - system reboot history, with kernel versions
• timedatectl - Query the system clock
• cal - current calendar (month and day)
• w - logged in users
• id
• last - last system logins
• pstree - processes in a tree-like diagram
• pmap - see loaded libraries
• netstat -pnltu
• netstat -nutlp

Secure Linux Servers by Default is a rant about hardening Linux servers and keeping them up-to-date. The article references:

• DevSecHardening Framework
• Ubuntu Security Features
• NSA Guide to Secure Configuration

There is also the CIS Debian Linux Benchmark for hardening Debian based Linux systems. I have this coded as a series of Salt States. Will need to make that public at some point.

While I'm at it, a few packages for file and system validation from a security and intrusion perspective:

• tiger - Report system security vulnerabilities, which can use tripwire or aide
• aide - Advanced Intrusion Detection Environment - static binary
• samhain - Data integrity and host intrusion alert system
• tripwire - file and directory integrity checker
• checksecurity - basic system security checks: setuid, sockets, passwd, disfree, ...
• systraq - monitor your system and warn when system files change - not recently maintained
• stealth - stealthy File Integrity Checker

• Zonemaster
• dnsviz
• EDNS Compliance Tester
• MX Toolbox - if you are not familiar with "dig" and the applicable RFCs
• dnssec-analyzer.verisignlabs.com - mainly about DNSSEC
• dnschecker.org/domain-health-checker.php - " ... of the eight warnings that it issued--and that the reporter thought needed attention--they were either incorrect, irrelevant, inscrutable, or simply wrong ..." - so be careful what you use
• emailaudit.com - checks SPF, DKIM, alignment, and DMARC.

While out for a run this afternoon, I hear the swoop of wings. Very, very close. Looking up, I see a large bird climbing back up from the pass, circling back around, and diving in for another overhead pass. Two, three, four, five feet miss, max. I think that happened five or six times. Second last time, I involuntarily ducked, it was that close. That bird had no fear. I felt I could have reached up and plucked the bird from the air!.

More TODO looking:

$ cat /proc/diskstats
 259       0 nvme0n1 63840307 12937870 2293335051 33430251 96127804 61305555 15099846424 191811657 0 96413608 239045439 0 0 0 0 3690911 13803530
 259       1 nvme0n1p1 393 1000 29745 67 2 0 2 13 0 404 81 0 0 0 0 0 0
 259       2 nvme0n1p2 416 506 23256 48 98 1 214 252 0 356 301 0 0 0 0 0 0
 259       3 nvme0n1p3 279609 8152 46434210 58163 765192 382955 254395816 7307992 0 3161560 7366155 0 0 0 0 0 0
 259       4 nvme0n1p4 5696585 40470 560165192 6789555 256010 14505 38464520 203018 0 2718956 6992574 0 0 0 0 0 0
 259       5 nvme0n1p5 5997688 132712 465611808 3669012 19168871 263191 2499497624 45842492 0 18926656 49511505 0 0 0 0 0 0
 259       6 nvme0n1p6 12243830 121349 803005424 12340941 59298520 1387653 11690638024 89369664 0 53198444 101710606 0 0 0 0 0 0
 259       7 nvme0n1p7 39621442 12633681 418051776 10572440 13066758 59257250 616850224 35479804 0 15527632 46052244 0 0 0 0 0 0
 179       0 mmcblk0 704 1577 43388 8432 142 96 11206 16617 0 4792 25049 0 0 0 0 0 0
 179       1 mmcblk0p1 14 0 112 68 0 0 0 0 0 76 68 0 0 0 0 0 0
 179       2 mmcblk0p2 31 0 2104 288 0 0 0 0 0 192 288 0 0 0 0 0 0
 179       3 mmcblk0p3 116 224 4960 1977 9 6 30 1627 0 2440 3605 0 0 0 0 0 0
 179       4 mmcblk0p4 197 1314 16138 1046 81 67 1184 10153 0 2904 11199 0 0 0 0 0 0
 179       5 mmcblk0p5 78 0 5056 1666 7 5 1040 1024 0 2164 2690 0 0 0 0 0 0
 179       6 mmcblk0p6 72 38 4746 1185 8 1 72 1210 0 2232 2396 0 0 0 0 0 0
 179       7 mmcblk0p7 138 1 8056 2103 37 17 8880 2601 0 2980 4704 0 0 0 0 0 0

TODO: look at what is covered and what is missing:

$ ls -al /sys/class/hwmon
total 0
drwxr-xr-x  2 root root 0 May 29 11:02 .
drwxr-xr-x 62 root root 0 May 28 15:09 ..
lrwxrwxrwx  1 root root 0 May 29 11:02 hwmon0 -> ../../devices/virtual/thermal/thermal_zone0/hwmon0
lrwxrwxrwx  1 root root 0 May 29 11:02 hwmon1 -> ../../devices/pci0000:00/0000:00:1d.0/0000:72:00.0/hwmon/hwmon1
lrwxrwxrwx  1 root root 0 May 29 11:02 hwmon2 -> ../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/hwmon/hwmon2
lrwxrwxrwx  1 root root 0 May 29 11:02 hwmon3 -> ../../devices/platform/coretemp.0/hwmon/hwmon3
lrwxrwxrwx  1 root root 0 May 29 11:02 hwmon4 -> ../../devices/virtual/thermal/thermal_zone2/hwmon4
lrwxrwxrwx  1 root root 0 May 29 11:02 hwmon5 -> ../../devices/virtual/thermal/thermal_zone4/hwmon5
rpb@nuc8i7hvk01:~/data/passwords$ sensors
pch_skylake-virtual-0
Adapter: Virtual device
temp1:        +55.5°C

amdgpu-pci-0100
Adapter: PCI adapter
vddgfx:      800.00 mV
edge:         +55.0°C  (crit = +89.0°C, hyst = -273.1°C)
power1:        9.04 W  (cap =  75.00 W)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +27.8°C  (crit = +119.0°C)
temp2:        +29.8°C  (crit = +119.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1:        +46.0°C

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +64.0°C  (high = +100.0°C, crit = +100.0°C)
Core 0:        +60.0°C  (high = +100.0°C, crit = +100.0°C)
Core 1:        +61.0°C  (high = +100.0°C, crit = +100.0°C)
Core 2:        +60.0°C  (high = +100.0°C, crit = +100.0°C)
Core 3:        +64.0°C  (high = +100.0°C, crit = +100.0°C)

nvme-pci-7200
Adapter: PCI adapter
Composite:    +42.9°C  (low  = -273.1°C, high = +80.8°C)
                       (crit = +80.8°C)
Sensor 1:     +42.9°C  (low  = -273.1°C, high = +65261.8°C)
Sensor 2:     +42.9°C  (low  = -273.1°C, high = +65261.8°C)

> In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
> rx/tx data packets. This feature is very important in many cases. So
> this patch allows AF_PACKET to obtain xsk packages.

You can use xdpdump to dump the packets from the XDP program before it gets redirected into the XSK: https://github.com/xdp-project/xdp-tools/tree/master/xdp-dump

Doens't currently work on egress, but if/when we get a proper TX hook that should be doable as well.

Wiring up XSK to AF_PACKET sounds a bit nonsensical: XSK is already a transport to userspace, why would you need a second one?

Yes, it is rather cool (credit to Eelco). Notice the extra info you can capture from 'exit', like XDP return codes, if_index, rx_queue. The tool uses the perf ring-buffer to send/copy data to userspace. This is actually surprisingly fast, but I still think AF_XDP will be faster (but it usually 'steals' the packet).

Another (crazy?) idea is to extend this (and xdpdump), is to leverage Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend xdp_redirect_map with broadcast support"). We now have a xdp_redirect_map flag BPF_F_BROADCAST, what if we create a BPF_F_CLONE_PASS flag?

The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the packet for the specified map target index (e.g AF_XDP map), but afterwards it does like veth/cpumap and creates an SKB from the xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack. (Feel free to kick me if this doesn't make any sense)

> This would be a smooth way to implement clone support for AF_XDP. If
> we had this and someone added AF_XDP support to libpcap, we could both
> capture AF_XDP traffic with tcpdump (using this clone functionality in
> the XDP program) and speed up tcpdump for dumping traffic destined for
> regular sockets. Would that solve your use case Xuan? Note that I have
> not looked into the BPF_F_CLONE_PASS code, so do not know at this
> point what it would take to support this for XSKMAPs.

Recently also ended up with something similar for our XDP LB to record pcaps [0] My question is.. tcpdump doesn't really care where the packet data comes from, so why not extending libpcap's Linux-related internals to either capture from perf RB or BPF ringbuf rather than AF_PACKET sockets? Cloning is slow, and if you need to end up creating an skb which is then cloned once again inside AF_PACKET it's even worse. Just relying and reading out, say, perf RB you don't need any clones at all.

Anyway, xdpdump does have a "pipe pcap to stdout" feature so you can do `xdpdump | tcpdump` and get the interactive output; and it will also save pcap information to disk, of course (using pcap-ng so it can also save metadata like XDP program name and return code).

From a bug report I learned about a tool called evtest: listens for and documents events from devices such as keyboards:

 evtest
No device specified, trying to scan all of /dev/input/event*
Available devices:
/dev/input/event0:      AT Translated Set 2 keyboard
/dev/input/event1:      Video Bus
/dev/input/event2:      PS/2 Generic Mouse
/dev/input/event3:      Apple, Inc Apple Keyboard
/dev/input/event4:      Apple, Inc Apple Keyboard
/dev/input/event5:      PixArt USB Optical Mouse
[...]
Select the device event number [0-24]: 3
Input driver version is 1.0.1
Input device ID: bus 0x3 vendor 0x5ac product 0x221 version 0x111
Input device name: "Apple, Inc Apple Keyboard"
[...]
Testing ... (interrupt to exit)
Event: time 1622014586.354418, type 1 (EV_KEY), code 28 (KEY_ENTER), value 0
Event: time 1622014586.354418, -------------- SYN_REPORT ------------
Event: time 1622014588.314431, type 1 (EV_KEY), code 59 (KEY_F1), value 1
Event: time 1622014588.314431, -------------- SYN_REPORT ------------
éEvent: time 1622014588.402405, type 1 (EV_KEY), code 59 (KEY_F1), value 0
....

The successor package is evemu-tools:

• evtest - utility to monitor Linux input device events
• evemu-tools - Linux Input Event Device Emulation Library - test tools

Guide: Installation of *VANILLA* Debian 11 (or later) on a Raspberry Pi 4 - check Debian Bug#985956