- dynamic dns:
- Dynamic DNS on your own domain: the important bit for me was the mention of using the "ddclient" client package for the client side. Now to search for something for the server side.
- Route-based VPN on Linux with WireGuard: I think I referenced this somewhere already, but I will do it again. And quoting from the article: "As an alternative to IPsec, WireGuard is an extremely simple (less than 5,000 lines of code) yet fast and modern VPN that utilizes state-of-the-art and opinionated cryptography (Curve25519, ChaCha20, Poly1305) and whose protocol, based on Noise, has been formally verified. It is currently available as an out-of-tree module for Linux but is likely to be merged when the protocol is not subject to change anymore. Compared to IPsec, its major weakness is its lack of interoperability."
- DKIM (DomainKeys Identified Mail) Modernization : notes on using Domain Keys in a Debian Stretch setting
- Reservoir sampling with few random bits: an excellent article describing the mechanism of randomly sampling from a stream: hold and choose, hold and choose, ...
- Using the switch on Turris Omnia with Debian: I have seen the Turris Omnia mentioned elsewhere, and this is an interesting reference on one mechansim of invoking hardware forwarding in the built in switch.
Wednesday, March 28. 2018
Random Bits
Linux CPU Frequency Adjustments
I have a Sager NP8157 laptop which came initially with Windows installed. I swapped out Windows and installed Linux. In this configuration, the fans seem to be running harder than I think they might with Windows. I havn't really tested that theory. But, anyway, the fans seem to be loud. This is an experiment to see if fan utilization can be reduced through a reduction in CPU use.
2018/08/01 - found out about the package laptop-mode-tools package. It has a good number of tools and accessories for power mode management and monitoring. powertop is another package which bears close examination for power management monitoring.
My version (Debian Stretch Backports Kernel is installed):
# uname -a Linux np8157 4.14.0-0.bpo.3-amd64 #1 SMP Debian 4.14.13-1~bpo9+1 (2018-01-14) x86_64 GNU/Linux
A way to find out how many processors are available:
# cat /proc/cpuinfo | grep processor | wc -l 8
Some Debian based CPU specific packages:
cpufreqd/stable 2.4.2-2+b1 amd64 fully configurable daemon for dynamic frequency and voltage scaling cpufrequtils/stable 008-1+b1 amd64 utilities to deal with the cpufreq Linux kernel feature cpuid/stable 20161201-1 amd64 tool to dump x86 CPUID information about the CPU(s) cpulimit/stable 2.4-1 amd64 tool for limiting the CPU usage of a process cpuset/stable 1.5.6-5 all Allows manipluation of cpusets and provides higher level fun cpustat/stable 0.02.01-1 amd64 periodic cpu utilization statistics cputool/stable 0.0.8-2+b1 amd64 Utility which manages CPU usage and system load
A simplified version of what would be found with 'cat /proc/cpuinfo':. Note here thatI have one physical processor socket with four cores, with each core having two threads, which falls under the Intel definition of hyper-=threading, with a result of 8 CPUs.
# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 158 Model name: Intel(R) Core(TM) i7-7820HK CPU @ 2.90GHz Stepping: 9 CPU MHz: 1109.773 CPU max MHz: 3900.0000 CPU min MHz: 800.0000 BogoMIPS: 5808.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
To see which are on-line and which are off-line:
To watch CPU utilization:root@np8157:~# cat /sys/devices//system/cpu/online 0-7 root@np8157:~# cat /sys/devices//system/cpu/offline
# watch grep \"cpu MHz\" /proc/cpuinfo Every 2.0s: grep "cpu MHz" /proc/cpuinfo np8157: Tue Mar 27 23:03:25 2018 cpu MHz : 1100.008 cpu MHz : 1100.077 cpu MHz : 1100.022 cpu MHz : 1100.093 cpu MHz : 1100.027 cpu MHz : 1100.053 cpu MHz : 1100.138 cpu MHz : 1100.013
A summary of the how the cpu is controlled. In this case, the govenor is the 'intel_pstate' controller using what I think is the 'powersave' governor.
# cpupower frequency-info analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: Cannot determine or is not supported. hardware limits: 800 MHz - 3.90 GHz available cpufreq governors: performance powersave current policy: frequency should be within 800 MHz and 3.90 GHz. The governor "powersave" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 1.10 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes
To turn off half of my cpu's (CPU 0 must always be enabled):
# for i in /sys/devices/system/cpu/cpu{1,3,5,7}/online ; do echo 0 > $i ; done # cat /sys/devices//system/cpu/online 0,2,4,6 # cat /sys/devices//system/cpu/offline 1,3,5,7
And turn them back on again:
# for i in /sys/devices/system/cpu/cpu{1,2,3,4,5,6,7}/online ; do echo 1 > $i ; done # cat /sys/devices//system/cpu/online 0-7
I didn't let things run very long, but my CPU temperatures didn't vary that much. Probably because everything is on the same die.
To apply the limitation on boot-up, modify 'etc/default/grub' with the something similar to the following, where n is in the interval 0 to the number of cpus present, and then run 'update-grub':
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash maxcpus=n"
- Bibliography
- How to disable CPU core(s) in a multicore machine?: provided the bash loop to perform multiple changes, as well as the GRUB line change
- :
- How to make use of Dynamic Frequency Scaling: scaling parameters for very old CPUs and kernels.
- CPU frequency and voltage scaling statistics in the Linux(TM) kernel: L i n u x c p u f r e q - s t a t s d r i v e r - information for users -
- CPU frequency and voltage scaling code in the Linux(TM) kernel: Clock scaling allows you to change the clock speed of the CPUs on the fly. This is a nice method to save battery power, because the lower the clock speed, the less power the CPU consumes.
- CPU Performance Scaling: from the kernel 4.14 admin guide
- intel_pstate CPU Performance Scaling Driver: Also from the same guide, for the governor which is used in recent Skylake CPUs. There is the possibility of setting "Energy vs Performance Hints", which is good for more investigation.
- Debian How To CpuFrequencyScaling: The important note here is that "cpufreq is being replaced by cpupower". Other web sites refer to cpufreq, which is now deprecated.
- CPU frequency scaling: The ArchLinux wiki on scaling. This article talks about the cpupower command along with its options. This is where I also saw the watch command.
- Programmatically disable CPU core: makes a reference to https://github.com/torvalds/linux/blob/master/drivers/cpufreq/intel_pstate.c code and how it operates.
- :
Saturday, March 24. 2018
SystemCtl and JournalCtl
An extract from something in the past:
Use
systemctl status .serviceafterwards to ensure that the service is both up, running, and running correctly, or use
journalctl --followin another term window.
If you have bash_completion on your distro, there is tab completion of every switch and every service that is actually installed. systemctl man systemd.index
gives an overview of every systemd-related man page, and man systemd.directives gives an index over (almost) all commands, config files, directives etc.
Change
I clipped this back on 2017/12/07. I didn't save the attribution.
How to change?
Often the status quo is not an acceptable course of action. The environment is changing around us and we need to make not just incremental improvements over what we are doing, but rather to rethink, reinvent, and reengineer what our strategies should be. The conventional frameworks start the process of reflection on our past, and that is often the wrong way to make you think of change. We want to create discontinuities not to reaffirm what we have done. By providing too much emphasis on competitor’s behavior, the conventional frameworks tend to anchor us in the existing industry practices.
We have found that the obvious answer to initiate a process of change is to start with the customer. What can we do to help the customer improve its business? How can we look into its full life cycle and detect novel ways of providing something that is truly unique? By focusing on the customer, we find it is much easier to detect opportunities to be truly unique and promote change.
How to pursue profitable growth?
Profits are revenues minus expenses. It does not require much creativity to reduce costs; there are always ways to curtail our expenditures. What is more demanding is to find ways of increasing our revenues profitably. Are there a lot of potential revenues and economic growth which are left under the table because of our inability to detect those opportunities? How could we assess the potential for growth?
One practice that is commonly used is sales forecast. I find that completely unacceptable – first, because sales should not be a subject of forecast. We forecast events over which we have no control, such as the weather. On the contrary, sales are something that we truly can and should influence. Second, sales forecast means an exercise of extrapolation of the past into the future. Again, this is unacceptable if we want to challenge our previous performance and if we want to engage in new forms of management that, hopefully, can make our past irrelevant – for the better.
Where is the answer? Again this resides on the customer. If we were to look at each customer individually, we will detect what potential exists by treating them differently, meaning offering more creative value propositions. This will allow us to examine “white spaces” – untapped opportunities – from the bottom up to come up with a much more effective growth strategy than the one resulting from competitive analysis.
How to spark creativity?
What we are saying, in various ways, is that we need to find mechanisms that allow us to be more creative, the so-called thinking out of the box. When I was using the conventional frameworks, I rarely found them to be the source of inspiration conducive to creating an innovative strategic environment. It might be my limitation while using those frameworks, but I doubt it. The emphasis on creating “competitive advantage” was focusing on the competitors. That is not the best way to do it. Instead, attempting to understand the granular needs of the customer and to seek the best ways to satisfy them provides the answer.
Sorting Ipv4 Addresses on the Command Line
From a mailing list:
>Does anyone have a script that sorts IPv4 addresses? >I've tried all sorts of tricks with "sort -n -k" etc, but can't seem to get it right. With an older sort: sort -t'.' -n -k1,1 -k2,2 -k3,3 -k4,4 With modern GNU sort, a "version syntax" sort is available: sort -V
And something else:
the canonical "sort | uniq -c | sort -nr" so beloved of admins everywhere
Friday, March 16. 2018
Linux Network Commands
# netstat -rn -f inet Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.7.28.23 0.0.0.0 UG 0 0 0 vlan28 10.7.1.1 10.7.3.7 255.255.255.255 UGH 0 0 0 enp14s0f2 10.7.1.2 10.7.3.4 255.255.255.255 UGH 0 0 0 enp14s0f3 10.7.1.33 10.7.28.23 255.255.255.255 UGH 0 0 0 vlan28 10.7.1.34 10.7.28.23 255.255.255.255 UGH 0 0 0 vlan28 10.7.1.41 10.7.3.7 255.255.255.255 UGH 0 0 0 enp14s0f2 .....
# ls -l /sys/class/net total 0 lrwxrwxrwx 1 root root 0 Mar 8 04:06 bond0 -> ../../devices/virtual/net/bond0 -rw-r--r-- 1 root root 4096 Mar 8 04:05 bonding_masters lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f0 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.0/net/enp12s0f0 lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f1 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.1/net/enp12s0f1 lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f2 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.2/net/enp12s0f2 lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f3 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.3/net/enp12s0f3 lrwxrwxrwx 1 root root 0 Mar 8 04:05 enp14s0f0 -> ../../devices/pci0000:00/0000:00:03.2/0000:0e:00.0/net/enp14s0f0 lrwxrwxrwx 1 root root 0 Mar 8 04:05 lo -> ../../devices/virtual/net/lo lrwxrwxrwx 1 root root 0 Mar 8 04:05 ovsbr0 -> ../../devices/virtual/net/ovsbr0 lrwxrwxrwx 1 root root 0 Mar 8 04:06 ovs-system -> ../../devices/virtual/net/ovs-system lrwxrwxrwx 1 root root 0 Mar 8 04:05 vlan19 -> ../../devices/virtual/net/vlan19
# cat /sys/class/net/enp8s0f0/statistics/tx_bytes 308623980
# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.9.0.0 10.9.2.32 255.255.255.0 UG 0 0 0 enp1s0 10.9.1.1 10.9.2.32 255.255.255.255 UGH 0 0 0 enp1s0
ss -ntl
# nstat -a #kernel IpInReceives 156764 0.0 IpInDelivers 156764 0.0 IpOutRequests 151083 0.0 IpOutNoRoutes 40 0.0 .....
- netstat --statistics
- netstat -s -- protocol stats
- netstat -r -- routing info from kernel
- netstat -g -- mulicast info
- netstat -i -- stats
- netstat -ic -- monitor continuously
- netstat -e -- extra information
- netstat -o -- timer related information
- netstat -p -- associated PID
- netstat -l -- listening sockets
- ss --summary
Useful combinations:
- netstat -antp
- netstat -lntp
- nload - console based histogram
- iftop - console based numeric traffic values
- iptraf - An IP traffic monitor that shows information on the IP traffic passing over your network.
- iptraf-ng - is a fork of the original project
- dstat - Combines vmstat, iostat, ifstat, netstat information and more
- sar
- Diagnostics
- strace -e open nstat 2>&1 > /dev/null|grep /proc
- strace -e open netstat -s 2>&1 > /dev/null|grep /proc
Saturday, March 10. 2018
PCI Devices on a Sager/Clevo NP8157
# lspci -knn 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:5910] (rev 05) Subsystem: CLEVO/KAPOK Computer Device [1558:65a1] 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 05) Kernel driver in use: pcieport Kernel modules: shpchp 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31) Subsystem: CLEVO/KAPOK Computer Sunrise Point-H USB 3.0 xHCI Controller [1558:65a1] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31) Subsystem: CLEVO/KAPOK Computer Sunrise Point-H Thermal subsystem [1558:65a1] Kernel driver in use: intel_pch_thermal Kernel modules: intel_pch_thermal 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31) Kernel driver in use: mei_me Kernel modules: mei_me 00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA Controller [AHCI mode] [8086:a103] (rev 31) Subsystem: CLEVO/KAPOK Computer Sunrise Point-H SATA Controller [AHCI mode] [1558:65a1] Kernel driver in use: ahci Kernel modules: ahci 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.6 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #7 [8086:a116] (rev f1) Kernel driver in use: pcieport Kernel modules: shpchp 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1) Kernel driver in use: pcieport Kernel modules: shpchp 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a152] (rev 31) Subsystem: CLEVO/KAPOK Computer Sunrise Point-H LPC Controller [1558:65a1] 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31) Subsystem: CLEVO/KAPOK Computer Sunrise Point-H PMC [1558:65a1] 00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a171] (rev 31) Subsystem: CLEVO/KAPOK Computer Device [1558:65a2] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31) Subsystem: CLEVO/KAPOK Computer Sunrise Point-H SMBus [1558:65a1] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104M [GeForce GTX 1070] [10de:1be1] (rev a1) Subsystem: CLEVO/KAPOK Computer GP104M [GeForce GTX 1070] [1558:65a4] Kernel driver in use: nvidia Kernel modules: nvidia 6d:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTL8411B PCI Express Card Reader [10ec:5287] (rev 01) Subsystem: CLEVO/KAPOK Computer RTL8411B PCI Express Card Reader [1558:65a1] Kernel driver in use: rtsx_pci Kernel modules: rtsx_pci 6d:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 12) Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1558:65a1] Kernel driver in use: r8169 Kernel modules: r8169 6e:00.0 Network controller [0280]: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter [168c:003e] (rev 32) Subsystem: Bigfoot Networks, Inc. QCA6174 802.11ac Wireless Network Adapter [1a56:1535] Kernel driver in use: ath10k_pci Kernel modules: ath10k_pci 6f:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804] Subsystem: Samsung Electronics Co Ltd Device [144d:a801] Kernel driver in use: nvme Kernel modules: nvme
Bonus section, identifying a computer:
$ sudo grep . /sys/devices/virtual/dmi/id/{bios,board,product}* /sys/devices/virtual/dmi/id/bios_date:03/09/2017 /sys/devices/virtual/dmi/id/bios_vendor:American Megatrends Inc. /sys/devices/virtual/dmi/id/bios_version:1.05.04RLS2 /sys/devices/virtual/dmi/id/board_asset_tag:Tag 12345 /sys/devices/virtual/dmi/id/board_name:P65_67HSHP /sys/devices/virtual/dmi/id/board_serial:Not Applicable /sys/devices/virtual/dmi/id/board_vendor:Notebook /sys/devices/virtual/dmi/id/board_version:Not Applicable /sys/devices/virtual/dmi/id/product_family:Not Applicable /sys/devices/virtual/dmi/id/product_name:P65_67HSHP /sys/devices/virtual/dmi/id/product_serial:Not Applicable /sys/devices/virtual/dmi/id/product_uuid:435BFA80-DBC6-0000-0000-000000000000 /sys/devices/virtual/dmi/id/product_version:Not Applicable
Bonus section, #2, using dmi information (ignore the grep for all the raw info):
# dmidecode | grep -A3 '^System Information' System Information Manufacturer: Notebook Product Name: P65_67HSHP Version: Not Applicable
Saturday, March 3. 2018
Extract Pictures from PDF File
On a Linux system, a program called 'pdfimages' is useful for extracting images from a .pdf file.
On Debian, the application is part of the poppler-utils package which has the following applications:
- pdfdetach -- lists or extracts embedded files (attachments)
- pdffonts -- font analyzer
- pdfimages -- image extractor
- pdfinfo -- document information
- pdfseparate -- page extraction tool
- pdfsig -- verifies digital signatures
- pdftocairo -- PDF to PNG/JPEG/PDF/PS/EPS/SVG converter using Cairo
- pdftohtml -- PDF to HTML converter
- pdftoppm -- PDF to PPM/PNG/JPEG image converter
- pdftops -- PDF to PostScript (PS) converter
- pdftotext -- text extraction
- pdfunite -- document merging tool
To see what sort of images are in a .pdf, run the -list command:
$ pdfimages -list /home/rpb/data/resources/books/The.Stock.Market.Course.pdf page num type width height color comp bpc enc interp object ID x-ppi y-ppi size ratio -------------------------------------------------------------------------------------------- 45 2 stencil 571 439 - 1 1 ccitt no 133 0 151 151 1718B 5.5% 85 3 image 398 267 rgb 3 8 jpeg no 253 0 86 86 22.6K 7.3%
A number of options, with '-all' being the easiest, are available to extract images, and to perform optional conversions:
$ pdfimages pdfimages version 0.48.0 Copyright 2005-2016 The Poppler Developers - http://poppler.freedesktop.org Copyright 1996-2011 Glyph & Cog, LLC Usage: pdfimages [options]-f : first page to convert -l : last page to convert -png : change the default output format to PNG -tiff : change the default output format to TIFF -j : write JPEG images as JPEG files -jp2 : write JPEG2000 images as JP2 files -jbig2 : write JBIG2 images as JBIG2 files -ccitt : write CCITT images as CCITT files -all : equivalent to -png -tiff -j -jp2 -jbig2 -ccitt -list : print list of images instead of saving -opw : owner password (for encrypted files) -upw : user password (for encrypted files) -p : include page numbers in output file names -q : don't print any messages or errors -v : print copyright and version info -h : print usage information -help : print usage information --help : print usage information -? : print usage information