- dynamic dns:
- Dynamic DNS on your own domain: the important bit for me was the mention of using the "ddclient" client package for the client side. Now to search for something for the server side.
- Route-based VPN on Linux with WireGuard: I think I referenced this somewhere already, but I will do it again. And quoting from the article: "As an alternative to IPsec, WireGuard is an extremely simple (less than 5,000 lines of code) yet fast and modern VPN that utilizes state-of-the-art and opinionated cryptography (Curve25519, ChaCha20, Poly1305) and whose protocol, based on Noise, has been formally verified. It is currently available as an out-of-tree module for Linux but is likely to be merged when the protocol is not subject to change anymore. Compared to IPsec, its major weakness is its lack of interoperability."
- DKIM (DomainKeys Identified Mail) Modernization : notes on using Domain Keys in a Debian Stretch setting
- Reservoir sampling with few random bits: an excellent article describing the mechanism of randomly sampling from a stream: hold and choose, hold and choose, ...
- Using the switch on Turris Omnia with Debian: I have seen the Turris Omnia mentioned elsewhere, and this is an interesting reference on one mechansim of invoking hardware forwarding in the built in switch.
Wednesday, March 28. 2018
Random Bits
Linux CPU Frequency Adjustments
I have a Sager NP8157 laptop which came initially with Windows installed. I swapped out Windows and installed Linux. In this configuration, the fans seem to be running harder than I think they might with Windows. I havn't really tested that theory. But, anyway, the fans seem to be loud. This is an experiment to see if fan utilization can be reduced through a reduction in CPU use.
2018/08/01 - found out about the package laptop-mode-tools package. It has a good number of tools and accessories for power mode management and monitoring. powertop is another package which bears close examination for power management monitoring.
My version (Debian Stretch Backports Kernel is installed):
# uname -a Linux np8157 4.14.0-0.bpo.3-amd64 #1 SMP Debian 4.14.13-1~bpo9+1 (2018-01-14) x86_64 GNU/Linux
A way to find out how many processors are available:
# cat /proc/cpuinfo | grep processor | wc -l 8
Some Debian based CPU specific packages:
cpufreqd/stable 2.4.2-2+b1 amd64 fully configurable daemon for dynamic frequency and voltage scaling cpufrequtils/stable 008-1+b1 amd64 utilities to deal with the cpufreq Linux kernel feature cpuid/stable 20161201-1 amd64 tool to dump x86 CPUID information about the CPU(s) cpulimit/stable 2.4-1 amd64 tool for limiting the CPU usage of a process cpuset/stable 1.5.6-5 all Allows manipluation of cpusets and provides higher level fun cpustat/stable 0.02.01-1 amd64 periodic cpu utilization statistics cputool/stable 0.0.8-2+b1 amd64 Utility which manages CPU usage and system load
A simplified version of what would be found with 'cat /proc/cpuinfo':. Note here thatI have one physical processor socket with four cores, with each core having two threads, which falls under the Intel definition of hyper-=threading, with a result of 8 CPUs.
# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 158 Model name: Intel(R) Core(TM) i7-7820HK CPU @ 2.90GHz Stepping: 9 CPU MHz: 1109.773 CPU max MHz: 3900.0000 CPU min MHz: 800.0000 BogoMIPS: 5808.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
To see which are on-line and which are off-line:
To watch CPU utilization:root@np8157:~# cat /sys/devices//system/cpu/online 0-7 root@np8157:~# cat /sys/devices//system/cpu/offline
# watch grep \"cpu MHz\" /proc/cpuinfo Every 2.0s: grep "cpu MHz" /proc/cpuinfo np8157: Tue Mar 27 23:03:25 2018 cpu MHz : 1100.008 cpu MHz : 1100.077 cpu MHz : 1100.022 cpu MHz : 1100.093 cpu MHz : 1100.027 cpu MHz : 1100.053 cpu MHz : 1100.138 cpu MHz : 1100.013
A summary of the how the cpu is controlled. In this case, the govenor is the 'intel_pstate' controller using what I think is the 'powersave' governor.
# cpupower frequency-info analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: Cannot determine or is not supported. hardware limits: 800 MHz - 3.90 GHz available cpufreq governors: performance powersave current policy: frequency should be within 800 MHz and 3.90 GHz. The governor "powersave" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 1.10 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes
To turn off half of my cpu's (CPU 0 must always be enabled):
# for i in /sys/devices/system/cpu/cpu{1,3,5,7}/online ; do echo 0 > $i ; done # cat /sys/devices//system/cpu/online 0,2,4,6 # cat /sys/devices//system/cpu/offline 1,3,5,7
And turn them back on again:
# for i in /sys/devices/system/cpu/cpu{1,2,3,4,5,6,7}/online ; do echo 1 > $i ; done # cat /sys/devices//system/cpu/online 0-7
I didn't let things run very long, but my CPU temperatures didn't vary that much. Probably because everything is on the same die.
To apply the limitation on boot-up, modify 'etc/default/grub' with the something similar to the following, where n is in the interval 0 to the number of cpus present, and then run 'update-grub':
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash maxcpus=n"
- Bibliography
- How to disable CPU core(s) in a multicore machine?: provided the bash loop to perform multiple changes, as well as the GRUB line change
- :
- How to make use of Dynamic Frequency Scaling: scaling parameters for very old CPUs and kernels.
- CPU frequency and voltage scaling statistics in the Linux(TM) kernel: L i n u x c p u f r e q - s t a t s d r i v e r - information for users -
- CPU frequency and voltage scaling code in the Linux(TM) kernel: Clock scaling allows you to change the clock speed of the CPUs on the fly. This is a nice method to save battery power, because the lower the clock speed, the less power the CPU consumes.
- CPU Performance Scaling: from the kernel 4.14 admin guide
- intel_pstate CPU Performance Scaling Driver: Also from the same guide, for the governor which is used in recent Skylake CPUs. There is the possibility of setting "Energy vs Performance Hints", which is good for more investigation.
- Debian How To CpuFrequencyScaling: The important note here is that "cpufreq is being replaced by cpupower". Other web sites refer to cpufreq, which is now deprecated.
- CPU frequency scaling: The ArchLinux wiki on scaling. This article talks about the cpupower command along with its options. This is where I also saw the watch command.
- Programmatically disable CPU core: makes a reference to https://github.com/torvalds/linux/blob/master/drivers/cpufreq/intel_pstate.c code and how it operates.
- :
Saturday, March 24. 2018
SystemCtl and JournalCtl
An extract from something in the past:
Use
systemctl status .serviceafterwards to ensure that the service is both up, running, and running correctly, or use
journalctl --followin another term window.
If you have bash_completion on your distro, there is tab completion of every switch and every service that is actually installed. systemctl man systemd.index
gives an overview of every systemd-related man page, and man systemd.directives gives an index over (almost) all commands, config files, directives etc.
Sorting Ipv4 Addresses on the Command Line
From a mailing list:
>Does anyone have a script that sorts IPv4 addresses? >I've tried all sorts of tricks with "sort -n -k" etc, but can't seem to get it right. With an older sort: sort -t'.' -n -k1,1 -k2,2 -k3,3 -k4,4 With modern GNU sort, a "version syntax" sort is available: sort -V
And something else:
the canonical "sort | uniq -c | sort -nr" so beloved of admins everywhere
Friday, March 16. 2018
Linux Network Commands
# netstat -rn -f inet Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.7.28.23 0.0.0.0 UG 0 0 0 vlan28 10.7.1.1 10.7.3.7 255.255.255.255 UGH 0 0 0 enp14s0f2 10.7.1.2 10.7.3.4 255.255.255.255 UGH 0 0 0 enp14s0f3 10.7.1.33 10.7.28.23 255.255.255.255 UGH 0 0 0 vlan28 10.7.1.34 10.7.28.23 255.255.255.255 UGH 0 0 0 vlan28 10.7.1.41 10.7.3.7 255.255.255.255 UGH 0 0 0 enp14s0f2 .....
# ls -l /sys/class/net total 0 lrwxrwxrwx 1 root root 0 Mar 8 04:06 bond0 -> ../../devices/virtual/net/bond0 -rw-r--r-- 1 root root 4096 Mar 8 04:05 bonding_masters lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f0 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.0/net/enp12s0f0 lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f1 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.1/net/enp12s0f1 lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f2 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.2/net/enp12s0f2 lrwxrwxrwx 1 root root 0 Mar 8 04:06 enp12s0f3 -> ../../devices/pci0000:00/0000:00:03.1/0000:0c:00.3/net/enp12s0f3 lrwxrwxrwx 1 root root 0 Mar 8 04:05 enp14s0f0 -> ../../devices/pci0000:00/0000:00:03.2/0000:0e:00.0/net/enp14s0f0 lrwxrwxrwx 1 root root 0 Mar 8 04:05 lo -> ../../devices/virtual/net/lo lrwxrwxrwx 1 root root 0 Mar 8 04:05 ovsbr0 -> ../../devices/virtual/net/ovsbr0 lrwxrwxrwx 1 root root 0 Mar 8 04:06 ovs-system -> ../../devices/virtual/net/ovs-system lrwxrwxrwx 1 root root 0 Mar 8 04:05 vlan19 -> ../../devices/virtual/net/vlan19
# cat /sys/class/net/enp8s0f0/statistics/tx_bytes 308623980
# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.9.0.0 10.9.2.32 255.255.255.0 UG 0 0 0 enp1s0 10.9.1.1 10.9.2.32 255.255.255.255 UGH 0 0 0 enp1s0
ss -ntl
# nstat -a #kernel IpInReceives 156764 0.0 IpInDelivers 156764 0.0 IpOutRequests 151083 0.0 IpOutNoRoutes 40 0.0 .....
- netstat --statistics
- netstat -s -- protocol stats
- netstat -r -- routing info from kernel
- netstat -g -- mulicast info
- netstat -i -- stats
- netstat -ic -- monitor continuously
- netstat -e -- extra information
- netstat -o -- timer related information
- netstat -p -- associated PID
- netstat -l -- listening sockets
- ss --summary
Useful combinations:
- netstat -antp
- netstat -lntp
- nload - console based histogram
- iftop - console based numeric traffic values
- iptraf - An IP traffic monitor that shows information on the IP traffic passing over your network.
- iptraf-ng - is a fork of the original project
- dstat - Combines vmstat, iostat, ifstat, netstat information and more
- sar
- Diagnostics
- strace -e open nstat 2>&1 > /dev/null|grep /proc
- strace -e open netstat -s 2>&1 > /dev/null|grep /proc
Saturday, March 3. 2018
Extract Pictures from PDF File
On a Linux system, a program called 'pdfimages' is useful for extracting images from a .pdf file.
On Debian, the application is part of the poppler-utils package which has the following applications:
- pdfdetach -- lists or extracts embedded files (attachments)
- pdffonts -- font analyzer
- pdfimages -- image extractor
- pdfinfo -- document information
- pdfseparate -- page extraction tool
- pdfsig -- verifies digital signatures
- pdftocairo -- PDF to PNG/JPEG/PDF/PS/EPS/SVG converter using Cairo
- pdftohtml -- PDF to HTML converter
- pdftoppm -- PDF to PPM/PNG/JPEG image converter
- pdftops -- PDF to PostScript (PS) converter
- pdftotext -- text extraction
- pdfunite -- document merging tool
To see what sort of images are in a .pdf, run the -list command:
$ pdfimages -list /home/rpb/data/resources/books/The.Stock.Market.Course.pdf page num type width height color comp bpc enc interp object ID x-ppi y-ppi size ratio -------------------------------------------------------------------------------------------- 45 2 stencil 571 439 - 1 1 ccitt no 133 0 151 151 1718B 5.5% 85 3 image 398 267 rgb 3 8 jpeg no 253 0 86 86 22.6K 7.3%
A number of options, with '-all' being the easiest, are available to extract images, and to perform optional conversions:
$ pdfimages pdfimages version 0.48.0 Copyright 2005-2016 The Poppler Developers - http://poppler.freedesktop.org Copyright 1996-2011 Glyph & Cog, LLC Usage: pdfimages [options]-f : first page to convert -l : last page to convert -png : change the default output format to PNG -tiff : change the default output format to TIFF -j : write JPEG images as JPEG files -jp2 : write JPEG2000 images as JP2 files -jbig2 : write JBIG2 images as JBIG2 files -ccitt : write CCITT images as CCITT files -all : equivalent to -png -tiff -j -jp2 -jbig2 -ccitt -list : print list of images instead of saving -opw : owner password (for encrypted files) -upw : user password (for encrypted files) -p : include page numbers in output file names -q : don't print any messages or errors -v : print copyright and version info -h : print usage information -help : print usage information --help : print usage information -? : print usage information