I am troubleshooting a computer which seems to have slow drive access, even under seemingly lightly loaded conditions. Running a package manager in an LXC container is the way to simulate/stimulate the issue for troubleshooting.
vmstat shows a short form summary of basic activity. The '1' tells it to run once a second:
# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 2904884 1060 8450140 0 0 10 22 1 1 0 0 99 0 0 1 0 0 2905008 1060 8450140 0 0 0 0 1720 2308 0 1 99 0 0 1 0 0 2900848 1060 8450156 0 0 0 0 3306 4287 2 2 97 0 0 1 0 0 2902240 1060 8450192 0 0 0 0 1724 2065 0 1 99 0 0 1 0 0 2903388 1060 8450188 0 0 0 0 2514 3030 1 1 98 0 0 2 0 0 2903784 1060 8450184 0 0 0 0 1589 2046 1 0 99 0 0
Pidstat can be run to capture over, for example, 10 seconds, to determine context switches:
# pidstat -w 10 1 Linux 4.14.0-0.bpo.3-amd64 (host01.ny1) 05/10/2018 _x86_64_ (16 CPU) ....
A variation of pidstat shows threads:
# pidstat -wt Linux 4.14.0-0.bpo.3-amd64 (host01.ny1) 05/10/2018 _x86_64_ (16 CPU) 01:55:29 PM UID TGID TID cswch/s nvcswch/s Command 01:55:29 PM 0 1 - 0.20 0.00 systemd 01:55:29 PM 0 - 1 0.20 0.00 |__systemd 01:55:29 PM 0 2 - 0.02 0.00 kthreadd 01:55:29 PM 0 - 2 0.02 0.00 |__kthreadd 01:55:29 PM 0 4 - 0.00 0.00 kworker/0:0H 01:55:29 PM 0 - 4 0.00 0.00 |__kworker/0:0H 01:55:29 PM 0 7 - 0.00 0.00 mm_percpu_wq 01:55:29 PM 0 - 7 0.00 0.00 |__mm_percpu_wq
Or with a sort:
pidstat -wt | sort -n -k6
The 'watch' command is a powerful command as it can be used to show deltas between screen refreshes. Here is an example format to watch /proc/interrupts:
# watch -tdn1 cat /proc/interrupts
By installing some performance monitoring tools, additional detail is possible:
# apt install linux-perf netperf
Then a simple capture example:
# capture events perf record -e context-switches -a # record the events #or with symbol info: perf record -e context-switches -a -g # after capture perf report
/proc has good per-pid status capability:
$ pid=500 $ grep ctxt /proc/$pid/status voluntary_ctxt_switches: 41 nonvoluntary_ctxt_switches: 16
To tune irq assignments, look to see the IRQs in use with
cat /proc/interrupts
"number in smp_affinity is a bitmap of cpus, represented in hex code. The rightmost bit is the least significant"
cpu_7 cpu_6 cpu_5 cpu_4 cpu_3 cpu_2 cpu_1 cpu_0 0 0 0 1 1 0 1 0 = 0001 1010 = 1a (in hex)
Then set the affinity:
echo 1a > /proc/irq/28/smp_affinity