Some articles on how to trace what is happening in the kernel:
- It took me a while, but I found (again) an older talk to Chandler Carruth's Tuning C++: Benchmarks, and CPUs, and Compilers! Oh My! at CppCon 2015. His talk is over an hour, but covers many different topics on tracing and tuning. He likes 'perf'. And has an important note about -fno-omit-frame-pointer to allow build the frame pointer which helps generate a call graph (see 33:00). At (42:00), how to disable the optimizer. Ok, now that I watched it, it is more about benchmarking than about tracing, but does have good points.
- From Hacker News we get Give me 15 minutes and I'll change your view of Linux tracing by Brendan Gregg.
- From the prolific blogger Julia Evans, Finding out if/why a server is dropping packets: using netstat, dropwatch, and perf. She has another article You Can Be A Kernel Hacker about strace, ptrace, writing kernel modules, ....
- A slightly different perspective on the same thing by Using netstat and dropwatch to observe packet loss on Linux servers by Blog O’ Matty
- From PackageCloud, a very detailed examination of Monitoring and Tuning the Linux Networking Stack: Receiving Data.
- NetOptimizer - Linux Kernel Network Developer has articles on 'making stuff scale'. Amongst a number of interesting articles: Reading live runtime kernel variables, another is about Mini-tutorial for netperf-wrapper setup on RHEL6/CentOS6, and another which demonstrates that the kernel has 10Gbit/s wirespeed.
Bonus article: Injecting a Running Process (Linux>, talks about analysis, injection, and tracing.