I saw a pre-release of "Linux Observability with BPF" by David Calavera and Lorenzo Fontana. Well, I'd say it is a release prior to pre-release. Some good information it has, but teases the details.
This post is about what it takes to get the first example functional: the things I saw, didn't see, and kludges I put in place. With the understanding and first step in place, it can be cleaned up and additional progress can be now proceed.
I have a mixed Debian Buster/Testing/Sid workstation installation. A messy system due to the vagaries of getting Linux Stretch/Buster installed onto a new Intel NUC nuc8i7hvk01 when firmware and display drivers were just getting rolled out. Kernel updates have made progress. Given the inclination, a nice simple clean Buster rebuild will probably work this time around.
But enough of that. Here is what I needed to do for building a simple eBPF program, building a simple installer, and getting it to install, pass the verifier, and achieve a successful conclusion.
$ uname -a
Linux nuc8i7hvk01 4.19.0-4-amd64 #1 SMP Debian 4.19.28-2 (2019-03-15) x86_64 GNU/Linux
The original example eBPF program looks like this:
#include <uapi/linux/bpf.h>
#define SEC(NAME) __attribute__((section(NAME), used))
SEC("tracepoint/syscalls/sys_enter_execve")
int bpf_prog(void *ctx) {
char msg[] = "Hello, BPF World!";
bpf_trace_printk(msg, sizeof(msg));
return 0;
}
char _license[] SEC("license") = "GPL";
For building, a few packages are required:
# need kernel header files for whatever image is installed
sudo apt install linux-headers-4.19.0-4-amd64 linux-image-4.19.0-4-amd64
# llvm and clang are required
sudo apt install clang llvm
# some of the following aren't needed, but might be useful later
sudo apt install libbpf-dev bpfcc-tools bpftrace
# and finish up with elf file tools
sudo apt install elfutils libelf-dev libelf1
Attempting a build, yields the following:
$ clang \
-I/usr/src/linux-headers-4.19.0-4-common/include \
-I/usr/include/x86_64-linux-gnu \
-O2 -target bpf -c hello_kern.c -o hello_kern.o
hello_kern.c:8:3: warning: implicit declaration of function 'bpf_trace_printk' is invalid in C99 [-Wimplicit-function-declaration]
bpf_trace_printk(msg, sizeof(msg));
^
1 warning generated.
For some reason, the definition of bpf_trace_printk is commented-out/not-supplied in /usr/src/linux-headers-4.19.0-3-common/include/uapi/linux/bpf.h. The file builds anyway. Getting ahead of myself,
loading the eBPF program generates an error:
$ ./hello_user
invalid relo for insn[11].code 0x85
bpf_load_program() err=1
The kernel didn't load the BPF program
Checking the assembly language, the verifier does not like the 'call -1', which really means 'unknown symbol':
$ llvm-objdump -S -no-show-raw-insn hello_kern.o
hello_kern.o: file format ELF64-BPF
Disassembly of section tracepoint/syscalls/sys_enter_execve:
bpf_prog1:
0: r1 = 33
1: *(u16 *)(r10 - 8 ) = r1
2: r1 = 7236284523806213712 ll
4: *(u64 *)(r10 - 16) = r1
5: r1 = 4764857262830019912 ll
7: *(u64 *)(r10 - 24) = r1
8: r1 = r10
9: r1 += -24
10: r2 = 18
11: call -1
12: r0 = 0
13: exit
A solution for this is to obtain the following project:
$ git clone https://github.com/netoptimizer/prototype-kernel.git
and then add an include file into the source to obtain a clean build. The source file then looks like:
#include <uapi/linux/bpf.h>
#include "prototype-kernel/kernel/samples/bpf/bpf_helpers.h"
#define SEC(NAME) __attribute__((section(NAME), used))
SEC("tracepoint/syscalls/sys_enter_execve")
int bpf_prog(void *ctx) {
char msg[] = "Hello, BPF World!";
bpf_trace_printk(msg, sizeof(msg));
return 0;
}
char _license[] SEC("license") = "GPL";
Showing the assembly code for a clean build shows a better 'call':
$ llvm-objdump -S -no-show-raw-insn hello_kern.o
hello_kern.o: file format ELF64-BPF
Disassembly of section tracepoint/syscalls/sys_enter_execve:
bpf_prog1:
0: r1 = 33
1: *(u16 *)(r10 - 8 ) = r1
2: r1 = 7236284523806213712 ll
4: *(u64 *)(r10 - 16) = r1
5: r1 = 4764857262830019912 ll
7: *(u64 *)(r10 - 24) = r1
8: r1 = r10
9: r1 += -24
10: r2 = 18
11: call 6
12: r0 = 0
13: exit
The second stage is to actually load the eBPF module. A loader is needed. In this case a small native c program can be created:
$ cat hello_user.c
#include <stdio.h>
// originally had:
// #include "bpf_load.h"
// changed to:
#include "linux-4.19.39/samples/bpf/bpf_load.h"
int main(int argc, char **argv) {
if (load_bpf_file("hello_kern.o") != 0) {
printf("The kernel didn't load the BPF program\\n");
return -1;
}
read_trace_pipe();
return 0;
}
To build this program required obtaining the kernel source to obtain the content of the samples/bpf directory for my kernel (approximate version):
$ wget --no-check-certificate \
https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.39.tar.xz
$ tar xvf linux-4.19.39.tar
For whatever reason, the sample directory is not complete. The following file needs to be copied over:
$ cp \
prototype-kernel/kernel/samples/bpf/perf-sys.h \
linux-4.19.39/samples/bpf/
A more complicated build line with gcc (native compiler) is needed for various headers and source:
$ gcc \
-I /usr/include/x86_64-linux-gnu \
-I /usr/src/linux-headers-4.19.0-4-amd64/arch/x86/include/generated \
-I /usr/src/linux-headers-4.19.0-4-amd64/include \
-I /usr/src/linux-headers-4.19.0-4-common/arch/x86/include \
-O2 -Wall -g -lelf -lbpf \
-o hello_user hello_user.c linux-4.19.39/samples/bpf/bpf_load.c
This results in a build with only a couple unused variables:
linux-4.19.39/samples/bpf/bpf_load.c: In function ‘parse_relo_and_apply’:
linux-4.19.39/samples/bpf/bpf_load.c:351:7: warning: unused variable ‘j’ [-Wunused-variable]
int j, map_idx;
^
linux-4.19.39/samples/bpf/bpf_load.c: In function ‘load_elf_maps_section’:
linux-4.19.39/samples/bpf/bpf_load.c:407:6: warning: unused variable ‘copy_sz’ [-Wunused-variable]
int copy_sz;
^~~~~~~
Running provides something like:
$ sudo ./hello_user
x2gocleansessio-20806 [005] .... 2408784.962224: 0: Hello, BPF World!
x2gocleansessio-20807 [003] .... 2408784.968652: 0: Hello, BPF World!
....
Success!
Another diagnostic tool to see the sections:
$ readelf -a hello_kern.o
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: Linux BPF
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 384 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 64 (bytes)
Number of section headers: 8
Section header string table index: 1
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .strtab STRTAB 0000000000000000 00000112
000000000000006c 0000000000000000 0 0 1
[ 2] .text PROGBITS 0000000000000000 00000040
0000000000000000 0000000000000000 AX 0 0 4
[ 3] tracepoint/syscal PROGBITS 0000000000000000 00000040
0000000000000070 0000000000000000 AX 0 0 8
[ 4] .rodata.str1.1 PROGBITS 0000000000000000 000000b0
0000000000000012 0000000000000001 AMS 0 0 1
[ 5] license PROGBITS 0000000000000000 000000c2
0000000000000004 0000000000000000 WA 0 0 1
[ 6] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 00000110
0000000000000002 0000000000000000 E 7 0 1
[ 7] .symtab SYMTAB 0000000000000000 000000c8
0000000000000048 0000000000000018 1 1 8
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)
There are no section groups in this file.
There are no program headers in this file.
There is no dynamic section in this file.
There are no relocations in this file.
The decoding of unwind sections for machine type Linux BPF is not currently supported.
Symbol table '.symtab' contains 3 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 5 _license
2: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 3 bpf_prog1
No version information found in this file.
A standard tool to list all the symbols in an Elf object file:
$ nm hello_user
U __assert_fail@@GLIBC_2.2.5
U bpf_create_map_in_map_node
U bpf_create_map_node
U bpf_load_program
0000000000006360 B bpf_log_buf
U bpf_map_update_elem
U bpf_raw_tracepoint_open
000000000000514c B __bss_start
00000000000051a0 b buf.7588
.... more stuff
2019/08/14 - Using Debian Kernel 5.2.7, but doesn't function as the following shows (if anyone knows the solution or underlying issue, please let me know):
# uname -a
Linux nuc8i7hvk01 4.19.0-5-amd64 #1 SMP Debian 4.19.37-6 (2019-07-18) x86_64 GNU/Linux
# cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_execve/id
677
# uname -a
Linux nuc8i7hvk01 5.2.0-2-amd64 #1 SMP Debian 5.2.7-1 (2019-08-07) x86_64 GNU/Linux
# cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_execve/id
Killed
2019/08/14 - bpftool in kernel - this is reachable via kernel source code:
wget --no-check-certificate http://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.2.7.tar.gz
tar zxvf linux-5.2.7.tar.gz
sudo apt install build-essential
cd linux-5.2.7/tools/bpf/bpftool/
make
# linux-5.2.7/tools/bpf/bpftool/bpftool feature
Scanning system configuration...
bpf() syscall for unprivileged users is enabled
JIT compiler is enabled
JIT compiler hardening is disabled
JIT compiler kallsyms exports are disabled
Global memory limit for JIT compiler for unprivileged users is 264241152 bytes
CONFIG_BPF is set to y
CONFIG_BPF_SYSCALL is set to y
CONFIG_HAVE_EBPF_JIT is set to y
CONFIG_BPF_JIT is set to y
CONFIG_BPF_JIT_ALWAYS_ON is not set
......
A flag to enable:
echo 1 > /proc/sys/net/core/bpf_jit_enable
