-
Notifications
You must be signed in to change notification settings - Fork 232
Tips for Debugging
This page gives some tips on debugging Gatekeeper using pktgen and gdb
This section describes how to install and use the DPDK pktgen program to generate packets. We provide an example based on our "XIA" server setup.
The XIA server has four DPDK-enabled ports that can be used to send and receive packets. The diagram below shows the four ports (along with their PCI addresses):
+-----------------------------------------+
| ----- ----- |
| | o | A (83:00.0) | o | B (83:00.1) |
| --|-- --|-- |
| | | |
| --|-- --|-- |
| | o | C (85:00.0) | o | D (85:00.1) |
| ----- ----- |
+-----------------------------------------+
Note that ports A and C are connected, and ports B and D are connected. The MAC addresses of the four ports are A (e8:ea:6a:06:1f:7c), B (e8:ea:6a:06:1f:7d), C (e8:ea:6a:06:21:b2), and D (e8:ea:6a:06:21:b3). The port MAC addresses may be used for generating specific packets in the following sections.
When you run multiple DPDK applications at the same time (such as two instances of pktgen, or pktgen and another DPDK application), you need to blacklist the PCI devices, or ports in this case, that you don't want to use. Depending on how you blacklist the devices, the port numbers that are assigned by DPDK will change.
For example, if you blacklist ports A, B, and C in one instance of pktgen, then port D is known as port 0 in that instance. If, at the same time, you blacklist ports A, C, and D in another instance of pktgen, then in that instance port B is known as port 0. To avoid ambiguity, in this document we will refer to the ports using these letter designations (A, B, C, D) instead of by numbers.
First, obtain the pktgen source code and build it:
$ git clone http://dpdk.org/git/apps/pktgen-dpdk
$ cd pktgen-dpdk
$ make
Once it compiles, do a setup step:
$ sudo -E ./setup.sh
The application should be ready to run at this point, but you may also encounter the need to perform the following step (still in the pktgen-dpdk directory):
$ cp Pktgen.lua app/app/x86_64-native-linuxapp-gcc/
This demonstration will show how we can generate packets on port A and send them to port C. To begin, open two terminals.
In the first terminal, go to the dpdk-pktgen directory and run:
$ cd app/app/x86_64-native-linuxapp-gcc/
$ sudo ./pktgen -l 0-2 --socket-mem 256 --file-prefix pg1 -b 83:00.1 -b 85:00.0 -b 85:00.1 -- -T -P -m "[1:2].0"
The "-c 7" option specifies which lcores are available for use, which is lcores 0, 1, and 2. The "-m [1:2].0" part specifies that lcores 1 and 2 will handle rx/tx on port 0, and lcore 0 is automatically assigned to the pktgen program for displaying statistics.
The "--socket-mem 256" option puts a limit on the memory used, which is often needed when multiple DPDK applications are run at the same time. The "--file-prefix pg1" option specifies a special file prefix to use for this DPDK application's meta information, which again is needed if there are multiple DPDK applications running (which we will have in the next part).
The "-b" options blacklist different ports -- in other words, excludes those ports from being used in the application. In this command, we only want to use port A (at PCI location 83:00.0), so we blacklist the other three ports: B (83:00.1), C (85:00.0), and D (85:00.1).
The "-T" option gives the display statistics some color and the "-P" makes all ports run in promiscuous mode.
In the second terminal, run the same command, but with different lcores and blacklisted ports:
$ cd app/app/x86_64-native-linuxapp-gcc/
$ sudo ./pktgen -c 70 --socket-mem 256 --file-prefix pg2 -b 83:00.0 -b 83:00.1 -b 85:00.1 -- -T -P -m "[5:6].0"
This does the same as the first command, but instead allows lcores 4, 5, and 6 to be used, uses a different file prefix, and blacklists all ports except for port C.
More information about the command-line parameters is here.
Once the applications are running, go to the terminal running pktgen on port A. You can start packets flowing using:
$ Pktgen> start 0
Remember that port numbering always starts from 0 within an application, so since we blacklisted all other ports, port 0 means port A.
When you do this, you should see the second terminal's statistics being updated. You could also start packets flowing on the second terminal (port C) and see the packets being received on port A -- on the second terminal, port C will also be called port 0, since that is the only active port in that application.
You can stop packets flowing with:
$ Pktgen> stop 0
And quit with:
$ Pktgen> quit
Since Gatekeeper is compiled with the -g option, gdb can be directly used with Gatekeeper. After it is compiled, you can run:
$ sudo gdb ./build/gatekeeper
Note that, to have more debugging information, it's better to compile Gatekeeper without the -O3 option.
From there, you can use gdb as described in its documentation. You will need to run Gatekeeper without hugepages, so once gdb starts, you can run:
(gdb) run
Sometimes one has to investigate the code of DPDK with `gdb` as well, and having DPDK compiled with debug information is very helpful in these cases. To compile the DPDK and sample applications with debugging information included and the optimization level set to 0, the EXTRA_CFLAGS environment variable should be set before compiling as follows:
export EXTRA_CFLAGS='-O0 -g'
Once the variable `EXTRA_CFLAGS` is exported, just call `make` as you would to compile DPDK.
Flow entries may have BPF programs associated with them, and one may need to verify that these programs were correctly compiled. To do so, one can disassemble a BPF program as follows:
$ llvm-objdump -S -no-show-raw-insn -section=init declined.bpf
declined.bpf: file format ELF64-BPF
Disassembly of section init:
declined_init:
0: r0 = 0
1: exit
The option -no-show-raw-insn suppresses the hexadecimal representation of the instructions, which is only useful if one is debugging the BPF VM itself.
Gatekeeper BPF programs have two sections: "init" and "pkt". Section "init" is used to initialize a BPF state, whereas the section "pkt" is used to process a packet associate with a flow.