pwru (packet, where are you?)

pwru is an eBPF-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities. It allows fine-grained introspection of kernel state to facilitate debugging network connectivity issues.

The following example shows where the packets of a curl request are dropped after installing an IP tables rule:

Running

Requirements

pwru requires >= 5.3 kernel to run. For --output-skb >= 5.9 kernel is required.

The following kernel configuration is required.

Option	Note
CONFIG_DEBUG_INFO_BTF=y	Available since >= 5.3
CONFIG_KPROBES=y
CONFIG_PERF_EVENTS=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y

You can use zgrep $OPTION /proc/config.gz to validate whether option is enabled.

Downloading

You can download the statically linked executable for x86_64 and amd64 from the release page.

Usage

$ pwru --help
Usage of ./pwru:
      --filter-dst-ip string      filter destination IP addr
      --filter-dst-port uint16    filter destination port
      --filter-func string        filter kernel functions to be probed by name (exact match, supports RE2 regular expression)
      --filter-mark uint32        filter skb mark
      --filter-netns uint32       filter netns inode
      --filter-proto string       filter L4 protocol (tcp, udp, icmp, icmp6)
      --filter-src-ip string      filter source IP addr
      --filter-src-port uint16    filter source port
      --kernel-btf string         specify kernel BTF file
      --kmods strings             list of kernel modules names to attach to
      --output-limit-lines uint   exit the program after the number of events has been received/printed
      --output-meta               print skb metadata
      --output-skb                print skb
      --output-stack              print stack
      --output-tuple              print L4 tuple
      --per-cpu-buffer int        per CPU buffer in bytes (default 4096)
      --timestamp string          print timestamp per skb ("current", "relative", "none") (default "none")
      --version                   show pwru version and exit

If multiple filters are specified, all of them have to match in order for a packet to be traced.

The --filter-func switch does an exact match on function names i.e. --filter-func=foo only matches foo(); for a wildcarded match, try --filter-func=".*foo.*" instead.

Running with Docker

Docker images for pwru are published at https://hub.docker.com/r/cilium/pwru.

An example how to run pwru with Docker:

docker run --privileged --rm -t --pid=host -v /sys/kernel/debug/tracing:/sys/kernel/debug/tracing cilium/pwru --filter-dst-ip=1.1.1.1

Note: In case the path /sys/kernel/debug/tracing doesn't exist in the container, you can see error similar to

docker: Error response from daemon: error while creating mount source path '/sys/kernel/debug/tracing': mkdir /sys/kernel/debug/tracing: no such file or directory

One way to avoid this is to mount the entire /sys/kernel/debug path from host to the container.

docker run --privileged --rm -t --pid=host -v /sys/kernel/debug:/sys/kernel/debug cilium/pwru --filter-dst-ip=1.1.1.1

Running on Kubernetes

The following example shows how to run pwru on a given node:

NODE=node-foobar
kubectl run pwru \
    --image=cilium/pwru:latest \
    --privileged=true \
    --attach=true -i=true --tty=true --rm=true \
    --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"kubernetes.io/hostname":"'$NODE'"}, "hostNetwork": true, "hostPID": true}}' \
    -- --filter-dst-ip=1.1.1.1 --output-tuple

Note: You may need to create a volume for /sys/kernel/debug/tracing and mount it for thepwru pod. Similar to docker, if /sys/kernel/debug/tracing path doesn't exist in the container, mount entire /sys/kernel/debug to the container.

Running on Vagrant

If you have Vagrant installed, you can run the above example with the following commands.

1. In a terminal (terminal 1), bring up the Vagrant box:

   $ vagrant up

   This will take a few minutes to download and provision the box.

2. Connect to the Vagrant box:

   $ vagrant ssh

3. Build pwru:

   $ cd /pwru
   $ make

4. Run pwru:

   $ sudo ./pwru --filter-dst-ip=1.1.1.1 --filter-dst-port=80 --filter-proto=tcp --output-tuple

5. In a new terminal (terminal 2), connect to the Vagrant box:

   $ vagrant ssh

6. In terminal 2, run curl to generate some traffic to 1.1.1.1:

   $ curl 1.1.1.1

   Observe the output of pwru in terminal 1.

7. In terminal 2, add an iptables rule to block traffic to 1.1.1.1:

   $ sudo iptables -t filter -I OUTPUT 1 -m tcp --proto tcp --dst 1.1.1.1/32 -j DROP

8. In terminal 2, run curl to generate some traffic to 1.1.1.1:

   $ curl 1.1.1.1

   Observe the output of pwru in terminal 1.

9. To clean up, press Ctrl+C to terminate pwru in terminal 1, exit both shells, and run:

   $ vagrant destroy

Developing

Dependencies

• Go >= 1.16
• LLVM/clang >= 1.12

Building

make

Alternatively, you can build in the Docker container:

make release

Contributing

pwru is an open source project licensed under GPLv2. Everybody is welcome to contribute. Contributors are required to follow the Contributor Covenant Code of Conduct and must adhere to the Developer Certificate of Origin by adding a Signed-off-by line to their commit messages.

Logo Credits

The detective gopher is based on the Go gopher designed by Renee French.