diff --git a/perf/benchmark/README.md b/perf/benchmark/README.md index d1cf840424..84bf548860 100644 --- a/perf/benchmark/README.md +++ b/perf/benchmark/README.md @@ -145,12 +145,41 @@ optional arguments: --no_clientsidecar do not run clientsidecar-only for all --bothsidecar run both clientsidecar and serversidecar --no_sidecar do not run clientsidecar and serversidecar + --custom_profiling_command + runs a custom profiling commands on the nodes for the client and server, + and produces a flamegraph based on that. + Example on-cpu profile using bcc tools for the envoy sidecar proxy: + --custom_profiling_command=\"profile-bpfcc -df {duration} -p {sidecar_pid}\" + - runner.py will replace {duration} with whatever was specified for --duration. + - runner.py will replace {sidecar_pid} with the actual process id of the envoy + sidecar process. + --custom_profiling_name + filename prefix for the result of any --custom_profiling_command + --envoy_profiler [heapprofiler|cpuprofiler] + yields visualizations using pprof over profiles collected via the built-in profiler + of the side cars. + NOTE: requires global.proxy.privileged=true,values.global.proxy.enableCoreDump=true ``` Note: - `runner.py` will run all combinations of the parameters given. However, in order to reduce ambiguity when generating the graph, it would be better to change one parameter at a time and fix other parameters - if you want to run with `--perf` flag to generate a flame graph, please make sure you have the permission to gather perf data, please refer to step 2 of this [README](https://github.com/istio/tools/tree/master/perf/benchmark/flame#setup-perf-tool) +- if you want to run with `--custom_profiling_command`, `profilingMode` must be set to `true` in `values.yaml`. Doing so will set up the client and server pods to run the perf/profiling container. It's worth noting that this container runs `--priviledged`, and that `hostIPC` and `hostPID` will also be enabled, +weakening security. Resulting flamegraphs will be written to `flame/flameoutput`. +- sample sidecar profiling commands for `--custom_profiling_command`: + - "profile-bpfcc -df {duration} -p {sidecar_pid}" sidecar on-cpu profile + - "offcputime-bpfcc -df {duration} -p {sidecar_pid}" sidecar off-cpu profile + - "offwaketime-bpfcc -df {duration} -p {sidecar_pid}" sidecar offwaketime profile + - "wakeuptime-bpfcc -f -p {sidecar_pid} {duration}" sidecar wakeuptime profile + - "perf record -F 99 -g -p {sidecar_pid} -- sleep {duration} && perf script | ~/FlameGraph/stackcollapse-perf.pl | c++filt -n" on-cpu perf-generated profile + - "stackcount-bpfcc -U *alloc* -df -D {duration} -p {sidecar_pid}" profile calls to `*alloc*` + - "perf record -e page-faults -g -p {sidecar_pid} -F 99 -- sleep {duration} && perf script | ~/FlameGraph/stackcollapse-perf.pl | c++filt -n" page faults +- It's also possible to run machine-wide profiling, for example: + - "profile-bpfcc -df {duration}" for obtaining a machine-wide on-cpu flamegraph. + - See [here](http://www.brendangregg.com/FlameGraphs/) for more examples and information. +- Enabling `profilingMode` in `values.yaml` will also bring up and expose Prometheus's `node_exporter` at the configured port (default: 9100), + accessible over http via `/metrics. For example: diff --git a/perf/benchmark/flame/README.md b/perf/benchmark/flame/README.md index 0d7f6e3120..bfa99298dd 100644 --- a/perf/benchmark/flame/README.md +++ b/perf/benchmark/flame/README.md @@ -8,61 +8,61 @@ 1. Colors are arbitrary. 1. Function names are sorted left to right. -This document shows how to gather performance data from within the `istio-proxy` container. +This document shows how to gather performance data from via the `perf` container. -## Setup Perf tool +## Setup the perf container -Flame graphs are created from data collected using linux `perf_events` by the `perf` tool. +Enable `profilingMode` in [values.yaml](../values.yaml). This will end up adding the perf +container to the server and client pods, which both will be running on separate nodes. -1. Ensure that `perf` is installed within the container. - Since `istio-proxy` container does not allow installation of new packages, build a new docker image. +Flame graphs and visualizations are created from data collected using linux `perf_events` +by the `perf` and [BCC tools](https://github.com/iovisor/bcc), as well as Envoy's built-in profiler. - ```plain - FROM gcr.io/istio-release/proxyv2:release-1.0-20180810-09-15 - # Install fpm tool - RUN sudo apt-get update && \ - sudo apt-get -qqy install linux-tools-generic - ``` +## Obtaining flame graphs - Build image and push docker image and use it in your deployment by adding the following annotation. +Flame graphs can be produced via `runner.py`, and will be stored in `flame/flameoutput`. - ```plain - "sidecar.istio.io/proxyImag" : - ``` +A few sample command line arguments. `{duration}` will be replaced by +whatever was passed for `--duration` to runner.py. `{sidecar_pid}` will +be replaced by `runner.py` with the process id of the Envoy sidecar. - This step will go away once the default debug image contains `perf` and related tools. +It is valid to omit `{sidecar_pid}` in `--custom_profiling_command`. +This may be useful for machine-wide profiling or arbitrary processes. -1. Ensure that you can run `perf record` +```bash +runner/runner.py ... --custom_profiling_command="profile-bpfcc -df {duration} -p {sidecar_pid}" --custom_profiling_name="bcc-oncputime-sidecar" - Running `perf record` from container requires the host to permit this activity. This is done by running the following command on the vm host. - For example, if you are running on a GKE cluster, you should `ssh` to the node using the command: +runner/runner.py ... --custom_profiling_command="offcputime-bpfcc -df {duration} -p {sidecar_pid}" --custom_profiling_name="bcc-offcputime-sidecar" - ```bash - gcloud compute ssh gke-perf-test-default-pool-xxxxxx - ``` +runner/runner.py ... --custom_profiling_command="offwaketime-bpfcc -df {duration} -p {sidecar_pid}" --custom_profiling_name="bcc-offwaketime-sidecar" - Then run the following command: +runner/runner.py ... --custom_profiling_command="wakeuptime-bpfcc -f -p {sidecar_pid} {duration}" --custom_profiling_name="bcc-wakeuptime-sidecar" - ```bash - sudo sysctl kernel.perf_event_paranoid=-1 - sudo sysctl kernel.kptr_restrict=0 - ``` +runner/runner.py ... --custom_profiling_command="stackcount-bpfcc -p {sidecar_pid} *alloc* -fD {duration}" --custom_profiling_name="bcc-stackcount-alloc" - This setting is very permissive so it must be used with care. +runner/runner.py ... --custom_profiling_command="perf record -F 99 -g -p {sidecar_pid} -- sleep {duration} && perf script | ~/FlameGraph/stackcollapse-perf.pl | c++filt -n" +--custom_profiling_name="perf-oncputime-sidecar" - If running perf still gives error:```You may not have permission to collect stats. Consider tweaking /proc/sys/kernel/perf_event_paranoid:``` - after running above commands, try ssh into node and run the container with --privileged flag. +runner/runner.py ... --custom_profiling_command="perf record -e page-faults -g -p {sidecar_pid} -- sleep {duration} && perf script | ~/FlameGraph/stackcollapse-perf.pl | c++filt -n" --custom_profiling_name="perf-pagefaults-sidecar" -1. Run [`get_proxy_perf.sh`](get_proxy_perf.sh) to get the profiling svg. The following command collects samples at `177Hz` for `20s`. The svg file should be created under `flameoutput` dir +``` - ```plain - ./get_proxy_perf.sh -p svc05-0-7-564865d756-pvjhn -n service-graph05 -s 177 -t 20 - ... - [ perf record: Woken up 1 times to write data ] - [ perf record: Captured and wrote 0.061 MB /etc/istio/proxy/perf.data (74 samples) ] +## Leveraging Istio's sidecar built-in profiling - Wrote /etc/istio/proxy/perf.data.perf - ... - generating svg file svc05-0-7-564865d756-pvjhn-2020-01-29-22-34-19.perf - ... - ``` +Istio's sidecar proxy (Envoy) is usually build with `tcmalloc`, and as such traditional memory profiling +methods may give unsatisfactory results. Fortunately, the proxy provides a built-in means to collect +profiling data, and the benchmark tool is able to leverage that. Doing so, however, requires a writeable +file system as well as priviliges to install new packages for the sidecar containers. The following +command (re)configures istio to satisfy these requirements: + +```bash +istioctl manifest apply --set "values.global.proxy.enableCoreDump=true" --set "values.global.proxy.privileged=true" +``` + +After doing so `runner.py` can be run with `--envoy_profiler [heapprofiler|cpuprofiler]`. This will start/stop +the built-in profilers of the sidecars, obtain the collected profiles, and visualize them via `pprof`. +The resulting output will end up in `flame/flameoutput` just like in the other flows: + +```bash +runner/runner.py --envoy_profiler cpuprofiler|heapprofiler ... +``` diff --git a/perf/benchmark/flame/flame.sh b/perf/benchmark/flame/flame.sh deleted file mode 100755 index ad09de7263..0000000000 --- a/perf/benchmark/flame/flame.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/bin/bash - -# Copyright Istio Authors -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -e - -WD=$(dirname "${0}") -WD=$(cd "${WD}" && pwd) - -FLAMEDIR="${WD}/FlameGraph" - -if ! command -v c++filt > /dev/null; then - echo "Install c++filt to demangle symbols" - exit 1 -fi - -cd "${WD}" || exit 1 - -if [[ ! -d ${FLAMEDIR} ]]; then - echo "Cloning FlameGraph repo in ${WD}" - git clone https://github.com/brendangregg/FlameGraph -fi - -# Given output of `perf script` produce a flamegraph -FILE=${1:?"perf script output"} -FILENAME=$(basename "${FILE}") -BASE=$(echo "${FILENAME}" | cut -d '.' -f 1) -SVGNAME="${BASE}.svg" - -mkdir -p "${WD}/flameoutput" -"${FLAMEDIR}/stackcollapse-perf.pl" "${FILE}" | c++filt -n | "${FLAMEDIR}/flamegraph.pl" --cp > "./flameoutput/${SVGNAME}" - -echo "Wrote ${SVGNAME}" -if [[ -n "${BUCKET}" ]];then - gsutil cp "${SVGNAME}" "${BUCKET}" -fi diff --git a/perf/benchmark/flame/get_perfdata.sh b/perf/benchmark/flame/get_perfdata.sh deleted file mode 100755 index 4881935ffd..0000000000 --- a/perf/benchmark/flame/get_perfdata.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash - -# Copyright Istio Authors -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -e - -WD=$(dirname "${0}") -WD=$(cd "${WD}" && pwd) - -FILENAME=${1:?"perffilename"} -DURATION=${2:?"duration"} -FREQ=${3:-"99"} - -PID=$(pgrep envoy) - -# This is specific to the kernel version -# example: /usr/lib/linux-tools-4.4.0-131/perf -# provided by `linux-tools-generic` -PERFDIR=$(find /usr/lib -name 'linux-tools-*' -type d | head -n 1) -if [[ -z "${PERFDIR}" ]]; then - echo "Missing perf tool. Install apt-get install linux-tools-generic" - exit 1 -fi - -PERF="${PERFDIR}/perf" - -"${PERF}" record -o "${WD}/${FILENAME}" -F "${FREQ}" -p "${PID}" -g -- sleep "${DURATION}" -"${PERF}" script -i "${WD}/${FILENAME}" --demangle > "${WD}/${FILENAME}.perf" - -echo "Wrote ${WD}/${FILENAME}.perf" diff --git a/perf/benchmark/flame/get_proxy_perf.sh b/perf/benchmark/flame/get_proxy_perf.sh deleted file mode 100755 index dd0f10e2b2..0000000000 --- a/perf/benchmark/flame/get_proxy_perf.sh +++ /dev/null @@ -1,77 +0,0 @@ -#!/bin/bash - -# Copyright Istio Authors -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -e - -function usage() { - echo "usage: - ./get_proxy_perf.sh -p -n -s -t