linkerd2-proxy/profiling

linkerd2-proxy Benchmarking and Profiling

This directory contains a set of scripts and configurations for benchmarking and profiling the Linkerd 2 proxy, using the benchmarking tools Fortio and iPerf. Additionally, it contains scripts for profiling the proxy using these benchmark configurations, as well as measuring latency.

Requirements

All the benchmarks are run in Docker containers using docker-compose. For most of the scripts in this directory, only docker-compose v3.7+ is required.

The profiling-perf script requires perf to be installed on the host, and is only guaranteed to work on Linux (it may also work in Docker for Mac's Linux VM, but your mileage may vary).

Running Tests

This directory contains three test scripts:

• benchmark.sh performs a benchmark of the proxy, recording accurate latency information.
• profiling-perf.sh performs CPU profiling with perf, outputting a flamegraph of the proxy's CPU usage.
• profiling-heap.sh performs heap profiling using memory_profiler, outputting a flamegraph of heap allocations and a .dat file that may be further analyzed using heaptrack or memory-profiler-cli.

The profiling scripts perform the same test as benchmark.sh, but the proxy is configured to collect additional profiling information. Due to the overhead of instrumenting the proxy to profile allocations or CPU usage, the latencies recorded during profiling are probably not representative of proxy performance "in the wild".

Whenever the proxy under test has changed (e.g. when making code changes, or checking out a different branch to compare), it's necessary to rebuild the test proxy Docker image. To do this, run the following command in the profiling directory:

$ docker-compose build proxy

Customizing Parameters

The scripts in this directory take the following environment variables to determine the test parameters. When not provided they default to the values as listed here.

• TCP="1": Enable/disable TCP benchmark.
• HTTP="1": Enable/disable HTTP benchmark.
• GRPC="1": Enable/disable gRPC benchmark.
• ITERATIONS="1": The number of times a single HTTP/gRPC benchmark run is repeated to observe the maximal tail latency.
• DURATION="10s": Execution time for a single HTTP/gRPC benchmark run.
• CONNECTIONS="4": Number of concurrent TCP connections for HTTP/gRPC.
• GRPC_STREAMS="4": Number of HTTP/2 streams in a connection.
• HTTP_RPS="4000 7000": Different target HTTP req/s numbers as space-separated list. It may not be achieved if too high. Please see the actual req/s in the log output.
• GRPC_RPS="4000 6000": As above but for gRPC.
• REQ_BODY_LEN="10 200": Length of the request body payload in byte as space-separated list.
• HIDE="1": Hide/show output of every command. The output of each benchmark utility is stored to log files in any case.

For example:

$ ITERATIONS=2 DURATION=2s CONNECTIONS=2 GRPC_STREAMS=2 HTTP_RPS="100" GRPC_RPS="100 1000" REQ_BODY_LEN="100 8000" ./benchmark.sh

Test Output

All test run output is written to the target/profile/${TIMESTAMP}/ directory in the root of the linkerd2-proxy repository. This includes the following:

• a summary CSV describing the observed 99th percentile latencies (or GBit/s for TCP)
• JSON files containing the raw data output by Fortio
• flamegraphs of the heap or CPU profile collected during the test, if running the profiling scripts
• heaptrack.dat files that can be analyzed using Heaptrack, when running the profiling-heap.sh script
• .folded perf output that may be used as input to generate additional flamegraphs, when running the profiling-perf.sh script

To remove all data generated by profiling, run:

$ make clean-profile

from the root of the repository.

Comparing the summary file of two branches

The plot.sh script generates graphs to compare the summary CSVs output by different test runs. For example:

$ cd profiling/
$ ./benchmark.sh ; git checkout main && ./benchmark.sh
$ ./plot.sh \
    ../target/profile/2019Jun19_15h13m12s/summary.txt \
    ../target/profile/.2019Jun19_15h34m26s/summary.txt \
    mybranch-vs-main-
$ eog mybranch-vs-main-*png

Consider using ./plot.sh --logy ... to switch the Y-axis to a logarithmic scale, if the difference between the values is too high to see the low values.

Another option is to quickly compare in textual form:

$ git diff --no-index --word-diff \
    ../target/profile/2019Jun19_15h13m12s/summary.txt \
    ../target/profile/.2019Jun19_15h34m26s/summary.txt