# Benchmarks

This folder provides Rust benchmarks.

## Running

```shell
$ cargo bench # Just run benchmarks
$ cargo bench -- --quick # Just run benchmarks, with less samples
$ cargo bench -- --profile-time 10 # run benchmarks with cpu profile; results will be in out/rust/criterion/<group>/<test>/profile/profile.pb
$ # Compare to a baseline
$ cargo bench -- --save-baseline <name> # save baseline
$ # ...change something...
$ cargo bench -- --baseline <name> # compare against it
```

## Performance

Ztunnel performance largely falls into throughput and latency.
While these are sometimes at odds with each other, as Ztunnel is a generic proxy, we aim to make it perform well on both metrics.

### Request flows

The primary responsibility of the proxy is copying bits between peers.
Currently, this is always either `TCP<-->TCP` or `TCP<-->HBONE`.

#### `TCP` to `TCP`

This is the simplest case, and common amongst many proxies.
[`copy.rs`](../src/copy.rs) does the bulk of the work, essentially just bi-directionally copying bytes between the two sockets.

Typical bi-di copies are using a fixed buffer.
To adapt to various workloads, we use dynamically sized buffers, that can grow from 1kb -> 16kb -> 256kb when enough traffic is received.
This allows high throughput workloads to perform well, without excessive memory costs for low-bandwidth services.

#### `TCP` to `HBONE`

This case ends up being much more complex, as we flow through HTTP2 and TLS.
The full flow looks as such (pseudocode):

```raw
copy_bidi():
    loop {
        data = tcp_in.read(up to 256k) # based on dynamic buffer size
        h2.write(data)
    }
h2::write(data):
    Buffer data as a DATA frame, up to a max of `max_send_buffer_size`. We configure this to 256k.
    Asyncronously, the connection driver will pick up this data and call `rustls.write_vectored([256bytes, rest of data])`.
rustls::write(data):
    data=encrypt(data)
    # TLS records are at most 16k
    # In practice I have observed at most 4 chunks; unclear where this is configured.
    tcp_out.write_vectored([chunks of 16k])
```

From an `iperf` load, this ends up looking something like this in `strace`:

```raw
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 55.21    0.841290           5    140711           writev
 44.78    0.682359          17     38481           recvfrom
 ```

This will be from `writev([16kb * 4])` calls and `recvfrom(256kb)`.

#### `HBONE` to `TCP`

This flow is substantially different from the inverse direction.
The receive flow is driven by `h2`. Under the hood this uses a [`LengthDelimitedCodec`](https://docs.rs/tokio-util/latest/tokio_util/codec/length_delimited/struct.LengthDelimitedCodec.html).
`h2` will attempt to decode 1 frame at a time, using an internal buffer.
This buffer starts at [`8kb`](https://github.com/tokio-rs/tokio/blob/ed4ddf443d93c3e14ae23699a5a2f81902ad1e66/tokio-util/src/codec/framed_impl.rs#L26) but will grow to meet the size of frames.
We allow up to a max of `1mb` frame sizes (`config.frame_size`).

Ultimately, this will call `rustls.read(buf)`.
This goes through a few indirections, but ultimately ends up in `rustls.deframer_buffer`.
This is what calls `read()` on the underlying IO, in our case the TCP connection.
This buffer is configured to do [`4kb`](https://github.com/rustls/rustls/blob/8a8023addb9ae311f66b16e272e85654c9588eeb/rustls/src/msgs/deframer.rs#L724) reads generally.

Upon reading the frame from the wire, these get [buffered up by `h2`](https://github.com/hyperium/h2/blob/4617f49b266d560a773372a90be283ba8b2400a9/src/proto/streams/stream.rs#L100).
We read these in [`recv_stream.poll_data`](../src/proxy/h2.rs), trigger by the `copy_bidirectional`.
Ultimately, this will write out 1 DATA frame worth of data to the upstream TCP connection

From an `iperf` load, this ends up looking something like this in `strace`:

```raw
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 61.08    1.253541          50     24703           sendto
 38.19    0.783733           2    360707         8 recvfrom
```

This will be from `sendto(256kb)` calls, with many `recvfrom()` calls ranging from 4k to 16k.

#### Comparison to Envoy

Under an `iperf` load, Envoy client:

```raw
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 68.24    1.363149           3    440440         1 sendto
 31.72    0.633584          11     55114        31 readv
```

This is from many `sendto(16k)` calls, and `readv([16k]*8)`.

Envoy Server:

```raw
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.24    1.199264           1    757275         8 recvfrom
 34.73    0.638315          26     23670           writev
```

This is from many calls of `recvfrom(5); recvfrom(16k)`, and `writev([16k]*16)`.

(All strace commands are looking at `-e trace=write,writev,read,recvfrom,sendto,readv`).
