community/network_programming_latency.md at a1206430ad195047399b51e26f5ad344e6e26aa6

Network programming latency SLIs/SLOs details

Status	SLI	SLO
WIP	Latency of programming in-cluster load balancing mechanism (e.g. iptables), measured from when service spec or list of its `Ready` pods change to when it is reflected in load balancing mechanism, measured as 99th percentile over last 5 minutes aggregated across all programmers¹	In default Kubernetes installation, 99th percentile per cluster-day <= X

[1]Aggregation across all programmers means that all samples from all programmers go into one large pool, and SLI is percentile from all of them.

As a user of vanilla Kubernetes, I want some guarantee how quickly new backends of my service will be targets of in-cluster load-balancing
As a user of vanilla Kubernetes, I want some guarantee how quickly deleted (or unhealthy) backends of my service will be removed from in-cluster load-balancing
As a user of vanilla Kubernetes, I want some guarantee how quickly changes to service specification (including creation) will be reflected in in-cluster load-balancing

We are consciously focusing on in-cluster load-balancing for the purpose of this SLI, as external load-balancing is clearly provider specific (which makes it hard to set the SLO for it).
However, in the future it should be possible to formulate the SLI for external load-balancing in pretty much the same way for consistency.
The SLI measuring end-to-end time from pod creation was also considered, but rejected due to being application specific, and thus introducing SLO would be impossible.

The SLI is aggregated across all "programmers", which is what is interesting for the end-user. It may happen that small percentage of programmers are completely unresponsive (if all others are fast), but that is desired - we need to allow slower/unresponsive nodes because at some scale it will be happening. The reason for doing it this way is feasibility for efficiently computing that:
- if we would be doing aggregation at the SLI level (i.e. the SLI would be formulated like "... reflected in in-cluster load-balancing mechanism and visible from 99% of programmers"), computing that SLI would be extremely difficult. It's because in order to decide e.g. whether pod transition to Ready state is reflected, we would have to know when exactly it was reflected in 99% of programmers (e.g. iptables). That requires tracking metrics on per-change base (which we can't do efficiently).

The method of measuring this SLI is not obvious, so for completeness we describe it here how it will be implemented with all caveats.

We assume that for the in-cluster load-balancing programming we are using Kubernetes Endpoints objects.
We will introduce a dedicated annotation for Endpoints object (name TBD).
Endpoints controller (while updating a given Endpoints object) will be setting value of that annotation to the timestamp of the change that triggered this update:

for pod transition between Ready and NotReady states, its timestamp is simply part of pod condition
TBD for service updates (ideally we will add LastUpdateTimestamp field in object metadata next to already existing CreationTimestamp. The data is already present at storage layer, so it won't be hard to propagate that.

The in-cluster load-balancing programmer will export a prometheus metric once done with programming. The latency of the operation is defined as difference between timestamp of then whe operation is done and timestamp recorded in the newly introduced annotation.

There are a couple of caveats to that measurement method:

Single Endpoints object may batch multiple pod state transition.
In that case, we simply choose the oldest one (and not expose all timestamps to avoid theoretically unbounded growth of the object). That makes the metric imprecise, but the batching period should be relatively small comparing to whole end-to-end flow.
A single pod may transition its state multiple times within batching period.
For that case, we will add additional cache in Endpoints controller caching the first observed transition timestamp for each pod. The cache will be cleared when controller picks up a pod into Endpoints object update. This is consistent with choosing the oldest update in the above point.
Initially, we may consider simply ignoring this fact.
Components may fall out of watch window history and thus miss some watch events.
This may be the case for both Endpoints controller or kube-proxy (or other network programmers if used instead). That becomes a problem when a single object changed multiple times in the meantime (otherwise informers will deliver handlers on relisting). Additionally, this can happen only when components are too slow in processing events (that would already be reflected in metrics) or (sometimes) after kube-apiserver restart. Given that, we are going to neglect this problem to avoid unnecessary complications for little or no gain.

TODO: Describe test scenario.