kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document exclusions for SLO/SLI API call latency 

Signed-off-by: andrewsykim <andrewsy@google.com>
This commit is contained in:
andrewsykim 2023-03-16 11:42:43 -04:00
parent 4d0eb229f7
commit 6f1cc290b9
2 changed files with 16 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
sig-scalability/slos/api_call_latency.md
View File

@ -4,12 +4,13 @@
| Status | SLI | SLO | | Status | SLI | SLO |
| --- | --- | --- | | --- | --- | --- |
| __Official__ | Latency<sup>[1](#footnote1)</sup> of mutating<sup>[2](#footnote2)</sup> API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day <= 1s | | __Official__ | Latency of processing<sup>[1](#footnote1)</sup> mutating<sup>[2](#footnote2)</sup> API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day <= 1s |
| __Official__ | Latency<sup>[1](#footnote1)</sup> of non-streaming read-only<sup>[3](#footnote3)</sup> API calls for every (resource, scope<sup>[4](#footnote4)</sup>) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day: (a) <= 1s if `scope=resource` (b) <= 30s<sup>[5](#footnote5)</sup> otherwise (if `scope=namespace` or `scope=cluster`) | | __Official__ | Latency of processing<sup>[1](#footnote1)</sup> non-streaming read-only<sup>[3](#footnote3)</sup> API calls for every (resource, scope<sup>[4](#footnote4)</sup>) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day: (a) <= 1s if `scope=resource` (b) <= 30s<sup>[5](#footnote5)</sup> otherwise (if `scope=namespace` or `scope=cluster`) |
<a name="footnote1">\[1\]</a> By latency of API call in this doc we mean time <a name="footnote1">\[1\]</a> The SLI only measures latency incurred by the processing
from the moment when apiserver gets the request to last byte of response sent time of the request. The processing time of a request is the moment when apiserver gets
to the user. the request to last byte of response sent to the user, excluding latency incurred by
webhooks and priority & fairness queue wait times.
<a name="footnote2">\[2\]</a> By mutating API calls we mean POST, PUT, DELETE <a name="footnote2">\[2\]</a> By mutating API calls we mean POST, PUT, DELETE
and PATCH. and PATCH.
@ -35,15 +36,15 @@ that users are fine with listing tens of thousands of objects taking more than
- As a user of vanilla Kubernetes, I want some guarantee how quickly I get the - As a user of vanilla Kubernetes, I want some guarantee how quickly I get the
response from an API call. response from an API call.
- As an administrator of Kubernetes cluster, if I know characteristics of my - As an administrator of Kubernetes cluster, if I know characteristics of my
external dependencies of apiserver (e.g custom admission plugins and webhooks) external dependencies of apiserver (e.g custom admission plugins, priority
I want to be able to provide guarantees for API calls latency to users of my & fairness configuration, and webhooks). I want to be able to provide
cluster. guarantees for API calls latency to users of my cluster.
### Other notes ### Other notes
- We obviously can’t give any guarantee in general, because cluster - We obviously can’t give any guarantee in general, because cluster
administrators are allowed to register custom admission plugins or webhooks, administrators are allowed to register custom admission plugins, webhooks,
which we don’t have any control about and they obviously impact API call and priority and fairness configurations, which we don’t have any control
latencies. about and they obviously impact API call latencies.
- As a result, we define the SLIs to be very generic (no matter how your - As a result, we define the SLIs to be very generic (no matter how your
cluster is set up), but we provide SLO only for default installations (where we cluster is set up), but we provide SLO only for default installations (where we
have control over what apiserver is doing). This doesn’t provide a false have control over what apiserver is doing). This doesn’t provide a false
@ -72,6 +73,8 @@ that all `core` components communicate with apiserver using protocol buffers.
stale data (being served from cache) and the SLO again has to be satisfied stale data (being served from cache) and the SLO again has to be satisfied
independently of that. This makes the careful choice of requests in tests independently of that. This makes the careful choice of requests in tests
important. important.
- The SLI & SLO excludes latency incurred by factors that are outside our control, specifically
from webhooks (1.23+) and API priority & fairness queue wait times (1.27+).
### TODOs ### TODOs
- We may consider treating `non-namespaced` resources as a separate bucket in - We may consider treating `non-namespaced` resources as a separate bucket in

4
sig-scalability/slos/slos.md
View File

@ -114,8 +114,8 @@ __TODO: Cluster churn should be moved to scalability thresholds.__
| Status | SLI | SLO | User stories, test scenarios, ... | | Status | SLI | SLO | User stories, test scenarios, ... |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| __Official__ | Latency of mutating API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= 1s | [Details](./api_call_latency.md) | | __Official__ | Latency of processing mutating API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= 1s | [Details](./api_call_latency.md) |
| __Official__ | Latency of non-streaming read-only API calls for every (resource, scope) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> (a) <= 1s if `scope=resource` (b) <= 30s otherwise (if `scope=namespace` or `scope=cluster`) | [Details](./api_call_latency.md) | | __Official__ | Latency of processing non-streaming read-only API calls for every (resource, scope) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> (a) <= 1s if `scope=resource` (b) <= 30s otherwise (if `scope=namespace` or `scope=cluster`) | [Details](./api_call_latency.md) |
| __Official__ | Startup latency of schedulable stateless pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= 5s | [Details](./pod_startup_latency.md) | | __Official__ | Startup latency of schedulable stateless pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= 5s | [Details](./pod_startup_latency.md) |
| __WIP__ | Startup latency of schedulable stateful pods, excluding time to pull images, run init containers, provision volumes (in delayed binding mode) and unmount/detach volumes (from previous pod if needed), measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= X where X depends on storage provider | [Details](./pod_startup_latency.md) | | __WIP__ | Startup latency of schedulable stateful pods, excluding time to pull images, run init containers, provision volumes (in delayed binding mode) and unmount/detach volumes (from previous pod if needed), measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= X where X depends on storage provider | [Details](./pod_startup_latency.md) |
| __WIP__ | Latency of programming in-cluster load balancing mechanism (e.g. iptables), measured from when service spec or list of its `Ready` pods change to when it is reflected in load balancing mechanism, measured as 99th percentile over last 5 minutes aggregated across all programmers | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= X | [Details](./network_programming_latency.md) | | __WIP__ | Latency of programming in-cluster load balancing mechanism (e.g. iptables), measured from when service spec or list of its `Ready` pods change to when it is reflected in load balancing mechanism, measured as 99th percentile over last 5 minutes aggregated across all programmers | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= X | [Details](./network_programming_latency.md) |