From 5a275c424c63ebeef15c18e8cb6d565a05fd8053 Mon Sep 17 00:00:00 2001 From: wojtekt Date: Mon, 27 Aug 2018 11:05:46 +0200 Subject: [PATCH] Make SLOs page more clear --- sig-scalability/slos/api_call_latency.md | 23 ++++++++++++++ .../slos/api_extensions_latency.md | 8 +++++ sig-scalability/slos/pod_startup_latency.md | 13 ++++++++ sig-scalability/slos/slos.md | 31 +++---------------- sig-scalability/slos/system_throughput.md | 6 ++++ sig-scalability/slos/watch_latency.md | 6 ++++ 6 files changed, 60 insertions(+), 27 deletions(-) diff --git a/sig-scalability/slos/api_call_latency.md b/sig-scalability/slos/api_call_latency.md index 9f40924dc..69af0f135 100644 --- a/sig-scalability/slos/api_call_latency.md +++ b/sig-scalability/slos/api_call_latency.md @@ -1,5 +1,28 @@ ## API call latency SLIs/SLOs details +### Definition + +| Status | SLI | SLO | +| --- | --- | --- | +| __Official__ | Latency[1](#footnote1) of mutating[2](#footnote2) API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day <= 1s | +| __Official__ | Latency[1](#footnote1) of non-streaming read-only[3](#footnote3) API calls for every (resource, scope[4](#footnote4)) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day (a) <= 1s if `scope=resource` (b) <= 5s if `scope=namespace` (c) <= 30s if `scope=cluster` | + +\[1\]By latency of API call in this doc we mean time +from the moment when apiserver gets the request to last byte of response sent +to the user. + +\[2\]By mutating API calls we mean POST, PUT, DELETE +and PATCH. + +\[3\]By non-streaming read-only API calls we mean GET +requests without `watch=true` option set. (Note that in Kubernetes internally +it translates to both GET and LIST calls). + +\[4\]A scope of a request can be either (a) `resource` +if the request is about a single object, (b) `namespace` if it is about objects +from a single namespace or (c) `cluster` if it spawns objects from multiple +namespaces. + ### User stories - As a user of vanilla Kubernetes, I want some guarantee how quickly I get the response from an API call. diff --git a/sig-scalability/slos/api_extensions_latency.md b/sig-scalability/slos/api_extensions_latency.md index 2681422cf..c5490ec19 100644 --- a/sig-scalability/slos/api_extensions_latency.md +++ b/sig-scalability/slos/api_extensions_latency.md @@ -1,5 +1,13 @@ ## API call extension points latency SLIs details +### Definition + +| Status | SLI | +| --- | --- | +| WIP | Admission latency for each admission plugin type, measured as 99th percentile over last 5 minutes | +| WIP | Webhook call latency for each webhook type, measured as 99th percentile over last 5 minutes +| WIP | Initializer latency for each initializer, measured as 99th percentile over last 5 minutes | + ### User stories - As an administrator, if API calls are slow, I would like to know if this is because slow extension points (admission plugins, webhooks, initializers) and diff --git a/sig-scalability/slos/pod_startup_latency.md b/sig-scalability/slos/pod_startup_latency.md index f8943a45d..04fdd63b0 100644 --- a/sig-scalability/slos/pod_startup_latency.md +++ b/sig-scalability/slos/pod_startup_latency.md @@ -1,5 +1,18 @@ ## Pod startup latency SLI/SLO details +### Definition + +| Status | SLI | SLO | +| --- | --- | --- | +| __Official__ | Startup latency of stateless[1](#footnote1) and schedulable[2](#footnote2) pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day <= 5s | + +[1\]A `stateless pod` is defined as a pod that doesn't +mount volumes with sources other than secrets, config maps, downward API and +empty dir. + +[2\]By schedulable pod we mean a pod that can be +scheduled in the cluster without causing any preemption. + ### User stories - As a user of vanilla Kubernetes, I want some guarantee how quickly my pods will be started. diff --git a/sig-scalability/slos/slos.md b/sig-scalability/slos/slos.md index 59d0d9b08..6f7ffff0d 100644 --- a/sig-scalability/slos/slos.md +++ b/sig-scalability/slos/slos.md @@ -100,37 +100,14 @@ Prerequisite: Kubernetes cluster is available and serving. | Status | SLI | SLO | User stories, test scenarios, ... | | --- | --- | --- | --- | -| __Official__ | Latency[1](#footnote1) of mutating[2](#footnote2) API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day[3](#footnote3) <= 1s | [Details](./api_call_latency.md) | -| __Official__ | Latency[1](#footnote1) of non-streaming read-only[4](#footnote3) API calls for every (resource, scope[5](#footnote4)) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day (a) <= 1s if `scope=resource` (b) <= 5s if `scope=namespace` (c) <= 30s if `scope=cluster` | [Details](./api_call_latency.md) | -| __Official__ | Startup latency of stateless[6](#footnode6) and schedulable[7](#footnote7) pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day <= 5s | [Details](./pod_startup_latency.md) | +| __Official__ | Latency of mutating API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day[1](#footnote1) <= 1s | [Details](./api_call_latency.md) | +| __Official__ | Latency of non-streaming read-only API calls for every (resource, scope pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day[1](#footnote1) (a) <= 1s if `scope=resource` (b) <= 5s if `scope=namespace` (c) <= 30s if `scope=cluster` | [Details](./api_call_latency.md) | +| __Official__ | Startup latency of stateless and schedulable pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day[1](#footnote1) <= 5s | [Details](./pod_startup_latency.md) | -\[1\]By latency of API call in this doc we mean time -from the moment when apiserver gets the request to last byte of response sent -to the user. - -\[2\]By mutating API calls we mean POST, PUT, DELETE -and PATCH. - -\[3\] For the purpose of visualization it will be a +\[1\] For the purpose of visualization it will be a sliding window. However, for the purpose of reporting the SLO, it means one point per day (whether SLO was satisfied on a given day or not). -\[4\]By non-streaming read-only API calls we mean GET -requests without `watch=true` option set. (Note that in Kubernetes internally -it translates to both GET and LIST calls). - -\[5\]A scope of a request can be either (a) `resource` -if the request is about a single object, (b) `namespace` if it is about objects -from a single namespace or (c) `cluster` if it spawns objects from multiple -namespaces. - -[6\]A `stateless pod` is defined as a pod that doesn't -mount volumes with sources other than secrets, config maps, downward API and -empty dir. - -[7\]By schedulable pod we mean a pod that can be -scheduled in the cluster without causing any preemption. - ### Burst SLIs/SLOs | Status | SLI | SLO | User stories, test scenarios, ... | diff --git a/sig-scalability/slos/system_throughput.md b/sig-scalability/slos/system_throughput.md index 5691b46df..eb3fc6afd 100644 --- a/sig-scalability/slos/system_throughput.md +++ b/sig-scalability/slos/system_throughput.md @@ -1,5 +1,11 @@ ## System throughput SLI/SLO details +### Definition + +| Status | SLI | SLO | +| --- | --- | --- | +| WIP | Time to start 30\*#nodes pods, measured from test scenario start until observing last Pod as ready | Benchmark: when all images present on all Nodes, 99th percentile <= X minutes | + ### User stories - As a user, I want a guarantee that my workload of X pods can be started within a given time diff --git a/sig-scalability/slos/watch_latency.md b/sig-scalability/slos/watch_latency.md index 2e698b4bc..21324e23d 100644 --- a/sig-scalability/slos/watch_latency.md +++ b/sig-scalability/slos/watch_latency.md @@ -1,5 +1,11 @@ ## Watch latency SLI details +### Definition + +| Status | SLI | +| --- | --- | +| WIP | Watch latency for every resource, (from the moment when object is stored in database to when it's ready to be sent to all watchers), measured as 99th percentile over last 5 minutes | + ### User stories - As an administrator, if Kubernetes is slow, I would like to know if the root cause of it is slow api-machinery (slow watch) or something farther the path