From 6ce3dd22371ec661005acd09b215a1842c72d3c9 Mon Sep 17 00:00:00 2001 From: Mike Spreitzer Date: Mon, 11 Apr 2022 22:03:12 -0400 Subject: [PATCH] Update APF metrics again --- .../cluster-administration/flow-control.md | 57 +++++++++++++++++-- 1 file changed, 53 insertions(+), 4 deletions(-) diff --git a/content/en/docs/concepts/cluster-administration/flow-control.md b/content/en/docs/concepts/cluster-administration/flow-control.md index 9e8f2a7923..68419b8231 100644 --- a/content/en/docs/concepts/cluster-administration/flow-control.md +++ b/content/en/docs/concepts/cluster-administration/flow-control.md @@ -471,11 +471,15 @@ poorly-behaved workloads that may be harming system health. requests, broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `request_kind` (which takes on the values `mutating` and `readOnly`). The observations are made - periodically at a high rate. + periodically at a high rate. Each observed value is a ratio, + between 0 and 1, of a number of requests divided by the + corresponding limit on the number of requests (queue length limit + for waiting and concurrency limit for executing). * `apiserver_flowcontrol_read_vs_write_request_count_watermarks` is a histogram vector of high or low water marks of the number of - requests broken down by the labels `phase` (which takes on the + requests (divided by the corresponding limit to get a ratio in the + range 0 to 1) broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `request_kind` (which takes on the values `mutating` and `readOnly`); the label `mark` takes on values `high` and `low`. The water marks are accumulated over @@ -502,11 +506,15 @@ poorly-behaved workloads that may be harming system health. values `waiting` and `executing`) and `priority_level`. Each histogram gets observations taken periodically, up through the last activity of the relevant sort. The observations are made at a high - rate. + rate. Each observed value is a ratio, between 0 and 1, of a number + of requests divided by the corresponding limit on the number of + requests (queue length limit for waiting and concurrency limit for + executing). * `apiserver_flowcontrol_priority_level_request_count_watermarks` is a histogram vector of high or low water marks of the number of - requests broken down by the labels `phase` (which takes on the + requests (divided by the corresponding limit to get a ratio in the + range 0 to 1) broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `priority_level`; the label `mark` takes on values `high` and `low`. The water marks are accumulated over windows bounded by the times when an observation @@ -514,6 +522,31 @@ poorly-behaved workloads that may be harming system health. `apiserver_flowcontrol_priority_level_request_count_samples`. These water marks show the range of values that occurred between samples. +* `apiserver_flowcontrol_priority_level_seat_count_samples` is a + histogram vector of observations of the utilization of a priority + level's concurrency limit, broken down by `priority_level`. This + utilization is the fraction (number of seats occupied) / + (concurrency limit). This metric considers all stages of execution + (both normal and the extra delay at the end of a write to cover for + the corresponding notification work) of all requests except WATCHes; + for those it considers only the initial stage that delivers + notifications of pre-existing objects. Each histogram in the vector + is also labeled with `phase: executing` (there is no seat limit for + the waiting phase). Each histogram gets observations taken + periodically, up through the last activity of the relevant sort. + The observations + are made at a high rate. + +* `apiserver_flowcontrol_priority_level_seat_count_watermarks` is a + histogram vector of high or low water marks of the utilization of a + priority level's concurrency limit, broken down by `priority_level` + and `mark` (which takes on values `high` and `low`). Each histogram + in the vector is also labeled with `phase: executing` (there is no + seat limit for the waiting phase). The water marks are accumulated + over windows bounded by the times when an observation was added to + `apiserver_flowcontrol_priority_level_seat_count_samples`. These + water marks show the range of values that occurred between samples. + * `apiserver_flowcontrol_request_queue_length_after_enqueue` is a histogram vector of queue lengths for the queues, broken down by the labels `priority_level` and `flow_schema`, as sampled by the @@ -556,6 +589,22 @@ poorly-behaved workloads that may be harming system health. and `priority_level` (indicating the one to which the request was assigned). +* `apiserver_flowcontrol_watch_count_samples` is a histogram vector of + the number of active WATCH requests relevant to a given write, + broken down by `flow_schema` and `priority_level`. + +* `apiserver_flowcontrol_work_estimated_seats` is a histogram vector + of the number of estimated seats (maximum of initial and final stage + of execution) associated with requests, broken down by `flow_schema` + and `priority_level`. + +* `apiserver_flowcontrol_request_dispatch_no_accommodation_total` is a + counter vec of the number of events that in principle could have led + to a request being dispatched but did not, due to lack of available + concurrency, broken down by `flow_schema` and `priority_level`. The + relevant sorts of events are arrival of a request and completion of + a request. + ### Debug endpoints When you enable the API Priority and Fairness feature, the `kube-apiserver`