* Fix typo

* Update flow-control.md
This commit is contained in:
Grant He 2023-05-09 18:32:58 -07:00 committed by GitHub
parent 93b47ffd34
commit a32bff3813
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 40 additions and 54 deletions

View File

@ -107,19 +107,18 @@ objects mentioned below.
### Seats Occupied by a Request ### Seats Occupied by a Request
The above description of concurrency management is the baseline story. The above description of concurrency management is the baseline story.
In it, requests have different durations but are counted equally at Requests have different durations but are counted equally at any given
any given moment when comparing against a priority level's concurrency moment when comparing against a priority level's concurrency limit. In
limit. In the baseline story, each request occupies one unit of the baseline story, each request occupies one unit of concurrency. The
concurrency. The word "seat" is used to mean one unit of concurrency, word "seat" is used to mean one unit of concurrency, inspired by the
inspired by the way each passenger on a train or aircraft takes up one way each passenger on a train or aircraft takes up one of the fixed
of the fixed supply of seats. supply of seats.
But some requests take up more than one seat. Some of these are **list** But some requests take up more than one seat. Some of these are **list**
requests that the server estimates will return a large number of requests that the server estimates will return a large number of
objects. These have been found to put an exceptionally heavy burden objects. These have been found to put an exceptionally heavy burden
on the server, among requests that take a similar amount of time to on the server. For this reason, the server estimates the number of objects
run. For this reason, the server estimates the number of objects that that will be returned and considers the request to take a number of seats
will be returned and considers the request to take a number of seats
that is proportional to that estimated number. that is proportional to that estimated number.
### Execution time tweaks for watch requests ### Execution time tweaks for watch requests
@ -294,10 +293,9 @@ HandSize | Queues | 1 elephant | 4 elephants | 16 elephants
### FlowSchema ### FlowSchema
A FlowSchema matches some inbound requests and assigns them to a A FlowSchema matches some inbound requests and assigns them to a
priority level. Every inbound request is tested against every priority level. Every inbound request is tested against FlowSchemas,
FlowSchema in turn, starting with those with numerically lowest --- starting with those with the numerically lowest `matchingPrecedence` and
which we take to be the logically highest --- `matchingPrecedence` and working upward. The first match wins.
working onward. The first match wins.
{{< caution >}} {{< caution >}}
Only the first matching FlowSchema for a given request matters. If multiple Only the first matching FlowSchema for a given request matters. If multiple
@ -311,7 +309,7 @@ ensure that no two FlowSchemas have the same `matchingPrecedence`.
A FlowSchema matches a given request if at least one of its `rules` A FlowSchema matches a given request if at least one of its `rules`
matches. A rule matches if at least one of its `subjects` *and* at least matches. A rule matches if at least one of its `subjects` *and* at least
one of its `resourceRules` or `nonResourceRules` (depending on whether the one of its `resourceRules` or `nonResourceRules` (depending on whether the
incoming request is for a resource or non-resource URL) matches the request. incoming request is for a resource or non-resource URL) match the request.
For the `name` field in subjects, and the `verbs`, `apiGroups`, `resources`, For the `name` field in subjects, and the `verbs`, `apiGroups`, `resources`,
`namespaces`, and `nonResourceURLs` fields of resource and non-resource rules, `namespaces`, and `nonResourceURLs` fields of resource and non-resource rules,
@ -319,12 +317,11 @@ the wildcard `*` may be specified to match all values for the given field,
effectively removing it from consideration. effectively removing it from consideration.
A FlowSchema's `distinguisherMethod.type` determines how requests matching that A FlowSchema's `distinguisherMethod.type` determines how requests matching that
schema will be separated into flows. It may be schema will be separated into flows. It may be `ByUser`, in which one requesting
either `ByUser`, in which case one requesting user will not be able to starve user will not be able to starve other users of capacity; `ByNamespace`, in which
other users of capacity, or `ByNamespace`, in which case requests for resources requests for resources in one namespace will not be able to starve requests for
in one namespace will not be able to starve requests for resources in other resources in other namespaces of capacity; or blank (or `distinguisherMethod` may be
namespaces of capacity, or it may be blank (or `distinguisherMethod` may be omitted entirely), in which all requests matched by this FlowSchema will be
omitted entirely), in which case all requests matched by this FlowSchema will be
considered part of a single flow. The correct choice for a given FlowSchema considered part of a single flow. The correct choice for a given FlowSchema
depends on the resource and your particular environment. depends on the resource and your particular environment.
@ -434,7 +431,7 @@ for an annotation with key `apf.kubernetes.io/autoupdate-spec`. If
there is such an annotation and its value is `true` then the there is such an annotation and its value is `true` then the
kube-apiservers control the object. If there is such an annotation kube-apiservers control the object. If there is such an annotation
and its value is `false` then the users control the object. If and its value is `false` then the users control the object. If
neither of those condtions holds then the `metadata.generation` of the neither of those conditions holds then the `metadata.generation` of the
object is consulted. If that is 1 then the kube-apiservers control object is consulted. If that is 1 then the kube-apiservers control
the object. Otherwise the users control the object. These rules were the object. Otherwise the users control the object. These rules were
introduced in release 1.22 and their consideration of introduced in release 1.22 and their consideration of
@ -513,13 +510,13 @@ poorly-behaved workloads that may be harming system health.
broken down by the labels `flow_schema` (indicating the one that broken down by the labels `flow_schema` (indicating the one that
matched the request), `priority_level` (indicating the one to which matched the request), `priority_level` (indicating the one to which
the request was assigned), and `reason`. The `reason` label will be the request was assigned), and `reason`. The `reason` label will be
have one of the following values: one of the following values:
* `queue-full`, indicating that too many requests were already * `queue-full`, indicating that too many requests were already
queued, queued.
* `concurrency-limit`, indicating that the * `concurrency-limit`, indicating that the
PriorityLevelConfiguration is configured to reject rather than PriorityLevelConfiguration is configured to reject rather than
queue excess requests, or queue excess requests.
* `time-out`, indicating that the request was still in the queue * `time-out`, indicating that the request was still in the queue
when its queuing time limit expired. when its queuing time limit expired.
* `cancelled`, indicating that the request is not purge locked * `cancelled`, indicating that the request is not purge locked
@ -527,9 +524,7 @@ poorly-behaved workloads that may be harming system health.
* `apiserver_flowcontrol_dispatched_requests_total` is a counter * `apiserver_flowcontrol_dispatched_requests_total` is a counter
vector (cumulative since server start) of requests that began vector (cumulative since server start) of requests that began
executing, broken down by the labels `flow_schema` (indicating the executing, broken down by `flow_schema` and `priority_level`.
one that matched the request) and `priority_level` (indicating the
one to which the request was assigned).
* `apiserver_current_inqueue_requests` is a gauge vector of recent * `apiserver_current_inqueue_requests` is a gauge vector of recent
high water marks of the number of queued requests, grouped by a high water marks of the number of queued requests, grouped by a
@ -545,23 +540,22 @@ poorly-behaved workloads that may be harming system health.
nanosecond, of the number of requests broken down by the labels nanosecond, of the number of requests broken down by the labels
`phase` (which takes on the values `waiting` and `executing`) and `phase` (which takes on the values `waiting` and `executing`) and
`request_kind` (which takes on the values `mutating` and `request_kind` (which takes on the values `mutating` and
`readOnly`). Each observed value is a ratio, between 0 and 1, of a `readOnly`). Each observed value is a ratio, between 0 and 1, of
number of requests divided by the corresponding limit on the number the number of requests divided by the corresponding limit on the
of requests (queue volume limit for waiting and concurrency limit number of requests (queue volume limit for waiting and concurrency
for executing). limit for executing).
* `apiserver_flowcontrol_current_inqueue_requests` is a gauge vector * `apiserver_flowcontrol_current_inqueue_requests` is a gauge vector
holding the instantaneous number of queued (not executing) requests, holding the instantaneous number of queued (not executing) requests,
broken down by the labels `priority_level` and `flow_schema`. broken down by `priority_level` and `flow_schema`.
* `apiserver_flowcontrol_current_executing_requests` is a gauge vector * `apiserver_flowcontrol_current_executing_requests` is a gauge vector
holding the instantaneous number of executing (not waiting in a holding the instantaneous number of executing (not waiting in a
queue) requests, broken down by the labels `priority_level` and queue) requests, broken down by `priority_level` and `flow_schema`.
`flow_schema`.
* `apiserver_flowcontrol_request_concurrency_in_use` is a gauge vector * `apiserver_flowcontrol_request_concurrency_in_use` is a gauge vector
holding the instantaneous number of occupied seats, broken down by holding the instantaneous number of occupied seats, broken down by
the labels `priority_level` and `flow_schema`. `priority_level` and `flow_schema`.
* `apiserver_flowcontrol_priority_level_request_utilization` is a * `apiserver_flowcontrol_priority_level_request_utilization` is a
histogram vector of observations, made at the end of each histogram vector of observations, made at the end of each
@ -587,11 +581,10 @@ poorly-behaved workloads that may be harming system health.
* `apiserver_flowcontrol_request_queue_length_after_enqueue` is a * `apiserver_flowcontrol_request_queue_length_after_enqueue` is a
histogram vector of queue lengths for the queues, broken down by histogram vector of queue lengths for the queues, broken down by
the labels `priority_level` and `flow_schema`, as sampled by the `priority_level` and `flow_schema`, as sampled by the enqueued requests.
enqueued requests. Each request that gets queued contributes one Each request that gets queued contributes one sample to its histogram,
sample to its histogram, reporting the length of the queue immediately reporting the length of the queue immediately after the request was added.
after the request was added. Note that this produces different Note that this produces different statistics than an unbiased survey would.
statistics than an unbiased survey would.
{{< note >}} {{< note >}}
An outlier value in a histogram here means it is likely that a single flow An outlier value in a histogram here means it is likely that a single flow
@ -655,13 +648,10 @@ poorly-behaved workloads that may be harming system health.
holding, for each priority level, the dynamic concurrency limit holding, for each priority level, the dynamic concurrency limit
derived in the last adjustment. derived in the last adjustment.
* `apiserver_flowcontrol_request_wait_duration_seconds` is a histogram * `apiserver_flowcontrol_request_wait_duration_seconds` is a histogram
vector of how long requests spent queued, broken down by the labels vector of how long requests spent queued, broken down by the labels
`flow_schema` (indicating which one matched the request), `flow_schema`, `priority_level`, and `execute`. The `execute` label
`priority_level` (indicating the one to which the request was indicates whether the request has started executing.
assigned), and `execute` (indicating whether the request started
executing).
{{< note >}} {{< note >}}
Since each FlowSchema always assigns requests to a single Since each FlowSchema always assigns requests to a single
@ -672,9 +662,7 @@ poorly-behaved workloads that may be harming system health.
* `apiserver_flowcontrol_request_execution_seconds` is a histogram * `apiserver_flowcontrol_request_execution_seconds` is a histogram
vector of how long requests took to actually execute, broken down by vector of how long requests took to actually execute, broken down by
the labels `flow_schema` (indicating which one matched the request) `flow_schema` and `priority_level`.
and `priority_level` (indicating the one to which the request was
assigned).
* `apiserver_flowcontrol_watch_count_samples` is a histogram vector of * `apiserver_flowcontrol_watch_count_samples` is a histogram vector of
the number of active WATCH requests relevant to a given write, the number of active WATCH requests relevant to a given write,
@ -686,16 +674,14 @@ poorly-behaved workloads that may be harming system health.
and `priority_level`. and `priority_level`.
* `apiserver_flowcontrol_request_dispatch_no_accommodation_total` is a * `apiserver_flowcontrol_request_dispatch_no_accommodation_total` is a
counter vec of the number of events that in principle could have led counter vector of the number of events that in principle could have led
to a request being dispatched but did not, due to lack of available to a request being dispatched but did not, due to lack of available
concurrency, broken down by `flow_schema` and `priority_level`. The concurrency, broken down by `flow_schema` and `priority_level`.
relevant sorts of events are arrival of a request and completion of
a request.
### Debug endpoints ### Debug endpoints
When you enable the API Priority and Fairness feature, the `kube-apiserver` When you enable the API Priority and Fairness feature, the `kube-apiserver`
serves the following additional paths at its HTTP[S] ports. serves the following additional paths at its HTTP(S) ports.
- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of - `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of
all the priority levels and the current state of each. You can fetch like this: all the priority levels and the current state of each. You can fetch like this:
@ -785,7 +771,7 @@ request, and it includes the following attributes.
execution of the request. execution of the request.
At higher levels of verbosity there will be log lines exposing details At higher levels of verbosity there will be log lines exposing details
of how APF handled the request, primarily for debug purposes. of how APF handled the request, primarily for debugging purposes.
### Response headers ### Response headers