* apiserver: add latency tracker for priority & fairness queue wait time
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* apiserver: exclude priority & fairness wait times to SLO/SLI latency metrics
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* apiserver: update TestLatencyTrackersFrom to check latency from PriorityAndFairnessTracker
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: add helper function observeQueueWaitTime to consolidate metric and latency tracker calls
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: replace time.Now() / time.Since() with clock.Now() / clock.Since() for better testability
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: add unit test TestQueueWaitTimeLatencyTracker to validate queue wait times recorded by latency tracker
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
---------
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Kubernetes-commit: ee18f602523e11a80823a659bed8f70f98a12914
Add new kube-apiserver SLI metric better reflecting that the metric is
an SLI and not an SLO and deprecate the existing
apiserver_request_slo_duration_seconds in 1.27. Although the metric is
still in alpha, we prefer deprecating it for one release since it is a
critical metric used for SLOs and to make sure that users that are using
it have time to make the transition.
Going forward we prefer going with SLI specific metrics, we will use
_sli_ instead of _slo_ so for consistency purposes.
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
Kubernetes-commit: 1493da92d9513e383f8382c7e80316a3fa6c94fa
this is specifically so that we have more structured information when
the metric is parsed and stored as a stable metric. This change does not
change the name of the actual metrics.
Change-Id: I861482401ad9a0ae12306b93abf91d6f76d7a407
Kubernetes-commit: 178e57c17b66eb572a961690bd10782aeb3c3582
Deprecate apiserver_longrunning_guage and apiserver_registered_watchers
Update
Fix name
update
Add back empty space
Update the deprecated version to 1.23
Kubernetes-commit: e2b6816953c9c52b98c129a3e9c2dac4ee925ef0
We've dropped the content-type field since it is effectively unbounded
(we had a sec-vuln about this before actually). We retain all other
fields, despite their unboundedness due to the fact that we can now
explicitly set bounds on label values.
Change-Id: Icc483fc6a17ea6382928f4448643cda6f3e21adb
Kubernetes-commit: cfd00de6866e636332bdcd3f46d6d2ffd8d2bc88
Aborted requests are the ones that were disrupted with http.ErrAbortHandler.
For example, the timeout handler will panic with http.ErrAbortHandler when a response to the client has been already sent
and the timeout elapsed.
Additionally, a new metric requestAbortsTotal was defined to count aborted requests. The new metric allows for aggregation for each group, version, verb, resource, subresource and scope.
Kubernetes-commit: 057986e32c1bb7284b0edbc161f0380f1548492f
- A: the timestamp a particular filter starts executing for a request.
- B: the timestamp the particular filter has completed and the next handler
in the chain has started executing.
We define filter latency as `B - A`. Introduce a new metric to track filter latency:
"apiserver_request_filter_duration_seconds"
This measures request filter latency distribution in seconds, for each filter type.
Kubernetes-commit: 99df0b15816cd59ed548235f33c7ab2add940eba
apiserver_request_duration_seconds does not take into account the
time a request spends in the server filters. If a filter takes longer
then the latency incurred will not be reflected in the apiserver
latency metrics.
For example, the amount of time a request spends in priority and
fairness machineries or in shuffle queues will not be accounted for.
- Add a server filter that attaches request received timestamp to the
request context very early in in the handler chain (as soon as
net/http hands over control to us).
- Use the above received timestamp in the apiserver latency metrics
apiserver_request_duration_seconds.
- Use the above received timestamp in the audit layer to set
RequestReceivedTimestamp.
Kubernetes-commit: d74ab9e1a4929be208d4529fd12b76d3fcd5d546
Introduce min, average, and standard deviation for the number of
executing mutating and readOnly requests.
Introduce min, max, average, and standard deviation for the number
waiting and number waiting per priority level.
Later:
Revised to use a series of windows
Use three individuals instead of array of powers
Later:
Add coarse queue count metrics, removed windowed avg and stddev
Add metrics for number of queued mutating and readOnly requests,
to complement metrics for number executing.
Later:
Removed windowed average and standard deviation because consumers can
derive such from integrals of consumer's chosen window.
Also replaced "requestKind" Prometheus label with "request_kind".
Later:
Revised to focus on sampling
Make the clock intrinsic to a TimedObserver
... so that the clock can be read while holding the observer's lock;
otherwise, forward progress is not guaranteed (and violations were
observed in testing).
Bug fixes and histogram buckets revision
SetX1 to 1 when queue length limit is zero, beause dividing by zero is nasty.
Remove obsolete argument in gen_test.go.
Add a bucket boundary at 0 for sample-and-water-mark histograms, to
distinguish zeroes from non-zeros.
This includes adding Integrator test.
Simplified test code.
More pervasively used "ctlr" instead of "ctl" as abbreviation for
"controller".
Kubernetes-commit: 57ecea22296797a93b0157169db0ff2e477f58d0
Andrew Sy Kim
Kevin Delgado
kkkkun
Damien Grisonnet
Paweł Banaszewski
Wojciech Tyczyński
Han Kang
Alexander Zielenski
Davanum Srinivas
Artur Żyliński
Abu Kashem
Reza Karimi
jyz0309
Yan Huang
Harry Zhang
tiloso
Daniel Smith
Jordan Liggitt
卢振兴10069964
Kagaya
Jiaxin Shan
yoyinzyc
Lukasz Szaszkiewicz
Marcus Alder
Mike Spreitzer