a559cb8be1
Unify references to group resource in metrics
...
Skip apiserver_storage_objects as it's a stable metrics that doesn't
have "group" label. As defined in KEP-1209 adding a label is considered
a breaking change so this change should be done via deprecation and
introduction of new metric.
Tested via:
```
kubectl get --raw /metrics | grep -i apiservice | grep -v APIServiceRegistrationController | grep -v aggregator_unavailable_apiservice | python3 -c "import sys; print('\n'.join([k+' ' + v for k,v in {a.split('{')[0]:'{'+a.split('{')[1].split('}')[0]+'}' for a in sys.stdin.readlines() if '#' not in a and '_sum' not in a and '_bucket' not in a}.items()]))"
```
Before:
```
apiserver_cache_list_fetched_objects_total {index="",resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_cache_list_returned_objects_total {resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_cache_list_total {index="",resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_longrunning_requests {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_request_body_size_bytes_count {resource="apiservices.apiregistration.k8s.io",verb="create"}
apiserver_request_duration_seconds_count {component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_sli_duration_seconds_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_total {code="201",component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_response_sizes_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="LIST",version="v1"}
apiserver_selfrequest_total {resource="apiservices",subresource="",verb="POST"}
apiserver_storage_events_received_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_evaluated_objects_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_fetched_objects_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_returned_objects_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_objects {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_dispatched_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_received_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_initializations_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_resource_version {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_events_sizes_count {group="apiregistration.k8s.io",kind="APIService",version="v1"}
apiserver_watch_events_total {group="apiregistration.k8s.io",kind="APIService",version="v1"}
etcd_request_duration_seconds_count {operation="listWithCount",type="/registry/apiregistration.k8s.io/apiservices/"}
etcd_requests_total {operation="listWithCount",type="/registry/apiregistration.k8s.io/apiservices/"}
watch_cache_capacity {resource="apiservices.apiregistration.k8s.io"}
```
After:
```
apiserver_cache_list_fetched_objects_total {group="apiregistration.k8s.io",index="",resource="apiservices"}
apiserver_cache_list_returned_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_cache_list_total {group="apiregistration.k8s.io",index="",resource="apiservices"}
apiserver_longrunning_requests {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_request_body_size_bytes_count {group="apiregistration.k8s.io",resource="apiservices",verb="create"}
apiserver_request_duration_seconds_count {component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_sli_duration_seconds_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_total {code="201",component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_response_sizes_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_selfrequest_total {group="apiregistration.k8s.io",resource="apiservices",subresource="",verb="WATCH"}
apiserver_storage_events_received_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_evaluated_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_fetched_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_returned_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_objects {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_dispatched_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_cache_events_received_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_cache_initializations_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_cache_resource_version {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_events_sizes_count {group="apiregistration.k8s.io",resource="apiservices",version="v1"}
apiserver_watch_events_total {group="apiregistration.k8s.io",resource="apiservices",version="v1"}
etcd_bookmark_counts {group="apiregistration.k8s.io",resource="apiservices"}
etcd_request_duration_seconds_count {group="apiregistration.k8s.io",operation="listWithCount",resource="apiservices"}
etcd_requests_total {group="apiregistration.k8s.io",operation="listWithCount",resource="apiservices"}
watch_cache_capacity {group="apiregistration.k8s.io",resource="apiservices"}
```
Kubernetes-commit: f712b01ddb55f6569b930ca714499051ba8cb311
2025-05-19 11:41:09 +02:00
61451f57b1
refactor: Add request method constants
...
This avoids linter errors from using inline strings as http request
methods.
Kubernetes-commit: e81887276c65acccc5486f2ff69d8bc54ac3c6ca
2025-04-29 16:45:30 -07:00
15884a5fd0
feat: Add function to reset label allow list for apiserver endpoints metrics
...
Kubernetes-commit: d3795eaf76e5ecc632325cfb598533c4e3c31704
2024-10-30 09:28:24 -07:00
1c49f6c8ba
endpoints/metrics: define watchListLatencies metric and associated functions
...
Kubernetes-commit: a97f4b7a3123c9768ec7136b6ca32be926e16cd6
2023-09-19 03:05:37 +02:00
986c8832e5
apiserver: introduce larger buckets for request_filter_duration_seconds metric
...
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Kubernetes-commit: c607984379234249403354bf1f9a2e86a740caf0
2023-06-28 14:19:51 +00:00
53a2449e3c
apiserver: exclude APF queue wait time from SLO latency metrics ( #116420 )
...
* apiserver: add latency tracker for priority & fairness queue wait time
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* apiserver: exclude priority & fairness wait times to SLO/SLI latency metrics
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* apiserver: update TestLatencyTrackersFrom to check latency from PriorityAndFairnessTracker
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: add helper function observeQueueWaitTime to consolidate metric and latency tracker calls
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: replace time.Now() / time.Since() with clock.Now() / clock.Since() for better testability
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
* flowcontrol: add unit test TestQueueWaitTimeLatencyTracker to validate queue wait times recorded by latency tracker
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
---------
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Kubernetes-commit: ee18f602523e11a80823a659bed8f70f98a12914
2023-03-15 07:35:21 +00:00
c40024ba00
drop Enabled() checks for ServerSideFieldValidation feature gate
...
Kubernetes-commit: 2d5ceb9b15347b8bb5af360470b6346370157042
2023-01-24 17:48:31 +00:00
c465dcc9ec
Fix normal request metric verb correctly
...
Signed-off-by: kkkkun <scuzk373x@gmail.com>
Kubernetes-commit: 0b6e641810fb7a24a74b8068fb772c1bb1eb447c
2023-01-03 13:16:46 +00:00
aec915f887
Fix CONNECT requests from others requests
...
Signed-off-by: kkkkun <scuzk373x@gmail.com>
Kubernetes-commit: 973f7c49b887b00c43874ad1085237632efcc300
2022-12-22 11:52:18 +08:00
fc3eec104b
Cleanup reportedVerb only from
...
Kubernetes-commit: 42f72a3fe9dc27c0b0f10fd7dc4ab4cb6f901314
2022-08-14 13:53:25 +08:00
9a8e2de22e
metrics: improve apiserver SLI metric name
...
Add new kube-apiserver SLI metric better reflecting that the metric is
an SLI and not an SLO and deprecate the existing
apiserver_request_slo_duration_seconds in 1.27. Although the metric is
still in alpha, we prefer deprecating it for one release since it is a
critical metric used for SLOs and to make sure that users that are using
it have time to make the transition.
Going forward we prefer going with SLI specific metrics, we will use
_sli_ instead of _slo_ so for consistency purposes.
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
Kubernetes-commit: 1493da92d9513e383f8382c7e80316a3fa6c94fa
2022-09-22 19:08:34 +02:00
e21a3f224f
Added simple tests and changed lazyVerb String method to the same as httplog uses
...
Kubernetes-commit: b31339231314ae8a72481cfda45539e35097450a
2022-11-07 17:13:26 +00:00
baa5a012e9
Lock ServerSideApply feature to true
...
Kubernetes-commit: 57c95fbfa12bc04456330d8b0b29f333106cf156
2022-09-27 11:13:00 +02:00
d5fac85195
clean-up apiserver metrics and use subsystem
...
this is specifically so that we have more structured information when
the metric is parsed and stored as a stable metric. This change does not
change the name of the actual metrics.
Change-Id: I861482401ad9a0ae12306b93abf91d6f76d7a407
Kubernetes-commit: 178e57c17b66eb572a961690bd10782aeb3c3582
2022-07-29 12:51:29 -07:00
48eb70e1d1
benchmark and metrics for new timestamp transformer comparison
...
add proper metrics
rename & improve documentation for path metric dimension
Kubernetes-commit: 40343793f7b9787b2d4b88f0a0439ce9e538075a
2022-03-01 14:24:12 -08:00
ad01c712d9
Update etcdRequestLatency metrics bucket size
...
Kubernetes-commit: fb372d07982463984dfdab7cf24d937b91b0bb83
2022-01-09 23:45:09 +08:00
30571358f5
Switch to v3 of github.com/emicklei/go-restful
...
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: ab690750df1d27409d31fd270d77b4390bac4431
2022-06-02 10:03:29 -04:00
a414002089
cleanup deprecated metrics and usages
...
Kubernetes-commit: f223b900907b71431d7b6ceefa1642bb44fd9d84
2022-06-01 11:55:14 -07:00
5e26f39c93
remove deprecated metric and promote the replacement to STABLE
...
Kubernetes-commit: 731397086b763b9a951a414d37018f7e4f03b99c
2022-05-31 09:05:31 -07:00
62aa70cd37
Fix requests scope classification
...
create (POST) custom resources should be scope:resource
Kubernetes-commit: 690c912369a6ca7f35363a307a843e91cadd7f6e
2022-05-12 14:27:51 +02:00
826d18bf6e
Track field validation in metrics
...
Kubernetes-commit: 0c432ccf58b968040bcf6009bc255eeff0092f5a
2022-03-29 18:38:58 +00:00
23c7d60c29
Deprecate apiserver_dropped_requests_total metric
...
Kubernetes-commit: b106b2621c454b8f38b9dc28935994c33426ee75
2022-03-25 14:58:41 +01:00
abc4243fac
Record dropped requests in apiserver_request_total metric
...
Kubernetes-commit: 14396349954be57abea7162d7fe091e58a80ec4b
2022-03-23 16:16:36 +01:00
72aa2c42fc
refactor: rename webhook duration tracker
...
Kubernetes-commit: 4a9b9028153c6984b9cf69067cc0a1aa12a00e73
2022-02-01 15:44:59 -05:00
9c968f178e
Fix logging resource-scoped watch requests as GET requests.
...
Kubernetes-commit: 43e924a5b613850f43c02ac5a74d3e81c1292f3d
2021-11-25 08:55:45 +01:00
78c055e084
Added requestSloLatencies metric
...
Kubernetes-commit: 0afa569499d480df4977568454a50790891860f5
2021-10-25 22:19:24 +00:00
0df77d8de7
Changed buckets for apiserver_request_duration_seconds metric
...
Kubernetes-commit: a8bd20a5af1174ce9f9a185b6d49f025a1045b03
2021-10-29 15:37:27 +00:00
193bf2c49a
making some apiserver metrics stable
...
Kubernetes-commit: 894e31f41aed5e6db4275bf8cf89fdd62fe25b1e
2021-11-02 17:53:53 +00:00
6bb07771dd
Fix bug: Specical GET should be transformed to WATCH
...
Kubernetes-commit: 5f98d8f79855af648a610b21d71583a6540a58cd
2021-11-04 21:33:30 +08:00
6013da2b94
apiserver: wrap ResponseWriter using abstraction
...
Kubernetes-commit: 0d50c969c587c8a6c16e0962118305ac652c5a6b
2021-09-20 11:21:56 -04:00
8a6934a20b
add new metric
...
Signed-off-by: jyz0309 <45495947@qq.com>
add comment
Signed-off-by: jyz0309 <45495947@qq.com>
add promlint
Signed-off-by: jyz0309 <45495947@qq.com>
address comment
Signed-off-by: jyz0309 <45495947@qq.com>
go fmt code
Signed-off-by: jyz0309 <45495947@qq.com>
Kubernetes-commit: 97ba475941f64f7ab41a92c51adeb4401146eed1
2021-07-20 19:28:33 +08:00
c6719af51e
Fix metrics reporting for the deprecated watch path
...
Kubernetes-commit: d9d41b70f64da65525d769ab279e3930d59435fe
2021-08-05 16:17:13 +02:00
8febdc018a
Deprecate apiserver_longrunning_guage and apiserver_registered_watchers
...
Deprecate apiserver_longrunning_guage and apiserver_registered_watchers
Update
Fix name
update
Add back empty space
Update the deprecated version to 1.23
Kubernetes-commit: e2b6816953c9c52b98c129a3e9c2dac4ee925ef0
2021-07-19 23:18:35 -07:00
0321f0c67e
fix CleanScope so we can resolve correct verb for apiserver_request_terminations_total
...
Kubernetes-commit: a3f57886a267c28d5a0193ca7193d4de9e9212d9
2021-07-07 19:07:49 -07:00
c94ba7a71a
Fix staticcheck in cluster & k8s.io/{apimachinery,apiserver}
...
Kubernetes-commit: c5d70e9db1e8f36e5eb6dae6ca72ad62c1c837d9
2021-06-05 22:15:15 +02:00
a08d7d5ddf
Make logged verbs match metric-reported verbs
...
Kubernetes-commit: 54fde5d119f8bb55aec18cb5aef01b6ea58dfea0
2021-06-16 17:08:54 -07:00
754e90a0a3
Promote apiserver_requested_deprecated_apis metric to stable
...
Kubernetes-commit: 7877539e3095669629e81633019836d2cdb7c368
2021-05-11 16:24:58 -04:00
549cbbf8de
fix broken link in some files
...
Kubernetes-commit: b29a5fb0746f772b38da570cd8fdc77396ffca31
2021-04-13 08:43:24 +08:00
b84fa01322
add more context to post timeout request activities
...
Kubernetes-commit: 66587afb60b10569aa4ed35d53a4a91fcc40abd2
2021-04-09 12:09:35 -04:00
fdbe19987d
promote apiserver_request_duration_seconds to STABLE
...
Change-Id: I1b050b812738719aedd7ac6f4794ec742812e12d
Kubernetes-commit: 69e3d89a38b198d6261e675a31853583f3cce8f7
2021-03-07 17:14:36 -08:00
1cd0838bc2
bump apiserver_request_total to STABLE status
...
We've dropped the content-type field since it is effectively unbounded
(we had a sec-vuln about this before actually). We retain all other
fields, despite their unboundedness due to the fact that we can now
explicitly set bounds on label values.
Change-Id: Icc483fc6a17ea6382928f4448643cda6f3e21adb
Kubernetes-commit: cfd00de6866e636332bdcd3f46d6d2ffd8d2bc88
2021-03-04 07:52:00 -08:00
2422ff2d3b
fix staticcheck for k8s.io/apiserver/pkg/endpoints
...
Kubernetes-commit: 9b02c89acd1c2c5664fdaa98884d17dfd5dad135
2021-02-27 15:37:00 +08:00
dfad5032fb
Fix ALPHA stability level reference link
...
Kubernetes-commit: e01a21469b9719f7d0e84021c032cd8f0016b5d2
2021-01-31 15:37:07 -08:00
74c8267082
add context to metrics in apiserver/endpoint
...
Kubernetes-commit: 266d67bd5124725befe775d580289b67a59f496c
2021-01-20 13:13:00 -08:00
383e99f309
stops puting a stacktrace of aborted requests in the logs
...
Aborted requests are the ones that were disrupted with http.ErrAbortHandler.
For example, the timeout handler will panic with http.ErrAbortHandler when a response to the client has been already sent
and the timeout elapsed.
Additionally, a new metric requestAbortsTotal was defined to count aborted requests. The new metric allows for aggregation for each group, version, verb, resource, subresource and scope.
Kubernetes-commit: 057986e32c1bb7284b0edbc161f0380f1548492f
2020-09-24 10:07:24 +02:00
d09cfb1050
change bucket from 0.0s to 0.0001s
...
Kubernetes-commit: 16ac2d84cf125499651dac434a793f4c76e32640
2020-10-30 09:18:40 -04:00
9b40000eaa
Measure how much time a request spends in server filter(s):
...
- A: the timestamp a particular filter starts executing for a request.
- B: the timestamp the particular filter has completed and the next handler
in the chain has started executing.
We define filter latency as `B - A`. Introduce a new metric to track filter latency:
"apiserver_request_filter_duration_seconds"
This measures request filter latency distribution in seconds, for each filter type.
Kubernetes-commit: 99df0b15816cd59ed548235f33c7ab2add940eba
2020-10-13 18:05:44 -04:00
ab7c669cad
apiserver self request metric
...
Kubernetes-commit: 6f85e5c11179040b3618bcce9abe4e8f769323ad
2020-09-23 19:54:03 +00:00
50cce843c9
Take into account latency incurred in server filters
...
apiserver_request_duration_seconds does not take into account the
time a request spends in the server filters. If a filter takes longer
then the latency incurred will not be reflected in the apiserver
latency metrics.
For example, the amount of time a request spends in priority and
fairness machineries or in shuffle queues will not be accounted for.
- Add a server filter that attaches request received timestamp to the
request context very early in in the handler chain (as soon as
net/http hands over control to us).
- Use the above received timestamp in the apiserver latency metrics
apiserver_request_duration_seconds.
- Use the above received timestamp in the audit layer to set
RequestReceivedTimestamp.
Kubernetes-commit: d74ab9e1a4929be208d4529fd12b76d3fcd5d546
2020-09-18 16:46:07 -04:00
58a16cc414
Fix verbs reporting in kube-apiserver metrics
...
Kubernetes-commit: 140631779686175456dabfbb15b8958de7112048
2020-07-29 08:29:38 +02:00
Marek Siarkowicz
Karl Isenberg
yongruilin
Lukasz Szaszkiewicz
Andrew Sy Kim
Kevin Delgado
kkkkun
Damien Grisonnet
Paweł Banaszewski
Wojciech Tyczyński
Han Kang
Alexander Zielenski
Davanum Srinivas
Artur Żyliński
Abu Kashem
Reza Karimi
jyz0309
Yan Huang
Harry Zhang
tiloso
Daniel Smith
Jordan Liggitt
卢振兴10069964
Kagaya
Jiaxin Shan
yoyinzyc
Marcus Alder