Commit Graph

108 Commits

Author SHA1 Message Date
Mike Spreitzer e28ab56bd4 Introduce more metrics on concurrency
Introduce min, average, and standard deviation for the number of
executing mutating and readOnly requests.

Introduce min, max, average, and standard deviation for the number
waiting and number waiting per priority level.

Later:

Revised to use a series of windows

Use three individuals instead of array of powers

Later:

Add coarse queue count metrics, removed windowed avg and stddev

Add metrics for number of queued mutating and readOnly requests,
to complement metrics for number executing.

Later:

Removed windowed average and standard deviation because consumers can
derive such from integrals of consumer's chosen window.

Also replaced "requestKind" Prometheus label with "request_kind".

Later:

Revised to focus on sampling

Make the clock intrinsic to a TimedObserver

... so that the clock can be read while holding the observer's lock;
otherwise, forward progress is not guaranteed (and violations were
observed in testing).

Bug fixes and histogram buckets revision

SetX1 to 1 when queue length limit is zero, beause dividing by zero is nasty.

Remove obsolete argument in gen_test.go.

Add a bucket boundary at 0 for sample-and-water-mark histograms, to
distinguish zeroes from non-zeros.

This includes adding Integrator test.

Simplified test code.

More pervasively used "ctlr" instead of "ctl" as abbreviation for
"controller".

Kubernetes-commit: 57ecea22296797a93b0157169db0ff2e477f58d0
2020-05-17 01:02:25 -04:00
Jordan Liggitt 50fc715287 Add audit annotation for requests to deprecated API endpoints
Kubernetes-commit: 9550f5080fde480c0712a0870c56b4e2415f8637
2020-07-06 15:03:24 -04:00
Li Bo 7f8013437d log tls handshake error at trace level to avoid error flooding, and add
metric to count such errors

Kubernetes-commit: eabb362aa37f7acc85bc64f2035dadc5511d3716
2020-05-20 11:29:44 +08:00
Jordan Liggitt c02923d8c7 apiserver: increment metric for deprecated API use
Kubernetes-commit: e4bb1daecf36aac3051d36a20dfdf7ea3050de58
2020-04-30 12:21:29 -04:00
Han Kang 64913bcbc2 fix a number of unbounded dimensions in request metrics (#89451)
* fix a number of unbounded dimensions in request metrics

* add test suite for cleanVerb and cleanContentType

* Properly validate that the content-type and charset (if applicable) are RFC compliant

* add additional test case

* truncate list of content-types

Change-Id: Ia5fe0d2e2c602e4def4b8e0849cc19f3f9251818

Kubernetes-commit: 6c588c3f441252f42fd37526297ed92d1e1f3acf
2020-05-29 08:29:03 +00:00
Han Kang fe6033d4a2 remove client label from apiserver request count metric since it is unbounded
Change-Id: I3a9eacebc9d9dc9ed6347260d9378cdcb5743431

Kubernetes-commit: cc3190968b1f14ddf4067abef849fc41bd6068dc
2020-01-29 12:25:55 -08:00
danielqsj 99a93eeef1 clean deprecated apiserver request metrics
Kubernetes-commit: 0e8b956e27b07092ca4873c09314e4cff930ab27
2020-01-10 17:09:26 +08:00
danielqsj ade6c3cc65 keep apiserver_request_latencies_summary
Kubernetes-commit: 2c2e1c5443a637353a3ab36cb8e51a9a60691042
2019-05-28 17:31:08 +08:00
danielqsj 669e87efbe clean SinceInMicroseconds, convert to SinceInSeconds
Kubernetes-commit: ab182552b485ceed96c3b05d284ae9480377daf8
2019-04-15 16:18:01 +08:00
danielqsj b31ce5f20e remove deprecated metrics of apiserver
Kubernetes-commit: 060756babbb6991cf9426c38be595d23e7124b4e
2019-04-12 16:15:13 +08:00
RainbowMango de8c866328 Hide apiserver metrics that have been deprecated in 1.14
Update E2E test accordingly.

Kubernetes-commit: 5e0695e339ff0887ebebddfa8630de191f7fa235
2019-10-12 20:25:10 +08:00
Han Kang 00d376192a rename metric for apiserver request terminations and reword corresponding documentation
Change-Id: I47a9c7b10614afe85bb652fa61984f91848d6d65

Kubernetes-commit: f8f1def5f1d92a588f48ebc01293e142f8dd63fd
2019-10-21 13:26:31 -07:00
Han Kang a5356f425d pluralize error metric name
Change-Id: I12eb94f41ded20ed5a16332ada13a7b34f75de18

Kubernetes-commit: 5e652fe12605027509870d5d7f821390f20b1cee
2019-10-08 15:49:12 -07:00
Han Kang 4a680138c0 Fix double counting issue for request metrics on timeout.
Currently we record request metrics during the normal request flow and
we also manually invoke `Record` in the timeout handler to record
timeouts. This means that we effectively double count whenever we
timeout. This PR renames the `Record` function to `RecordRequestError`
to more accurately reflect the intended side-effect of the function
call.

Change-Id: Ie37fd0c1e501bd525640a434433d364a5fd6dde2

Kubernetes-commit: 4c6e7247878477a1f2efc26df7f141258010374f
2019-10-02 15:04:29 -07:00
RainbowMango 7687c3a4e8 Migrate prometheus bucket functionality to metrics stability framework.
Kubernetes-commit: 1f9ab1b85c5560a155b1702d1b5d28aa2d95a1b7
2019-09-12 17:10:44 +08:00
Han Kang 3e6e1db500 add some documentation around the metrics stability migration changes for clarity
Kubernetes-commit: 4e5d906c4d008f914b0ede26ea91533d6343dec5
2019-08-26 19:15:30 -07:00
Han Kang b9084e350a migrate kube-apiserver metrics to stability framework
Kubernetes-commit: 466980dd747e06e55451301c624eecccfa505123
2019-08-22 15:38:42 -07:00
wojtekt 983198c58c Fix verb reporting in metrics
Kubernetes-commit: 715f538c7012e190c6bcf276fa329da261ec5d00
2019-08-20 15:40:34 +02:00
Maciej Borsz 9baeaec78d Add 'apiserver_watch_events_sizes'.
That metric can be used to estimate the size of watch events sent out to
the clients.

Kubernetes-commit: 4af189aab9cca61d1a701133596f6f82b1ca0719
2019-07-23 12:55:11 +02:00
Maciej Borsz f6a1588a67 Register WatchEvents metric
Kubernetes-commit: bc100cfd0ccd7849b66e29eba62cd2119a83a68a
2019-07-12 14:08:40 +02:00
Maciej Borsz df684d906a Add 'apiserver_watch_events_total' metric.
We found this metric useful to understand the CPU usage of the
kube-apiserver process.

Kubernetes-commit: 58d3338ddd5e406988fdfaa9019fd7200996c49f
2019-06-05 14:27:09 +02:00
Clayton Coleman 70165311bb Avoid 3 object creations when no dryRun parameter is provided
These allocations should only occur rarely.

```
BenchmarkGet-12          	  100000	    108013 ns/op	   17732 B/op	     149 allocs/op

BenchmarkGet-12          	  100000	    109045 ns/op	   17608 B/op	     146 allocs/op
```

Kubernetes-commit: 59b4f47b225029068bf015117ada1a06d54fbd59
2019-03-21 20:57:36 -04:00
Clayton Coleman f24566b3c2 Avoid an allocation on all requests when checking for an old user agent
ReplaceAllStrings always allocates, while scanning a relatively short
regex twice is slightly more CPU immediately but less later.

```
BenchmarkGet-12          	  100000	    108824 ns/op	   17818 B/op	     152 allocs/op

BenchmarkGet-12          	  100000	    108013 ns/op	   17732 B/op	     149 allocs/op
```

Kubernetes-commit: 83c41eab1d6f55c9ccf61744bfc345c95d3a0664
2019-03-21 20:48:00 -04:00
jennybuckley 638c313e5b Feature-gate the APPLY metric value
Kubernetes-commit: 6e512eb8758fb32878d7baa33daf3c49561086c0
2019-03-07 15:14:43 -08:00
jennybuckley 4a5bf1da2a use utilsets.NewString
Kubernetes-commit: 077dd28df4025c69b89e75b0f32cb0954401ec80
2019-03-07 14:20:10 -08:00
jennybuckley ac3850113b handle validation errors correctly
Kubernetes-commit: 51b75460aa356a6b07cd5d727cc3ad6c00a13f20
2019-03-06 15:57:18 -08:00
jennybuckley 7624354bd9 Track dry-run and apply in metrics
Kubernetes-commit: 4ac0116a5d826d97cc839dca55e3351ef68f2db1
2019-03-06 14:30:04 -08:00
danielqsj 822a44db47 convert latency in mertics name to duration
Kubernetes-commit: f7b437cae02c93a0c83a3ad6c044f43751c20346
2019-02-22 21:40:13 +08:00
wojtekt 0760bae7df Adjust buckets in apiserver request latency metrics
Kubernetes-commit: d0508c7e872f60826d68c58c458cfd865554b486
2019-02-01 15:59:09 +01:00
danielqsj da40ddb2dd Move etcd latency metrics to histogram and update test case
Kubernetes-commit: 8b418631c08cf66ee57c9ec31fe95372a0a3e075
2019-01-02 13:50:02 +08:00
danielqsj e0406b8675 Change endpoints metrics to conform guideline
Kubernetes-commit: 88c4b644005c5f674e9d03abae7e1cccda4fd23e
2018-12-26 22:29:13 +08:00
Md. Emruz Hossain 5c0ad5c6cf Fix a typo
Kubernetes-commit: 83ff55ccc39ecc1adadd96486d5c1a5c02466e33
2018-12-31 12:28:33 +06:00
Dr. Stefan Schimanski b4ca200ede apiserver: add component label to request metrics
Kubernetes-commit: 7190b17e5a52a7c0673c2e8a30cbe95f2847a6df
2018-11-29 16:30:15 +01:00
Dr. Stefan Schimanski 0f8de8e61f apiserver: add group+version to request metrics
Kubernetes-commit: ff6e028755163ef52ad135da9f5d800c5749632c
2018-11-29 16:03:37 +01:00
wojtekt 7ff366ebcb Fix apiserver metrics
Kubernetes-commit: 3d9552cc388d66851a687cf50a93a6e2c9166c35
2018-06-29 14:15:53 +02:00
wojtekt 27dca16e75 Track number of registered watchers in apiserver
Kubernetes-commit: bdd9492da0c992e532c5e5d07cae3b68c9b7b78f
2018-05-14 11:22:10 +02:00
Wang Guoliang 156ee1e04e should use time.Since instead of time.Now().Sub
Kubernetes-commit: 89669283fe84619f0c15af8495ccb28a752bb1e8
2018-02-11 21:17:00 +08:00
Martin Vladev 3c79460222 Register Prometheus etcdmetrics only for apiserver
Removed automatic registration with `init` funciton and use `Register` function
to register metrics for etcd storage only when requested.

Kubernetes-commit: 40cf7880135b56e2d88a04d5fce08303b249eb34
2018-04-20 17:19:13 +03:00
Di Xu 9beeb59216 fix all the typos across the project
Kubernetes-commit: 48388fec7eaad4ac8d84fbe20673ffacf41964a1
2018-02-09 14:53:53 +08:00
Nikhita Raghunath 2cc4871e76 Collect prometheus metrics for custom resources
Since we have a custom handler for apiextensions-apiserver,
we need to record the metrics here.

Kubernetes-commit: 74cd45fb21b349dd037e3bfd844459ca5834cca1
2017-12-28 14:36:31 +05:30
halfcrazy 6f8c3a80da fix typo in package apiserver
Kubernetes-commit: 0da91a8577ddfdeaff985cbb6c0da69d5a2ffc81
2018-02-01 03:04:33 +08:00
Marek Grabowski f2c38580dc Add a metric to track usage of inflight request limit.
Kubernetes-commit: 000d7bac29b9239a29531a526d382394d8d60353
2018-01-16 15:48:20 +00:00
Marek Grabowski d8aa7399d2 Add apiserver metric for number of requests dropped by 'inflight-request' filters.
Kubernetes-commit: 32c66c2b56afe0c716169d9705f84172155ddbb4
2018-01-16 15:48:20 +00:00
hzxuzhonghu 15434cbd47 move prometheus init to k8s.io/apiserver/pkg/endpoints/metrics/metrics.go
Kubernetes-commit: 631119a7d65e01e48b5d8a46d7300b20c65262e1
2018-01-16 15:28:16 +08:00
Clayton Coleman 4175a07239 Track gauge of all long running API requests
Allows a caller to know how many exec, log, proxy, and watch calls are
running at the current moment.

Kubernetes-commit: fabce1b893f96bdf466c1fdb1fcf825210c008ae
2017-09-19 20:35:29 -04:00
Clayton Coleman 1caba4b96e Collapse all metrics handlers into common code
Remove the MonitorRequest method and replace with a method that takes
request.RequestInfo, which is our default way to talk about API objects.
Preserves existing semantics for calls.

Kubernetes-commit: 10e6dc5ed3573118c56fa8823b387e47c8e8ae06
2017-09-11 14:53:18 -04:00
Kubernetes Publisher 8fc21589fe Increase sliding window to 5hr for request_latencies metric
Kubernetes-commit: e1ba3da16c9bebccb3902952d56b59f2e2dbb900
2017-09-22 11:42:05 +00:00
Kubernetes Publisher 7727b5a3a8 Report scope on all apiserver metrics
Counting list of namespaces is != list across all namespaces (same for
latency)

Kubernetes-commit: 545aba778d5d039a3b8a0f0939fdf8f8261ae1a8
2017-09-22 11:42:04 +00:00
Kubernetes Publisher 4c578f11a1 Normalize WATCHLIST to WATCH in metrics
This causes confusion and doesn't match what we authorize on

Kubernetes-commit: 5e46d5b5450f7e7db87e860440f1bb1f8f523ffa
2017-09-22 11:42:04 +00:00
Shyam Jeedigunta d156370a82 Add apiserver metric for response sizes split by namespace scope
Kubernetes-commit: 5facb62806a7f5d442bff8f77418b53cd58544f9
2017-07-29 13:55:24 +00:00
Clayton Coleman c521c8f6b5 Record 429 and timeout errors to prometheus
Allows gathering of load being shed.

Kubernetes-commit: 2e33a2f0bc8ac82aecadcb19cf6e41259454d182
2017-07-16 04:08:41 +00:00
Clayton Coleman 2f49bbbf91 Subresources are not included in apiserver prometheus metrics
Subresources are very often completely different code paths and errors
generated on those code paths are important to distinguish.

Kubernetes-commit: ad431c454c1306fdcc2134a3626444984d350f46
2017-06-13 20:47:28 +00:00
Shyam Jeedigunta b10ed3c56c Copy static variable 'verb' before instrumenting APIserver call to prevent overwriting
Kubernetes-commit: 4d457f55f4b190012440c8a454a2aee338a42c4d
2017-05-19 17:28:21 +00:00
Matt Liggett 3b01985067 Use regexp instead of substring to do search and replace.
enisoc pointed out how ToLower can change (lengthen even!) the length of
a string given arbitrary input.

Kubernetes-commit: fe202fcfc47c7e0fa7d75dc933c9988bb5a53e5e
2017-04-17 20:35:48 +00:00
Matt Liggett 6c87ad1f20 Drop leading path of KUBECTL.EXE if it shows up in User-Agent.
Kubernetes-commit: 4cff7c3d30f4f6b4377d52ec29ec4866f1d28bb9
2017-04-14 20:35:35 +00:00
Dominik Schulz 37527becd4 Clean user agent
This commit cleans common browser user-agents to reduce the metrics
cardinality in exported prometheus metrics.

Kubernetes-commit: b5c89a8be68e031e12d6750ed6cc470de4cc3fdf
2017-03-18 19:56:09 +00:00
deads2k 1b0d4c5781 tweak a list to a watch verb when requested 2017-02-23 09:48:09 -05:00
Dr. Stefan Schimanski dc90e9a003 genericapiserver: move packages
Towards a sane k8s.io/apiserver package structure.
2017-01-19 10:27:27 -05:00