Commit Graph

274 Commits

Author SHA1 Message Date
Davanum Srinivas a22f7973bf Rename `cluster` to `storage_cluster_id` for apiserver_storage_size_bytes metric
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: fb5140f562289f3ffe596b4a3af4650b850a0941
2024-04-11 15:06:03 -04:00
Marek Siarkowicz 67b6245fc3 Fix enabling consistent list from watch cache also works for resourceVersion=0
Kubernetes-commit: 0b8e79580eb3a63ca7707626b4894adfb9125586
2024-03-04 19:35:34 +01:00
Marek Siarkowicz 483da2032d Fix non-recursive list when consistent list from cache is enabled
Kubernetes-commit: a527cab9fce0b0234db8b4e1e95a0fc20e135df1
2024-03-04 17:59:04 +01:00
Han Kang f615696539 bump the stability level of apiserver_storage_size_bytes to STABLE
Kubernetes-commit: f38852768e312fe7b9775b92f7228371a0a96f90
2024-02-16 09:13:46 -08:00
HirazawaUi bc8676d59a Add decoding time to the audit log
Kubernetes-commit: 20fe2a3539e90f7554f94359ac3b4058a5bbb363
2023-10-25 22:52:11 +08:00
James Blair b6487a8ac1 Fix etcd repository path to prevent redirects.
Signed-off-by: James Blair <mail@jamesblair.net>

Kubernetes-commit: b6c1f8ef08c3451f17048447e107c509a8ed950e
2023-11-02 09:31:37 +13:00
Robert Svarinskis fdd26489db Fix etcd storage_events_received_total metric not being registered
Kubernetes-commit: ceb7ca93875cd74bea60500812cf7f024d3b1f2d
2023-12-04 09:12:21 +00:00
HirazawaUi 88cd5253fa make the decode function respect the timeout context
Kubernetes-commit: 3fb16abfca43f53048a28758f2366cb52e31d2b7
2023-11-08 23:39:10 +08:00
Siyuan Zhang cef6db2cfb k8s.io/apiserver/storage/etcd: refactor etcd GetList.
Extract the logic to determine withRev to a separate method for better readability.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 624169c5b50ee8a6e9a761e9488134985334817e
2023-10-20 12:59:46 -07:00
Wojciech Tyczynski 6caf326620 Revert "Make the decode function respect the timeout context"
Kubernetes-commit: 98a2f22e740ccd2c30711f1b21d6383f1b91595e
2023-10-31 16:27:17 +01:00
Siyuan Zhang ef409f941b k8s.io/apiserver/storage/etcd: refactor etcd GetList.
Reorder some code.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: a968f51fa2f87ed57f9e48ba436e11421c403b27
2023-10-20 12:39:51 -07:00
Siyuan Zhang 49886c205c k8s.io/apiserver/storage/etcd: refactor etcd GetList.
reduce redundant update of withRev after request.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 84ec5e2eccbc07b17f3b3e3e00dc3996105e0346
2023-10-20 12:06:46 -07:00
HirazawaUi be73f76247 Make the decode function respect the timeout context
Kubernetes-commit: f78b367db6393a449b8f456e725cbe155d9b90e6
2023-10-25 23:12:16 +08:00
Wojciech Tyczyński 324d2bc3cf Cleanup paging parameters in etcd3 store
Kubernetes-commit: 3f4d3b67682335db510f85deb65b322127a3a0a1
2023-10-20 15:38:59 +02:00
Wojciech Tyczyński 5bf4f58ab8 Remove storageConfig.Paging parameter
Kubernetes-commit: b386120da239bf9652fc02b2d2cbbd0fcc3cd121
2023-10-20 15:35:58 +02:00
machine424 340fd67b3a storage/etcd3/metrics: add unit test for apiserver_storage_objects
Kubernetes-commit: a96b97e40c0837f727c8e6d42f021b66922bb26c
2023-09-22 16:12:40 +02:00
Siyuan Zhang 77032c52b8 k8s.io/apiserver/storage: add 3 new unit tests for delete.
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 26a4e06c92c248748dd3c50c74d75f8adc3c6823
2023-10-11 10:38:01 -07:00
Siyuan Zhang 70af178d56 k8s.io/apiserver/storage: add a new TestCreate case.
Add a test case of create with rv set.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 2f923e356e042d9dce88e8f78abf30f414051e71
2023-10-11 10:24:31 -07:00
Siyuan Zhang e15d4d2e0b k8s.io/apiserver/storage/etcd: refactor getCurrentState.
Extract getCurrentState as a separate method that can be reused.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: ebca5d438d9cb2c82d0b99dbcb0aeca8879db441
2023-10-11 09:56:07 -07:00
Lukasz Szaszkiewicz 90ba08909c storage/etcd: the watcher supports the API streaming
Kubernetes-commit: ca562fd280a9dd5db952eb8c14b93fcf6668cc49
2023-08-22 14:02:39 +02:00
Lukasz Szaszkiewicz fe7ae035ef storage/etcd3/watcher_test: refactor TestWatchErrorWhenNoNewFunc to a table test
Kubernetes-commit: 4b60c18183f11207d0cb2d89bf28090156222c61
2023-09-19 10:35:09 +02:00
machine424 98d028d0d0 clarify the description of the metric apiserver_storage_objects
Kubernetes-commit: 084deb7a7cf147bbafd131ab1a36852a79f08a64
2023-09-12 11:16:36 +02:00
Lukasz Szaszkiewicz 957c305d94 storage/etcd: add TestWatchDispatchBookmarkEvents unit test
Kubernetes-commit: 875b00137fdfbc74756a0fc5b1c9b2adbeb78e55
2023-09-01 13:45:44 +02:00
Marek Siarkowicz 8b7824a7e8 Avoid creating local variables that don't change
Having local variables gives false impression that this is overwritten
in the function block.

Kubernetes-commit: e01bd641447a315e28fab8148e99ac6afba9bcd7
2023-08-30 16:51:40 +02:00
Marek Siarkowicz 90adbb75eb Flatten switch case
Kubernetes-commit: 10553a1966892e305252c65ebeb9043416304f48
2023-08-30 16:42:34 +02:00
Marek Siarkowicz 5f9b5dda94 Refactor common WithRange case
From API call WithRange and WithPrefix work the same, they just set the range end.
The difference is when the range end is provided:
* WithRange(end) requires providing the end while calling
* WithPrefix() calculates the end based on key provided to the Get.

For example, those are equal:
* client.Get(ctx, "/pods/", WithPrefix())
* client.Get(ctx, "/pods/", WithRange(GetPrfixRangeEnd("/pods/")))

As keyPrefix is equal preparedKey there should not be a difference.

Kubernetes-commit: 1f4f2a5d6014dc8f98b25a9484d4a6064a6ae18e
2023-08-30 16:26:20 +02:00
Marek Siarkowicz a8fba27b08 Remove duplicated returnRV variable
returnRV was was equal to withRev, but updated at different time.

When preparing the request they are set equal to each other.

The only difference was during the for loop.
returnRV was always set no matter if pagination was enabled, while withRev only when paginating.

Kubernetes-commit: be4692864bb983e94e8d7b6b6aa1a9c22fe23bce
2023-08-30 17:36:05 +02:00
wackxu f40bea647b paginate initial list inside the storage watcher
Signed-off-by: wackxu <xushiwei5@huawei.com>

Kubernetes-commit: f5d6c65186d63647a46400762f849d500e6ac591
2023-07-28 16:43:34 +08:00
Lukasz Szaszkiewicz 206231e0d0 storage/etcd3: error when progressNotify option set and newFunc was provided for a registry
Kubernetes-commit: 1988c31fc8115bf9eec1adf2bffed3fd677d1a9f
2023-08-28 17:50:42 +02:00
Patrick Ohly 037fa7c2c2 apiserver: fix data race in etcd metrics
7a63997c8a1a9ba1 added a global variable which gets set multiple times by
different goroutines in integration tests, leading to a data race:

WARNING: DATA RACE
Write at 0x00000a626928 by goroutine 87080:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Previous write at 0x00000a626928 by goroutine 87079:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Mutex locking avoids the data race. Whether this variable really can be used
safely by those concurrent (?) tests is a different question...

Kubernetes-commit: 13a8ad12b8296c0360afe3f66218027dae6c1805
2023-08-25 10:42:17 +02:00
Lukasz Szaszkiewicz f75c503352 storage/factory: extend the Create method by newList and resourcePrefix params
Kubernetes-commit: ccabc01093a1344ebb27c32c946e9da3b8e91fd2
2023-07-28 09:53:01 +02:00
Wojciech Tyczyński f56b9ee7f5 Graduate RemainingItemCount to GA
Kubernetes-commit: 4e2e059c7b205d2e4b246a262128223258a49498
2023-07-21 15:22:51 +02:00
Lukasz Szaszkiewicz 7ecfbf7914 storage/etcd: simplify passing the transformer
Kubernetes-commit: a5600b6925169a7d99376d52472e5f8e8635082b
2023-08-21 12:27:41 +02:00
Lukasz Szaszkiewicz dc7ff6e034 storage/etcd: remove newWatcher function
Kubernetes-commit: 6161752ecbe434b8fb157dd3a30bb5c0cef7a23e
2023-08-21 12:13:12 +02:00
Lukasz Szaszkiewicz 2d9be35745 storage/etcd: no-op, refactor watcher.Watch method signature
Kubernetes-commit: f7e659db236286ca15707bf08acb08dc0ac4ab0e
2023-07-24 12:33:03 +02:00
Damien Grisonnet 245d131967 apiserver/etcd3: fix segv during metric collection
Fix a segfault when collecting the storage size metrics when the getters
used to collect the data on etcd haven't been initialized properly. This
happens when the EtcdOptions are not applied which is the case for
aggregated apiservers that don't care about storage.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

Kubernetes-commit: c6efaf16c1ed07ce37485b7a272628f653cbf06f
2023-08-10 17:01:17 +02:00
Marek Siarkowicz e9acd0c76d Fix the semantic meaning of etcd server within component statuses and metrics.
Instead of numerating all the etcd endpoints known by apiserver, we will
group them by purpose. `etcd-0` will be the default etcd, `etcd-1` will
be the first resource override, `etcd-2` will be the second override and
so on.

Kubernetes-commit: 03aad1f823cb719fa6e6b6d33fefa2a2140cc760
2023-07-19 14:25:54 +02:00
Marek Siarkowicz 573a8d6d05 Improve apiserver storage size metric to allow it's graduation
Change name to make it compliant with prometheus guidelines.
Calculate it on demand instead of periodic to comply with prometheus standards.
Replace "endpoint" with "server" label to make it semantically consistent with storage factory

Kubernetes-commit: 7a63997c8a1a9ba14f2bdc478fdf33cf88f48d80
2023-06-22 11:56:09 +02:00
Marek Siarkowicz cd751eb82e Implement conditionalProgressRequester that allows requesting watch progress notification if watch cache is not fresh
Kubernetes-commit: 98461be8ffa7383152c442414a16adb217e98080
2023-07-10 18:10:49 +02:00
Marek Siarkowicz df9896fd00 Test consistent List
Kubernetes-commit: b36fdd68b72cd5c64ea5be3917846067644da983
2023-07-03 17:22:22 +02:00
Marek Siarkowicz 96d1438818 Refactor WithRequireLeader to make it part of the etcd store
Kubernetes-commit: a9af2de8fdd1f2799b2017b19027958c69ffb04d
2023-06-21 14:21:24 +02:00
Marek Siarkowicz fb965cd2c4 Make etcd component status consistent with health probes
Co-authored-by: Antonio Ojea <antonio.ojea.garcia@gmail.com>

Kubernetes-commit: a60314c47ef28dd04b086a2ad4b4bfd40b6f86a4
2023-06-05 16:26:12 +02:00
Lukasz Szaszkiewicz decaec94cd storage/etcd: skip SendInitialEvents if the request is backward compatible
otherwise an error will be returned.
backward compatibility is defined as RV = "" || RV = "O" and AllowWatchBookmark is set to false.
in that case we rely on 267eb25e60/staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go (L260)

Kubernetes-commit: f2de1a00b8fd7a22f9d0b2c1ace69be41d304f83
2023-05-08 13:04:31 +02:00
kkkkun f4c24327c3 Fix etcd3 config changed
Kubernetes-commit: 57969c1b52ce0c3c27ccbf02ecc52ebc1b8533d9
2023-05-06 19:40:40 +08:00
Wojciech Tyczyński 68a91d7597 Enable a bunch of generic storage tests for watchcache.
Kubernetes-commit: 88e2c34439016e5feddfed529698ed71e81904d9
2023-04-17 19:03:51 +02:00
Wojciech Tyczyński ca97e27fb8 Refactor watch bookmark tests to allow sharing between etcd3 and watchcache
Kubernetes-commit: 0297329795efc0f2bd24751e18f66b0aa79c222d
2023-04-17 14:22:52 +02:00
Wojciech Tyczyński 072d278e39 Generalize few watch tests from cacher
Kubernetes-commit: 45e836a968acc113cb03768cd8c730bea89bd332
2023-04-17 21:48:08 +02:00
iyear b1805a9a4d apiserver/etcd: add request and request_error metrics (#117222)
* apiserver/etcd: add request and request_error metrics

Signed-off-by: iyear <ljyngup@gmail.com>

* apiserver/etcd: rename etcdRequestCounts metric

Signed-off-by: iyear <ljyngup@gmail.com>

---------

Signed-off-by: iyear <ljyngup@gmail.com>

Kubernetes-commit: 4f6714ef42dcdb1b35bb2d8581c1ef83d59f02d1
2023-04-12 22:02:45 +08:00
scott d681755704 ftr(etcd): add benchmarks
Kubernetes-commit: db92cc1388021495e47dbe257a9e6559031cfe51
2023-03-29 21:52:38 +08:00
Jordan Liggitt 8ea1930d95 Recognize etcd/grpc cancel errors correctly
Kubernetes-commit: 267eb25e60955fe8e438c6311412e7cf7d028acb
2023-03-08 15:51:25 -05:00