Commit Graph

684 Commits

Author SHA1 Message Date
Siyuan Zhang a0dede6875 k8s.io/apiserver/storage: add some ResourceVersion validation in GetList unit tests
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: baac8bb573c8efb314b037f4fbac116556c03c83
2023-10-11 10:46:30 -07:00
Siyuan Zhang e15d4d2e0b k8s.io/apiserver/storage/etcd: refactor getCurrentState.
Extract getCurrentState as a separate method that can be reused.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: ebca5d438d9cb2c82d0b99dbcb0aeca8879db441
2023-10-11 09:56:07 -07:00
Lukasz Szaszkiewicz db34e9b7c7 storage/util: introduce HasInitialEventsEndBookmarkAnnotation
Kubernetes-commit: 70ae9c441cee9d78a88c0526592796ac4f491096
2023-09-18 11:39:08 +02:00
Lukasz Szaszkiewicz 90ba08909c storage/etcd: the watcher supports the API streaming
Kubernetes-commit: ca562fd280a9dd5db952eb8c14b93fcf6668cc49
2023-08-22 14:02:39 +02:00
Lukasz Szaszkiewicz e7424d8207 storage/testing/watcher_tests: add new flavour to RunWatchSemantics test and fill the gap
the new flavour proves that a storage layer follows RV
which is equal to the global rv

Kubernetes-commit: 0e238c4c28f11d1ec7cf5f506c81195c1cc463fe
2023-09-22 09:32:23 +02:00
Lukasz Szaszkiewicz f0dfb17949 storage/testing/watcher_tests: refactor RunWatchSemantics tests
Kubernetes-commit: dc5cfe3cff289ecb97153716bfd5817579d00fb3
2023-09-21 12:16:16 +02:00
Lukasz Szaszkiewicz c8dcfeed34 storage/testing/watcher_tests: make TestCacherWatchSemantics storage agnostic
Kubernetes-commit: 91bb75883c613d45563f3b7c01a69dde8194bfdc
2023-09-15 12:28:36 +02:00
Lukasz Szaszkiewicz c13e210d56 storage/testing/watcher_tests: move TestCacherWatchSemantics (no-op)
Kubernetes-commit: afbb1a6ef98b548b9e57b168614ca6e15fd0034c
2023-09-14 11:19:36 +02:00
Lukasz Szaszkiewicz 3c3bb86696 storage/testing/utils: add helper functions
Kubernetes-commit: 3a96baf3fa8e837159b27b98b05c5024143b06a3
2023-09-15 12:26:11 +02:00
Lukasz Szaszkiewicz fe7ae035ef storage/etcd3/watcher_test: refactor TestWatchErrorWhenNoNewFunc to a table test
Kubernetes-commit: 4b60c18183f11207d0cb2d89bf28090156222c61
2023-09-19 10:35:09 +02:00
Lukasz Szaszkiewicz f5adfca95e storage/testing: a simple refactor
Kubernetes-commit: 39af594f312ad46715dfb705d71ad4e607e27e5d
2023-09-14 11:16:10 +02:00
machine424 98d028d0d0 clarify the description of the metric apiserver_storage_objects
Kubernetes-commit: 084deb7a7cf147bbafd131ab1a36852a79f08a64
2023-09-12 11:16:36 +02:00
Rita Zhang 2bed5d11d9 kmsv2: add apiserver identity to metrics
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 43ccf6c4e8f173d981edebb6146c58b523fc21b7
2023-09-05 13:03:18 -07:00
Monis Khan 87aad93082 kmsv2: add legacy data integration test
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 95121fe846f875c4f3feb2b1bf42a9553566f097
2023-09-01 11:21:38 -04:00
Monis Khan 49c6151dee kmsv2: fix race in simpleCache.set when setting cache size metric
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: b10697c7880848d7ec110fd6b3e67015bbe74fa8
2023-08-27 15:14:04 -04:00
Lukasz Szaszkiewicz 957c305d94 storage/etcd: add TestWatchDispatchBookmarkEvents unit test
Kubernetes-commit: 875b00137fdfbc74756a0fc5b1c9b2adbeb78e55
2023-09-01 13:45:44 +02:00
Lukasz Szaszkiewicz 51e79bdd97 storage: document ProgressNotify from storage.ListOptions
At first glance, it seems that the fields storage.ListOptions.ProgressNotify and storage.ListOptions.Predicate.AllowWatchBookmarks
are the same. Unfortunately, this is not the case.

This PR documents the differences and motivations for why these fields are actually distinct.

Kubernetes-commit: 6058540f3d0edc405a1f1b8a96bd82ceca99c240
2023-09-01 10:48:36 +02:00
Marek Siarkowicz 8b7824a7e8 Avoid creating local variables that don't change
Having local variables gives false impression that this is overwritten
in the function block.

Kubernetes-commit: e01bd641447a315e28fab8148e99ac6afba9bcd7
2023-08-30 16:51:40 +02:00
Marek Siarkowicz 90adbb75eb Flatten switch case
Kubernetes-commit: 10553a1966892e305252c65ebeb9043416304f48
2023-08-30 16:42:34 +02:00
Marek Siarkowicz 5f9b5dda94 Refactor common WithRange case
From API call WithRange and WithPrefix work the same, they just set the range end.
The difference is when the range end is provided:
* WithRange(end) requires providing the end while calling
* WithPrefix() calculates the end based on key provided to the Get.

For example, those are equal:
* client.Get(ctx, "/pods/", WithPrefix())
* client.Get(ctx, "/pods/", WithRange(GetPrfixRangeEnd("/pods/")))

As keyPrefix is equal preparedKey there should not be a difference.

Kubernetes-commit: 1f4f2a5d6014dc8f98b25a9484d4a6064a6ae18e
2023-08-30 16:26:20 +02:00
Marek Siarkowicz a8fba27b08 Remove duplicated returnRV variable
returnRV was was equal to withRev, but updated at different time.

When preparing the request they are set equal to each other.

The only difference was during the for loop.
returnRV was always set no matter if pagination was enabled, while withRev only when paginating.

Kubernetes-commit: be4692864bb983e94e8d7b6b6aa1a9c22fe23bce
2023-08-30 17:36:05 +02:00
wackxu f40bea647b paginate initial list inside the storage watcher
Signed-off-by: wackxu <xushiwei5@huawei.com>

Kubernetes-commit: f5d6c65186d63647a46400762f849d500e6ac591
2023-07-28 16:43:34 +08:00
Lukasz Szaszkiewicz 206231e0d0 storage/etcd3: error when progressNotify option set and newFunc was provided for a registry
Kubernetes-commit: 1988c31fc8115bf9eec1adf2bffed3fd677d1a9f
2023-08-28 17:50:42 +02:00
Patrick Ohly 037fa7c2c2 apiserver: fix data race in etcd metrics
7a63997c8a1a9ba1 added a global variable which gets set multiple times by
different goroutines in integration tests, leading to a data race:

WARNING: DATA RACE
Write at 0x00000a626928 by goroutine 87080:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Previous write at 0x00000a626928 by goroutine 87079:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Mutex locking avoids the data race. Whether this variable really can be used
safely by those concurrent (?) tests is a different question...

Kubernetes-commit: 13a8ad12b8296c0360afe3f66218027dae6c1805
2023-08-25 10:42:17 +02:00
Lukasz Szaszkiewicz d8d65bebc5 storage/util: introduce AnnotateInitialEventsEndBookmark function
// AnnotateInitialEventsEndBookmark adds a special annotation to the given object
// which indicates that the initial events have been sent.
//
// Note that this function assumes that the obj's annotation
// field is a reference type (i.e. a map).

Kubernetes-commit: 47d9a47a08856613e2e6ae6aa8a1bdeb1e281f97
2023-08-24 12:19:10 +02:00
Lukasz Szaszkiewicz f75c503352 storage/factory: extend the Create method by newList and resourcePrefix params
Kubernetes-commit: ccabc01093a1344ebb27c32c946e9da3b8e91fd2
2023-07-28 09:53:01 +02:00
Lukasz Szaszkiewicz 52849e2b21 storage/cacher: duplicate logging when a watch is closed due to unresponsiveness.
Kubernetes-commit: 1327a57aaf8f5ffaea3d33204dbe64dccef4e0ef
2023-08-23 12:51:50 +02:00
Wojciech Tyczyński f56b9ee7f5 Graduate RemainingItemCount to GA
Kubernetes-commit: 4e2e059c7b205d2e4b246a262128223258a49498
2023-07-21 15:22:51 +02:00
Wojciech Tyczyński 3030f660a8 Graduate APIListChunking to GA
Kubernetes-commit: 6acfa3cb4ac876e46ead5ba4772ba18e480435ce
2023-07-21 11:35:21 +02:00
Lukasz Szaszkiewicz 7ecfbf7914 storage/etcd: simplify passing the transformer
Kubernetes-commit: a5600b6925169a7d99376d52472e5f8e8635082b
2023-08-21 12:27:41 +02:00
Lukasz Szaszkiewicz dc7ff6e034 storage/etcd: remove newWatcher function
Kubernetes-commit: 6161752ecbe434b8fb157dd3a30bb5c0cef7a23e
2023-08-21 12:13:12 +02:00
Lukasz Szaszkiewicz 6f2daefacf storage/util: move GetCurrentResourceVersionFromStorage
Kubernetes-commit: 9a253d896a096b4e1ffccf4b1f84e5cac1e1aad0
2023-07-26 15:53:13 +02:00
Lukasz Szaszkiewicz 2d9be35745 storage/etcd: no-op, refactor watcher.Watch method signature
Kubernetes-commit: f7e659db236286ca15707bf08acb08dc0ac4ab0e
2023-07-24 12:33:03 +02:00
Antonio Ojea 49f7deccb5 fix race on etcd client constructor for healthchecks
Change-Id: Id29b5b377989dcb5377316cfcdea367071a47365

Kubernetes-commit: 77b3bb0f69bfa4e5eb56ba484f724476304616cc
2023-08-08 13:55:14 +00:00
Rita Zhang 7ebae7f76f kmsv2: add metric for DEK cache filled
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 3cbecf218dfea2e99ae95310ac03406d1d87a072
2023-08-09 12:28:01 -07:00
Damien Grisonnet 245d131967 apiserver/etcd3: fix segv during metric collection
Fix a segfault when collecting the storage size metrics when the getters
used to collect the data on etcd haven't been initialized properly. This
happens when the EtcdOptions are not applied which is the case for
aggregated apiservers that don't care about storage.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

Kubernetes-commit: c6efaf16c1ed07ce37485b7a272628f653cbf06f
2023-08-10 17:01:17 +02:00
Monis Khan 8e93c650b5 kmsv2: KDF based nonce extension
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: bf49c727ba10881d5378e9242f31dc00dede51be
2023-03-25 14:41:04 -04:00
Marek Siarkowicz e9acd0c76d Fix the semantic meaning of etcd server within component statuses and metrics.
Instead of numerating all the etcd endpoints known by apiserver, we will
group them by purpose. `etcd-0` will be the default etcd, `etcd-1` will
be the first resource override, `etcd-2` will be the second override and
so on.

Kubernetes-commit: 03aad1f823cb719fa6e6b6d33fefa2a2140cc760
2023-07-19 14:25:54 +02:00
Nilekh Chaudhari 3373784322 feat: improves metric and logging
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: 2c8288ac873b89ae6e351df5e805b1a825aae60d
2023-07-07 23:01:05 +00:00
Nilekh Chaudhari 36a1803532 chore: hashes keyID
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: 131216fa8f2dd13f2585e2010717733f4cb2c1e2
2023-06-29 20:32:27 +00:00
Marek Siarkowicz 573a8d6d05 Improve apiserver storage size metric to allow it's graduation
Change name to make it compliant with prometheus guidelines.
Calculate it on demand instead of periodic to comply with prometheus standards.
Replace "endpoint" with "server" label to make it semantically consistent with storage factory

Kubernetes-commit: 7a63997c8a1a9ba14f2bdc478fdf33cf88f48d80
2023-06-22 11:56:09 +02:00
Marek Siarkowicz 9002dac854 Implement ConsistentListFromCache feature gate
Request bookmark every 100ms when there is at least one request blocked on revision not present in watch cache.

Kubernetes-commit: 39bb8f4bb1d013937aceac6c387563ffe13545c5
2023-06-06 15:49:46 +02:00
Marek Siarkowicz 23cd6bbea9 Fix TestConditionalProgressRequester and TestWaitUntilFreshAndListTimeout flakes
Kubernetes-commit: c1decb6763d2abf76d96aee8641ad56a23e0ba52
2023-07-12 13:36:51 +02:00
Marek Siarkowicz cd751eb82e Implement conditionalProgressRequester that allows requesting watch progress notification if watch cache is not fresh
Kubernetes-commit: 98461be8ffa7383152c442414a16adb217e98080
2023-07-10 18:10:49 +02:00
Marek Siarkowicz df9896fd00 Test consistent List
Kubernetes-commit: b36fdd68b72cd5c64ea5be3917846067644da983
2023-07-03 17:22:22 +02:00
Wojciech Tyczyński b560936651 Generalize watch storage tests
Kubernetes-commit: 8266c4d934d42a5175a84bff10fda4bf36f13817
2023-06-27 18:12:30 +02:00
Monis Khan b90de05302 no-op: split transformer interface
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 9354e78289fde933f485c663df960b0bc6e24df5
2023-03-24 16:23:40 -04:00
Marek Siarkowicz eda3579943 Test all cases for LIST request cache bypass
Kubernetes-commit: e4cf103c9b98391e5e135c3991cd8f380bfe22fe
2023-06-21 14:12:04 +02:00
Marek Siarkowicz 72bea89ab7 Refactor shouldDelegateList/shouldListFromStorage to better explain decisions
Kubernetes-commit: 406899360b55b3a0e28970f369288a2bb8a804fe
2023-06-21 14:02:46 +02:00
Marek Siarkowicz 96d1438818 Refactor WithRequireLeader to make it part of the etcd store
Kubernetes-commit: a9af2de8fdd1f2799b2017b19027958c69ffb04d
2023-06-21 14:21:24 +02:00