Commit Graph

709 Commits

Author SHA1 Message Date
xyz-li 36207c1fc4 apiserver: fix watch namespace
For request like '/api/v1/watch/namespaces/*', don't set scope.namespace.
Because the func `addWatcher` add a watcher to allWatchers with the value `scope.namespace` not empty.
But the function `dispatchEvent` dispatch event with an empty namespace.

Signed-off-by: xyz-li <hui0787411@163.com>

Kubernetes-commit: 70fdb7b1c533454a2cd5dfa666a0251dbdf54400
2024-05-27 17:48:49 +08:00
Wojciech Tyczyński d06319266f Don't sort under lock
Kubernetes-commit: f6072e0d73dcb33edd3f2b1e656bb6d0cf81f9f0
2023-11-23 18:13:43 +01:00
Davanum Srinivas 818604e006 Rename `cluster` to `storage_cluster_id` for apiserver_storage_size_bytes metric
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: d995bfa258c4c9ee6e0c866451b8305a8077bca3
2024-04-11 15:06:03 -04:00
Marek Siarkowicz d432100923 Undo double run of the TestWatchSemantics test to avoid hitting timeout
Kubernetes-commit: cf2a337154f14b64923d3dace9bc2cab188dbb01
2024-03-19 15:16:52 +01:00
Lukasz Szaszkiewicz 0946a0336e apiserver/storage/cacher: decrease the running time of tests in the cacher package.
It turns out that kube has a custom timeout for tests of 3 minutes.
The tests in the cacher package are utilizing nearly the
entire time and are being terminated, resulting in failing jobs.

Before the change, the TestWatchSemantics took ~43s to run. With this simple change, it now takes ~18s.

When we created the tests, we didn't measure the running time and assumed that waiting 1 second on a watch channel
to make sure no more events are received was sufficient.
This PR decreases the waiting time to 300 milliseconds.
Modern computers can perform many tasks within that time.
In addition to that, the tests are serial in nature, meaning that there is no other
actor that could add items to the database, which could result in receiving new items.

After the change the total running time decreased by 17%.
Before the tests needed ~176s after they need ~146s.
The changes also improved TestWatchSemanticInitialEventsExtended.

Kubernetes-commit: d9ca300598e7195545ca38ab9e5e640a2379d553
2024-03-12 09:15:55 +01:00
Marek Siarkowicz 582da82650 Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage.
Kubernetes-commit: f8f08542c911b0fd620a26ca038b4f255b7a6217
2024-03-14 15:20:29 +01:00
Wojciech Tyczyński 21e0f5b77d Ensure that initial events are sorted for WatchList
Kubernetes-commit: ff2189b7c2b383a9f260444a8371bcf22d65baa6
2023-09-26 18:39:44 +02:00
Marek Siarkowicz b9037e3894 Test that separation of streams work by using progress notifies
Kubernetes-commit: b1e1d68cfb9acb1849423f7d0d67e87da3a359c2
2024-02-29 17:51:46 +01:00
Marek Siarkowicz 88805caf85 Prevent watch cache starvation, by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior
Kubernetes-commit: 4009acb0cf17a5500041bd40514a62256bef69d9
2024-02-27 11:25:42 +01:00
Lukasz Szaszkiewicz 7c5f6db7bf cacher: when forgeting a watcher, call stopWatcherLocked multiple times
It's possible that the watcher is already not in the structure (e.g. in case of
simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked()
on a watcher multiple times.

Kubernetes-commit: 7e35823690df01bd019a88d3346bd3ac820afaca
2023-10-30 14:24:39 +01:00
Antonio Ojea bdad50b280 Revert "cacher: when forgeting a watcher, call stopWatcherLocked multiple times"
This reverts commit bbca4a4b9add0f6c58e132500fd89dd39ee077f4.

Kubernetes-commit: c2cb3209138d852520da2743b9bd3a9795b2b7fb
2023-10-31 15:28:01 +00:00
Wojciech Tyczynski 6caf326620 Revert "Make the decode function respect the timeout context"
Kubernetes-commit: 98a2f22e740ccd2c30711f1b21d6383f1b91595e
2023-10-31 16:27:17 +01:00
Lukasz Szaszkiewicz 3f81d0cca7 cacher: when forgeting a watcher, call stopWatcherLocked multiple times
It's possible that the watcher is already not in the structure (e.g. in case of
simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked()
on a watcher multiple times.

Kubernetes-commit: bbca4a4b9add0f6c58e132500fd89dd39ee077f4
2023-10-30 14:24:39 +01:00
Siyuan Zhang ef409f941b k8s.io/apiserver/storage/etcd: refactor etcd GetList.
Reorder some code.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: a968f51fa2f87ed57f9e48ba436e11421c403b27
2023-10-20 12:39:51 -07:00
Siyuan Zhang 49886c205c k8s.io/apiserver/storage/etcd: refactor etcd GetList.
reduce redundant update of withRev after request.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 84ec5e2eccbc07b17f3b3e3e00dc3996105e0346
2023-10-20 12:06:46 -07:00
HirazawaUi be73f76247 Make the decode function respect the timeout context
Kubernetes-commit: f78b367db6393a449b8f456e725cbe155d9b90e6
2023-10-25 23:12:16 +08:00
Nilekh Chaudhari f0fe5d558c chore: updates api doc
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: d3de47ceeedd1fae3f3c95595186a028002526d6
2023-10-17 22:17:43 +00:00
Rita Zhang 26219aabef [KMSv2] promote KMSv2 and KMSv2KDF to GA
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: a9b1adbafc7fe52f669dc98aada21bc3e46cdce3
2023-10-24 09:50:45 -07:00
Anish Ramasekar bfdac7f8f4 [KMSv2] Add tracing
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 8d3a25c7c98d77419111a02917f459aab8970087
2023-10-09 23:43:46 +00:00
Wojciech Tyczyński 324d2bc3cf Cleanup paging parameters in etcd3 store
Kubernetes-commit: 3f4d3b67682335db510f85deb65b322127a3a0a1
2023-10-20 15:38:59 +02:00
Wojciech Tyczyński 5bf4f58ab8 Remove storageConfig.Paging parameter
Kubernetes-commit: b386120da239bf9652fc02b2d2cbbd0fcc3cd121
2023-10-20 15:35:58 +02:00
tao.yang 47998d1ee6 cleanup: omit comparison with bool constants
Signed-off-by: tao.yang <tao.yang@daocloud.io>

Kubernetes-commit: b35357b6c08f21ba0fd312536051394c2567ec79
2023-09-04 16:59:23 +08:00
machine424 340fd67b3a storage/etcd3/metrics: add unit test for apiserver_storage_objects
Kubernetes-commit: a96b97e40c0837f727c8e6d42f021b66922bb26c
2023-09-22 16:12:40 +02:00
Siyuan Zhang 77032c52b8 k8s.io/apiserver/storage: add 3 new unit tests for delete.
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 26a4e06c92c248748dd3c50c74d75f8adc3c6823
2023-10-11 10:38:01 -07:00
Siyuan Zhang 70af178d56 k8s.io/apiserver/storage: add a new TestCreate case.
Add a test case of create with rv set.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 2f923e356e042d9dce88e8f78abf30f414051e71
2023-10-11 10:24:31 -07:00
Siyuan Zhang a0dede6875 k8s.io/apiserver/storage: add some ResourceVersion validation in GetList unit tests
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: baac8bb573c8efb314b037f4fbac116556c03c83
2023-10-11 10:46:30 -07:00
Siyuan Zhang e15d4d2e0b k8s.io/apiserver/storage/etcd: refactor getCurrentState.
Extract getCurrentState as a separate method that can be reused.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: ebca5d438d9cb2c82d0b99dbcb0aeca8879db441
2023-10-11 09:56:07 -07:00
Lukasz Szaszkiewicz db34e9b7c7 storage/util: introduce HasInitialEventsEndBookmarkAnnotation
Kubernetes-commit: 70ae9c441cee9d78a88c0526592796ac4f491096
2023-09-18 11:39:08 +02:00
Lukasz Szaszkiewicz 90ba08909c storage/etcd: the watcher supports the API streaming
Kubernetes-commit: ca562fd280a9dd5db952eb8c14b93fcf6668cc49
2023-08-22 14:02:39 +02:00
Lukasz Szaszkiewicz e7424d8207 storage/testing/watcher_tests: add new flavour to RunWatchSemantics test and fill the gap
the new flavour proves that a storage layer follows RV
which is equal to the global rv

Kubernetes-commit: 0e238c4c28f11d1ec7cf5f506c81195c1cc463fe
2023-09-22 09:32:23 +02:00
Lukasz Szaszkiewicz f0dfb17949 storage/testing/watcher_tests: refactor RunWatchSemantics tests
Kubernetes-commit: dc5cfe3cff289ecb97153716bfd5817579d00fb3
2023-09-21 12:16:16 +02:00
Lukasz Szaszkiewicz c8dcfeed34 storage/testing/watcher_tests: make TestCacherWatchSemantics storage agnostic
Kubernetes-commit: 91bb75883c613d45563f3b7c01a69dde8194bfdc
2023-09-15 12:28:36 +02:00
Lukasz Szaszkiewicz c13e210d56 storage/testing/watcher_tests: move TestCacherWatchSemantics (no-op)
Kubernetes-commit: afbb1a6ef98b548b9e57b168614ca6e15fd0034c
2023-09-14 11:19:36 +02:00
Lukasz Szaszkiewicz 3c3bb86696 storage/testing/utils: add helper functions
Kubernetes-commit: 3a96baf3fa8e837159b27b98b05c5024143b06a3
2023-09-15 12:26:11 +02:00
Lukasz Szaszkiewicz fe7ae035ef storage/etcd3/watcher_test: refactor TestWatchErrorWhenNoNewFunc to a table test
Kubernetes-commit: 4b60c18183f11207d0cb2d89bf28090156222c61
2023-09-19 10:35:09 +02:00
Lukasz Szaszkiewicz f5adfca95e storage/testing: a simple refactor
Kubernetes-commit: 39af594f312ad46715dfb705d71ad4e607e27e5d
2023-09-14 11:16:10 +02:00
machine424 98d028d0d0 clarify the description of the metric apiserver_storage_objects
Kubernetes-commit: 084deb7a7cf147bbafd131ab1a36852a79f08a64
2023-09-12 11:16:36 +02:00
Rita Zhang 2bed5d11d9 kmsv2: add apiserver identity to metrics
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 43ccf6c4e8f173d981edebb6146c58b523fc21b7
2023-09-05 13:03:18 -07:00
Monis Khan 87aad93082 kmsv2: add legacy data integration test
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 95121fe846f875c4f3feb2b1bf42a9553566f097
2023-09-01 11:21:38 -04:00
Monis Khan 49c6151dee kmsv2: fix race in simpleCache.set when setting cache size metric
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: b10697c7880848d7ec110fd6b3e67015bbe74fa8
2023-08-27 15:14:04 -04:00
Lukasz Szaszkiewicz 957c305d94 storage/etcd: add TestWatchDispatchBookmarkEvents unit test
Kubernetes-commit: 875b00137fdfbc74756a0fc5b1c9b2adbeb78e55
2023-09-01 13:45:44 +02:00
Lukasz Szaszkiewicz 51e79bdd97 storage: document ProgressNotify from storage.ListOptions
At first glance, it seems that the fields storage.ListOptions.ProgressNotify and storage.ListOptions.Predicate.AllowWatchBookmarks
are the same. Unfortunately, this is not the case.

This PR documents the differences and motivations for why these fields are actually distinct.

Kubernetes-commit: 6058540f3d0edc405a1f1b8a96bd82ceca99c240
2023-09-01 10:48:36 +02:00
Marek Siarkowicz 8b7824a7e8 Avoid creating local variables that don't change
Having local variables gives false impression that this is overwritten
in the function block.

Kubernetes-commit: e01bd641447a315e28fab8148e99ac6afba9bcd7
2023-08-30 16:51:40 +02:00
Marek Siarkowicz 90adbb75eb Flatten switch case
Kubernetes-commit: 10553a1966892e305252c65ebeb9043416304f48
2023-08-30 16:42:34 +02:00
Marek Siarkowicz 5f9b5dda94 Refactor common WithRange case
From API call WithRange and WithPrefix work the same, they just set the range end.
The difference is when the range end is provided:
* WithRange(end) requires providing the end while calling
* WithPrefix() calculates the end based on key provided to the Get.

For example, those are equal:
* client.Get(ctx, "/pods/", WithPrefix())
* client.Get(ctx, "/pods/", WithRange(GetPrfixRangeEnd("/pods/")))

As keyPrefix is equal preparedKey there should not be a difference.

Kubernetes-commit: 1f4f2a5d6014dc8f98b25a9484d4a6064a6ae18e
2023-08-30 16:26:20 +02:00
Marek Siarkowicz a8fba27b08 Remove duplicated returnRV variable
returnRV was was equal to withRev, but updated at different time.

When preparing the request they are set equal to each other.

The only difference was during the for loop.
returnRV was always set no matter if pagination was enabled, while withRev only when paginating.

Kubernetes-commit: be4692864bb983e94e8d7b6b6aa1a9c22fe23bce
2023-08-30 17:36:05 +02:00
wackxu f40bea647b paginate initial list inside the storage watcher
Signed-off-by: wackxu <xushiwei5@huawei.com>

Kubernetes-commit: f5d6c65186d63647a46400762f849d500e6ac591
2023-07-28 16:43:34 +08:00
Lukasz Szaszkiewicz 206231e0d0 storage/etcd3: error when progressNotify option set and newFunc was provided for a registry
Kubernetes-commit: 1988c31fc8115bf9eec1adf2bffed3fd677d1a9f
2023-08-28 17:50:42 +02:00
Patrick Ohly 037fa7c2c2 apiserver: fix data race in etcd metrics
7a63997c8a1a9ba1 added a global variable which gets set multiple times by
different goroutines in integration tests, leading to a data race:

WARNING: DATA RACE
Write at 0x00000a626928 by goroutine 87080:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Previous write at 0x00000a626928 by goroutine 87079:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics.SetStorageMonitorGetter()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go:231 +0x104
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.(*EtcdOptions).ApplyWithStorageFactoryTo()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/etcd.go:242 +0xbd
  k8s.io/kubernetes/pkg/controlplane/apiserver.BuildGenericConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/config.go:124 +0x1c3d
  k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:218 +0xeb
  k8s.io/kubernetes/cmd/kube-apiserver/app.NewConfig()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/config.go:74 +0xd5
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:299 +0x2e97
  k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServerOrDie()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:423 +0xb2
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:486 +0x1dd
  k8s.io/kubernetes/test/integration/controlplane.testReconcilersAPIServerLease.func7()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/controlplane/kube_apiserver_test.go:488 +0x47

Mutex locking avoids the data race. Whether this variable really can be used
safely by those concurrent (?) tests is a different question...

Kubernetes-commit: 13a8ad12b8296c0360afe3f66218027dae6c1805
2023-08-25 10:42:17 +02:00
Lukasz Szaszkiewicz d8d65bebc5 storage/util: introduce AnnotateInitialEventsEndBookmark function
// AnnotateInitialEventsEndBookmark adds a special annotation to the given object
// which indicates that the initial events have been sent.
//
// Note that this function assumes that the obj's annotation
// field is a reference type (i.e. a map).

Kubernetes-commit: 47d9a47a08856613e2e6ae6aa8a1bdeb1e281f97
2023-08-24 12:19:10 +02:00