Commit Graph

734 Commits

Author SHA1 Message Date
Wojciech Tyczyński b1a1768456 Relax WatchSemanticsTest to make it faster
Kubernetes-commit: a26ee123952eeb86467e61ff2eb744c9e736a045
2024-03-19 20:23:56 +01:00
Lukasz Szaszkiewicz 9bfd27dc1c apiserver/storage: use metav1.InitialEventsAnnotationKey const
Kubernetes-commit: ed2b92c3dfb7d56155021824f0d9a61609dac051
2024-03-11 11:40:21 +01:00
Lukasz Szaszkiewicz d4f6eff589 apiserver/pkg/storage/utils: remove unused EverythingFunc
since we don't provide compatibility guarantees for the storage
package it is okay to simply remove unused function.

Kubernetes-commit: a40f25f8e6516d1a59169cf88db8b3850a8c48c7
2024-03-11 11:47:02 +01:00
Davanum Srinivas a22f7973bf Rename `cluster` to `storage_cluster_id` for apiserver_storage_size_bytes metric
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: fb5140f562289f3ffe596b4a3af4650b850a0941
2024-04-11 15:06:03 -04:00
Marek Siarkowicz 7a3a726271 Undo double run of the TestWatchSemantics test to avoid hitting timeout
Kubernetes-commit: 225749eb68b3aa3485a1f2dc5d0cafb5dfe53421
2024-03-19 15:16:52 +01:00
Marek Siarkowicz 13a815b7c8 Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage.
Kubernetes-commit: 0130072b053f85fb736c24d34552208cdd1bccfe
2024-03-14 15:20:29 +01:00
Marek Siarkowicz 67b6245fc3 Fix enabling consistent list from watch cache also works for resourceVersion=0
Kubernetes-commit: 0b8e79580eb3a63ca7707626b4894adfb9125586
2024-03-04 19:35:34 +01:00
Lukasz Szaszkiewicz b8c7d7868f apiserver/storage/cacher: deflake TestGetListNonRecursiveWithConsistentListFromCache
Kubernetes-commit: c44cc9a575f3bf1248b2fdc8e3a7ba61ab844618
2024-03-14 09:08:29 +01:00
Lukasz Szaszkiewicz a2c5722d64 apiserver/storage/cacher: decrease running time of TestWatchNotHangingOnStartupFailure
before:
go test -v -race -count 1 -run ^TestWatchNotHangingOnStartupFailure$
ok  	k8s.io/apiserver/pkg/storage/cacher	6.775s

after:
go test -v -race -count 1 -run ^TestWatchNotHangingOnStartupFailure$
ok  	k8s.io/apiserver/pkg/storage/cacher	2.781s

Kubernetes-commit: f5d945eb43c7bf8036a4bad8c22448e1146a7498
2024-03-14 08:59:47 +01:00
Lukasz Szaszkiewicz c14671349b apiserver/storage/cacher: decrease of running time of TestWaitUntilWatchCacheFreshAndForceAllEvents
The individual cases can be safely run in parallel.

Before
go test -race  -run TestWaitUntilWatchCacheFreshAndForceAllEvents
ok  	k8s.io/apiserver/pkg/storage/cacher	10.787s

After:
go test -race  -run TestWaitUntilWatchCacheFreshAndForceAllEvents
ok  	k8s.io/apiserver/pkg/storage/cacher	4.857s

Kubernetes-commit: 3ecbb4dee00a5dd1e43e24a5952c2a90ef507ef1
2024-03-12 10:50:44 +01:00
Lukasz Szaszkiewicz 361687d2ad apiserver/storage/cacher: decrease the running time of tests in the cacher package.
It turns out that kube has a custom timeout for tests of 3 minutes.
The tests in the cacher package are utilizing nearly the
entire time and are being terminated, resulting in failing jobs.

Before the change, the TestWatchSemantics took ~43s to run. With this simple change, it now takes ~18s.

When we created the tests, we didn't measure the running time and assumed that waiting 1 second on a watch channel
to make sure no more events are received was sufficient.
This PR decreases the waiting time to 300 milliseconds.
Modern computers can perform many tasks within that time.
In addition to that, the tests are serial in nature, meaning that there is no other
actor that could add items to the database, which could result in receiving new items.

After the change the total running time decreased by 17%.
Before the tests needed ~176s after they need ~146s.
The changes also improved TestWatchSemanticInitialEventsExtended.

Kubernetes-commit: 5a74c8e2202044b664efce4be5d86d700e74506f
2024-03-12 09:15:55 +01:00
Lukasz Szaszkiewicz 80f9ab2a6a apiserver/storage/cacher: decrease of running time of TestEmptyWatchEventCache
updates the test to wait 300 ms instead of 3s
the watch was established otherwise
we would be blocking on a call to cache.Watch(...)
in addition to that, the tests are serial in nature,
meaning that there is no other actor
that could add items to the database,
which could result in receiving new items.

Before:
go test -race  -run TestEmptyWatchEventCache
ok  	k8s.io/apiserver/pkg/storage/cacher	8.450s

After:
go test -race  -run TestEmptyWatchEventCache
ok  	k8s.io/apiserver/pkg/storage/cacher	2.635s

Kubernetes-commit: 926122c035a4f47a880db24d1a0be7ec129dd44d
2024-03-12 13:34:04 +01:00
Marek Siarkowicz 483da2032d Fix non-recursive list when consistent list from cache is enabled
Kubernetes-commit: a527cab9fce0b0234db8b4e1e95a0fc20e135df1
2024-03-04 17:59:04 +01:00
Marek Siarkowicz 3a83dc12eb Fix SetFeatureGateDuringTest handling of Parallel tests
Stop using defer as parallel subtest will might result in main test
finishing before subtest.

Fatal when same flag is set twice.

Kubernetes-commit: 9fcf279e2b91e7549190a433373f256fb5aebe85
2024-03-05 21:56:40 +01:00
Lukasz Szaszkiewicz b3f5f43260 storage/cacher: mark the addition of a metric for waitUntilFreshAndBlock as completed
Kubernetes-commit: 221ad9f7c25cc4da36e97c5feca3fc60bbe5bbfa
2024-03-05 10:23:23 +01:00
Marek Siarkowicz 743b53428c Test that separation of streams work by using progress notifies
Kubernetes-commit: 1cf4cec449cb29718a694e25f4750452af3f491d
2024-02-29 17:51:46 +01:00
Marek Siarkowicz e810084a4b Prevent watch cache starvation, by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior
Kubernetes-commit: 31d404b182d2985ce0d3c43f75d80c29a708beda
2024-02-27 11:25:42 +01:00
Sunil Shivanand 9ffd1e2039 Add apiserver_watch_cache_read_wait metric to cache refresh time
Signed-off-by: Sunil Shivanand <padlar@live.com>

Kubernetes-commit: e6ed0f37c65fb22c16f5afa408bc4de166070ebc
2024-02-08 12:39:50 +01:00
xigang 2eff540b7c cleanup: if triggerValue has a value, fast break
Signed-off-by: xigang <wangxigang2014@gmail.com>

Kubernetes-commit: d72448a41c24911a57b24cabdef3ca63ee048bd4
2024-03-04 10:29:31 +08:00
Lukasz Szaszkiewicz 816c9a3d12 apiserver/storage: improve RunWatchSemanticInitialEventsExtended test
changes the test to populate the underlying data store with
more data to trigger potential ordering issues.

Kubernetes-commit: 20ded275705a6e11c1113cbeedad4de94e2dc666
2024-01-10 11:08:35 +01:00
Wojciech Tyczyński 45b7f21179 Ensure that initial events are sorted for WatchList
Kubernetes-commit: 92bdc7b3873800e6130176e49acdf5e17110e5b9
2023-09-26 18:39:44 +02:00
Lukasz Szaszkiewicz e53bac21d8 storage/watch_cache: rework getAllEventsSinceLocked
Kubernetes-commit: ecaf2093f51fed5f544520b0ac00fb33a474b7f5
2024-02-26 12:22:05 +01:00
Lukasz Szaszkiewicz 19bd56380e storage/cacher: add TestGetWatchCacheResourceVersion, TestGetBookmarkAfterResourceVersionLockedFunc
Kubernetes-commit: d629d3fa355ec90f618663b0933d28d335489c54
2024-02-21 10:06:42 +01:00
Lukasz Szaszkiewicz 76172aaa1f storage/cacher: ensure the cache is at the Most Recent ResourceVersion when streaming was requested
Kubernetes-commit: f90bcf649e0f3dc233f49882468f949b0f00ac4f
2024-01-17 14:10:04 +01:00
Han Kang f615696539 bump the stability level of apiserver_storage_size_bytes to STABLE
Kubernetes-commit: f38852768e312fe7b9775b92f7228371a0a96f90
2024-02-16 09:13:46 -08:00
HirazawaUi bc8676d59a Add decoding time to the audit log
Kubernetes-commit: 20fe2a3539e90f7554f94359ac3b4058a5bbb363
2023-10-25 22:52:11 +08:00
Lukasz Szaszkiewicz ca8d0aaf91 client-go/reflector: make UseWatchList a pointer
until #115478(use streaming against the etcd storage)
is resolved the cacher need a way to disable the streaming.

Kubernetes-commit: 41e706600aea7468f486150d951d3b8948ce89d5
2024-01-19 13:48:29 +01:00
weilaaa febd537a31 use build-in max and min func to instead of k8s.io/utils/integer funcs
Kubernetes-commit: eb8f3f194fed16484162aebdaab69168e02f8cb4
2023-12-15 15:09:11 +08:00
James Blair b6487a8ac1 Fix etcd repository path to prevent redirects.
Signed-off-by: James Blair <mail@jamesblair.net>

Kubernetes-commit: b6c1f8ef08c3451f17048447e107c509a8ed950e
2023-11-02 09:31:37 +13:00
Wojciech Tyczyński 2b59a3f14c Don't sort under lock
Kubernetes-commit: 3e1dbccf4d3de59b31596f1664ff5ac9c02d1eea
2023-11-23 18:13:43 +01:00
Robert Svarinskis fdd26489db Fix etcd storage_events_received_total metric not being registered
Kubernetes-commit: ceb7ca93875cd74bea60500812cf7f024d3b1f2d
2023-12-04 09:12:21 +00:00
ahutsunshine 16536b9b63 support pod namespace indexer
fix comments

optimize code

small optimization for the namespace scope check

Kubernetes-commit: d8bd150784bb4825ae891dd0ec84625bdba0f2b8
2023-11-29 15:51:24 +08:00
HirazawaUi 88cd5253fa make the decode function respect the timeout context
Kubernetes-commit: 3fb16abfca43f53048a28758f2366cb52e31d2b7
2023-11-08 23:39:10 +08:00
Siyuan Zhang cef6db2cfb k8s.io/apiserver/storage/etcd: refactor etcd GetList.
Extract the logic to determine withRev to a separate method for better readability.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 624169c5b50ee8a6e9a761e9488134985334817e
2023-10-20 12:59:46 -07:00
Lukasz Szaszkiewicz 7c5f6db7bf cacher: when forgeting a watcher, call stopWatcherLocked multiple times
It's possible that the watcher is already not in the structure (e.g. in case of
simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked()
on a watcher multiple times.

Kubernetes-commit: 7e35823690df01bd019a88d3346bd3ac820afaca
2023-10-30 14:24:39 +01:00
Antonio Ojea bdad50b280 Revert "cacher: when forgeting a watcher, call stopWatcherLocked multiple times"
This reverts commit bbca4a4b9add0f6c58e132500fd89dd39ee077f4.

Kubernetes-commit: c2cb3209138d852520da2743b9bd3a9795b2b7fb
2023-10-31 15:28:01 +00:00
Wojciech Tyczynski 6caf326620 Revert "Make the decode function respect the timeout context"
Kubernetes-commit: 98a2f22e740ccd2c30711f1b21d6383f1b91595e
2023-10-31 16:27:17 +01:00
Lukasz Szaszkiewicz 3f81d0cca7 cacher: when forgeting a watcher, call stopWatcherLocked multiple times
It's possible that the watcher is already not in the structure (e.g. in case of
simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked()
on a watcher multiple times.

Kubernetes-commit: bbca4a4b9add0f6c58e132500fd89dd39ee077f4
2023-10-30 14:24:39 +01:00
Siyuan Zhang ef409f941b k8s.io/apiserver/storage/etcd: refactor etcd GetList.
Reorder some code.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: a968f51fa2f87ed57f9e48ba436e11421c403b27
2023-10-20 12:39:51 -07:00
Siyuan Zhang 49886c205c k8s.io/apiserver/storage/etcd: refactor etcd GetList.
reduce redundant update of withRev after request.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 84ec5e2eccbc07b17f3b3e3e00dc3996105e0346
2023-10-20 12:06:46 -07:00
HirazawaUi be73f76247 Make the decode function respect the timeout context
Kubernetes-commit: f78b367db6393a449b8f456e725cbe155d9b90e6
2023-10-25 23:12:16 +08:00
Nilekh Chaudhari f0fe5d558c chore: updates api doc
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: d3de47ceeedd1fae3f3c95595186a028002526d6
2023-10-17 22:17:43 +00:00
Rita Zhang 26219aabef [KMSv2] promote KMSv2 and KMSv2KDF to GA
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: a9b1adbafc7fe52f669dc98aada21bc3e46cdce3
2023-10-24 09:50:45 -07:00
Anish Ramasekar bfdac7f8f4 [KMSv2] Add tracing
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 8d3a25c7c98d77419111a02917f459aab8970087
2023-10-09 23:43:46 +00:00
Wojciech Tyczyński 324d2bc3cf Cleanup paging parameters in etcd3 store
Kubernetes-commit: 3f4d3b67682335db510f85deb65b322127a3a0a1
2023-10-20 15:38:59 +02:00
Wojciech Tyczyński 5bf4f58ab8 Remove storageConfig.Paging parameter
Kubernetes-commit: b386120da239bf9652fc02b2d2cbbd0fcc3cd121
2023-10-20 15:35:58 +02:00
tao.yang 47998d1ee6 cleanup: omit comparison with bool constants
Signed-off-by: tao.yang <tao.yang@daocloud.io>

Kubernetes-commit: b35357b6c08f21ba0fd312536051394c2567ec79
2023-09-04 16:59:23 +08:00
machine424 340fd67b3a storage/etcd3/metrics: add unit test for apiserver_storage_objects
Kubernetes-commit: a96b97e40c0837f727c8e6d42f021b66922bb26c
2023-09-22 16:12:40 +02:00
Siyuan Zhang 77032c52b8 k8s.io/apiserver/storage: add 3 new unit tests for delete.
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 26a4e06c92c248748dd3c50c74d75f8adc3c6823
2023-10-11 10:38:01 -07:00
Siyuan Zhang 70af178d56 k8s.io/apiserver/storage: add a new TestCreate case.
Add a test case of create with rv set.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Kubernetes-commit: 2f923e356e042d9dce88e8f78abf30f414051e71
2023-10-11 10:24:31 -07:00