Commit Graph

159 Commits

Author SHA1 Message Date
Abu Kashem 3a0975b22c refactor: add delete options for Delete method in storage interface
Kubernetes-commit: bc0ea34bc380e073a7278216fe6690a1b9aee48c
2024-09-19 10:31:08 -04:00
Lukasz Szaszkiewicz 36e57697d1 apiserver/handlers/watch: encode initialEventsListBlueprint with watchEncoder (#127587)
* apiserver/handlers/get: construct versionedList

* storage/cacher: document caching the serialization of bookmark events

* endpoints/handlers/response: add watchListTransformer

* endpoints/handlers/watch: wire watchListTransformer

Kubernetes-commit: fbf1a0dc181ccbeb9925ad9c284d913a25c16562
2024-10-01 11:55:50 +00:00
Chaunceyctx f7eddd4bda send bookmark right now after sending all items in watchCache store
Kubernetes-commit: 723920253349ee3c272c5b5a77e9d19548c1533c
2024-08-29 17:31:36 +08:00
Vadim Rutkovsky 7da1f042b3 tracing: add span for cacher.Get
Also updates tracing integration tests for cacher.GetList

Kubernetes-commit: dff0075e7cd687f77fd38337e9ba487bb2437ecc
2024-09-12 14:18:40 +02:00
Marek Siarkowicz 6c5ee08ccf Implement fallback for consistent reads from cache
Kubernetes-commit: 35962561e44425fe5e23f19aeccba9269fab3a56
2024-07-30 18:57:22 +02:00
Wojciech Tyczyński a643e14347 Implement resilient watchcache initialization post-start-hook
Kubernetes-commit: a5772bd42593f6492f5169eef49bc9884f95abba
2024-06-13 11:02:18 +02:00
Lukasz Szaszkiewicz 5e71b4e71d apiserver/storage/watchcache: WaitUntilFreshAndList supports path prefix
Kubernetes-commit: 2f9660db6b0ba37ff383559b1b0324c635f1eb66
2024-06-26 14:34:32 +02:00
Lukasz Szaszkiewicz 54645c663b apiserver/storage/cacher: cache supports pagination
Kubernetes-commit: 09e85983d84b5f6c3fed6c09dd0adcbdde7e9d5f
2024-06-10 10:14:38 +02:00
Paco Xu 1c9c46740e Revert "apiserver/storage/cacher: consistent read from cache supports limit"
Kubernetes-commit: df17ea2e37d1b36dcda10fe20de5484c41c615b5
2024-06-26 11:51:12 +08:00
Lukasz Szaszkiewicz 2aadb1cb38 cacher: returns an error when watch list was requested and storage.RequestWatchProgress is disabled
Kubernetes-commit: cb8cbc60a15dd340ea834974355bd2f03453727c
2024-06-17 15:29:59 +02:00
Lukasz Szaszkiewicz 062ed2e6c0 apiserver/storage/cacher: cache supports pagination
Kubernetes-commit: 87536f367dd67b4fd9b1c7ee31189896921f2f9e
2024-06-10 10:14:38 +02:00
Marek Siarkowicz 2abbfffe9a Fix TestGetListRecursivePrefix for all types of LIST
Kubernetes-commit: c207f967ac3122d3a2a5ae4befdc5ada22347323
2024-06-19 10:31:37 +02:00
xyz-li e5e53b7acc apiserver: fix watch namespace
For request like '/api/v1/watch/namespaces/*', don't set scope.namespace.
Because the func `addWatcher` add a watcher to allWatchers with the value `scope.namespace` not empty.
But the function `dispatchEvent` dispatch event with an empty namespace.

Signed-off-by: xyz-li <hui0787411@163.com>

Kubernetes-commit: 818fabe37b3fd7cebe36a43244120388977373cd
2024-05-27 17:48:49 +08:00
Mauri de Souza Meneguzzo 48995fc82a Fix flaking TestStoreListResourceVersion
Kubernetes-commit: e59988d769a87ec3efb12af0f9909f7549eecf05
2024-06-11 23:12:23 -03:00
Wojciech Tyczyński 5a0e942d09 Implement ResilientWatchCacheInitialization
Kubernetes-commit: a8ef6e9f0104a44023162bb8229fb677ec80beb1
2024-04-29 14:19:46 +02:00
Eric Lin 9cab6e4eee cacher: apply key for initial events
For case of SendInitialEvents, a buffer of objects is created. That
process takes a significant amount of memory and CPU when the resource
is of a large volume. Many objects may be not relevant when key is provided.
This commit applies key when composing the buffer for SendInitialEvents.

Signed-off-by: Eric Lin <exlin@google.com>

Kubernetes-commit: d9c6c8aa5047d724e0ebc8907f5fee4b10012ae3
2024-05-04 10:46:02 +00:00
Lukasz Szaszkiewicz 175dd7c904 cacher: dispatchEvents is synced to watchCache.listResourceVersion
The internal informer populates the RV as soon as it conducts
The first successful sync with the underlying store.
The cache must wait until this first sync is completed to be deemed ready.
Since we cannot send a bookmark when the lastProcessedResourceVersion is 0,
we poll aggressively for the first list RV before entering the dispatch loop.

Kubernetes-commit: a20abdb1f425b215ce969ef7114281741fce249d
2024-06-03 10:26:50 +02:00
Lukasz Szaszkiewicz 438190132c storage/cacher: waitUntilWatchCacheFreshAndForceAllEvents checks if storage.RequestWatchProgress is supported
Kubernetes-commit: 2db69ae1cd0073da97a0efdacf5df0ad42abce4c
2024-05-14 12:53:43 +02:00
Lukasz Szaszkiewicz 0750099ebf storage/cacher: dispatchEvents use progressRequester
Kubernetes-commit: 33f81ee23fc3f48d2f3aee85661944792c117694
2024-05-08 15:43:23 +02:00
ah8ad3 8137baec35 Feat: warn user if etcd version is not supported for RequestWatchProgress
Kubernetes-commit: 9f8273a5c5d6d087c0839145616c828ab9a62f60
2024-04-29 15:56:35 +03:30
Marek Siarkowicz 13a815b7c8 Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage.
Kubernetes-commit: 0130072b053f85fb736c24d34552208cdd1bccfe
2024-03-14 15:20:29 +01:00
Marek Siarkowicz 67b6245fc3 Fix enabling consistent list from watch cache also works for resourceVersion=0
Kubernetes-commit: 0b8e79580eb3a63ca7707626b4894adfb9125586
2024-03-04 19:35:34 +01:00
Lukasz Szaszkiewicz b3f5f43260 storage/cacher: mark the addition of a metric for waitUntilFreshAndBlock as completed
Kubernetes-commit: 221ad9f7c25cc4da36e97c5feca3fc60bbe5bbfa
2024-03-05 10:23:23 +01:00
Marek Siarkowicz e810084a4b Prevent watch cache starvation, by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior
Kubernetes-commit: 31d404b182d2985ce0d3c43f75d80c29a708beda
2024-02-27 11:25:42 +01:00
xigang 2eff540b7c cleanup: if triggerValue has a value, fast break
Signed-off-by: xigang <wangxigang2014@gmail.com>

Kubernetes-commit: d72448a41c24911a57b24cabdef3ca63ee048bd4
2024-03-04 10:29:31 +08:00
Lukasz Szaszkiewicz 76172aaa1f storage/cacher: ensure the cache is at the Most Recent ResourceVersion when streaming was requested
Kubernetes-commit: f90bcf649e0f3dc233f49882468f949b0f00ac4f
2024-01-17 14:10:04 +01:00
Lukasz Szaszkiewicz ca8d0aaf91 client-go/reflector: make UseWatchList a pointer
until #115478(use streaming against the etcd storage)
is resolved the cacher need a way to disable the streaming.

Kubernetes-commit: 41e706600aea7468f486150d951d3b8948ce89d5
2024-01-19 13:48:29 +01:00
ahutsunshine 16536b9b63 support pod namespace indexer
fix comments

optimize code

small optimization for the namespace scope check

Kubernetes-commit: d8bd150784bb4825ae891dd0ec84625bdba0f2b8
2023-11-29 15:51:24 +08:00
Lukasz Szaszkiewicz 7c5f6db7bf cacher: when forgeting a watcher, call stopWatcherLocked multiple times
It's possible that the watcher is already not in the structure (e.g. in case of
simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked()
on a watcher multiple times.

Kubernetes-commit: 7e35823690df01bd019a88d3346bd3ac820afaca
2023-10-30 14:24:39 +01:00
Antonio Ojea bdad50b280 Revert "cacher: when forgeting a watcher, call stopWatcherLocked multiple times"
This reverts commit bbca4a4b9add0f6c58e132500fd89dd39ee077f4.

Kubernetes-commit: c2cb3209138d852520da2743b9bd3a9795b2b7fb
2023-10-31 15:28:01 +00:00
Lukasz Szaszkiewicz 3f81d0cca7 cacher: when forgeting a watcher, call stopWatcherLocked multiple times
It's possible that the watcher is already not in the structure (e.g. in case of
simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked()
on a watcher multiple times.

Kubernetes-commit: bbca4a4b9add0f6c58e132500fd89dd39ee077f4
2023-10-30 14:24:39 +01:00
tao.yang 47998d1ee6 cleanup: omit comparison with bool constants
Signed-off-by: tao.yang <tao.yang@daocloud.io>

Kubernetes-commit: b35357b6c08f21ba0fd312536051394c2567ec79
2023-09-04 16:59:23 +08:00
Wojciech Tyczyński 3030f660a8 Graduate APIListChunking to GA
Kubernetes-commit: 6acfa3cb4ac876e46ead5ba4772ba18e480435ce
2023-07-21 11:35:21 +02:00
Lukasz Szaszkiewicz 6f2daefacf storage/util: move GetCurrentResourceVersionFromStorage
Kubernetes-commit: 9a253d896a096b4e1ffccf4b1f84e5cac1e1aad0
2023-07-26 15:53:13 +02:00
Marek Siarkowicz 9002dac854 Implement ConsistentListFromCache feature gate
Request bookmark every 100ms when there is at least one request blocked on revision not present in watch cache.

Kubernetes-commit: 39bb8f4bb1d013937aceac6c387563ffe13545c5
2023-06-06 15:49:46 +02:00
Marek Siarkowicz cd751eb82e Implement conditionalProgressRequester that allows requesting watch progress notification if watch cache is not fresh
Kubernetes-commit: 98461be8ffa7383152c442414a16adb217e98080
2023-07-10 18:10:49 +02:00
Marek Siarkowicz 72bea89ab7 Refactor shouldDelegateList/shouldListFromStorage to better explain decisions
Kubernetes-commit: 406899360b55b3a0e28970f369288a2bb8a804fe
2023-06-21 14:02:46 +02:00
Wojciech Tyczyński 53b8c6919c Refactor cacher.ListerWatcher code structure
Kubernetes-commit: 3f247e59edfd4083242ad7271d076a38291760ff
2023-04-25 19:03:20 +02:00
Lukasz Szaszkiewicz 0df626818a cacher: do not popExpiredWatchers when the cacher hasn't dispatched any event
If the cacher hasn't seen any event (when lastProcessedResourceVersion is zero) and
the bookmarkTimer has ticked then we shouldn't popExpiredWatchers. This is
because the watchers wont' be re-added and will miss future bookmark events when
the cacher finally receives an event via the c.incoming chan.

Kubernetes-commit: 6db4cbfde7babfb34f5cd1059c769ec2d870f12a
2023-04-19 15:29:13 +02:00
Lukasz Szaszkiewicz 55dd2b5223 cacher avoid double locking (#117410)
* cacher: remove locking from watcherBookmarkTimeBuckets

it turns out that the watcherBookmarkTimeBuckets
is called from only three places/methods: startDispatching, finishDispatching and Watch.
All these methods acquire c.Lock() before touching watcherBookmarkTimeBuckets.

Thus we could remove explicit locking in
watcherBookmarkTimeBuckets since the access is already synced.

* cacher: rename watcherBookmarkTimeBuckets methods to indicate that proper synchronisation must be used

Kubernetes-commit: eab66a687b282266f0520b79166f7f55828ffd28
2023-04-19 15:13:11 +02:00
Lukasz Szaszkiewicz cea854521c cacher: prevent a potential deadlock
waitUntilWatchCacheFreshAndForceAllEvents must be called without
a read lock held otherwise the watchcache won't be able to make
progress (i.e. the watchCache.processEvent method that requries acquiring an exclusive lock)

the deadlock can happen only when the alpha watchlist feature flag is on
and the client specifically requests streaming.

Kubernetes-commit: 476e407ffd2ab393840d3f7a9fd01b71698738a3
2023-04-06 09:54:02 +02:00
wangxiang b1bc32fa21 optimize watch-cache getlist (#116327)
* ftr(watch-cache): add benchmarks

* ftr(kube-apiserver): faster watch-cache getlist

* refine: testcase name

* - refine var name make it easier to convey meaning
- add comment to explain why we need to apply for a slice of runtime.Object instead of making a slice of ListObject.Items directly.

Kubernetes-commit: 75f17eb38fc8bbcb360d43dffce6e27a7159d43f
2023-04-12 00:28:10 +00:00
Wojciech Tyczyński 107350e97d Fix missed watch events
Kubernetes-commit: 3277d85604d7d8845f56f670fb83b65982cdddf3
2023-03-01 12:16:40 +01:00
Lukasz Szaszkiewicz 515ef372fb cache/controller: Add ENABLE_CLIENT_GO_WATCH_LIST_ALPHA
Kubernetes-commit: 966b26d55c22f7fbf20841a3a993de4f984d88db
2023-03-07 12:34:11 +01:00
Lukasz Szaszkiewicz 04eaa5f0f0 cacher: Add WaitUntilWatchCacheFreshAndForceAllEvents method
this method waits until cache is at least
as fresh as given requestedWatchRV if sendInitialEvents was requested.
Additionally, it instructs the caller whether it should ask for
all events from the cache (full state) or not.

Kubernetes-commit: 21fb98105043d1a15ef48089ef231931851d2d15
2023-03-06 14:58:31 +01:00
Lukasz Szaszkiewicz ff7564e339 cacher: Add support for consistent streaming
design details https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/3157-watch-list#design-details

Kubernetes-commit: 7c7e7733050c22ab855c1b1c717b3514b953503e
2023-02-27 13:32:38 +01:00
Lukasz Szaszkiewicz 10e39104a7 Disable SendInitialEvents for etcd and watch cache
Kubernetes-commit: 7b67bbbcf361983e563663424e13f2b6cf6d5392
2023-03-01 11:00:12 +01:00
Antonio Ojea 33153a1931 cacher allow context cancellation if not ready (#116024)
* cacher allow context cancellation if not ready

Replace the sync.Cond variable with a channel so we can use the
context cancellation signal.

Co-authored-by: Wojciech Tyczy<C5><84>ski <wojtekt@google.com>
Change-Id: I2f75313a6337feee440ece4c1e873c32a12560dd

* wait again on pending state

Change-Id: I1ad79253a5a5d56a4d9611125825b1f7ad552be8

---------

Co-authored-by: Wojciech Tyczy<C5><84>ski <wojtekt@google.com>

Kubernetes-commit: 3b17aece1fa492e98aa82b948597b3641961195f
2023-02-28 11:23:16 +00:00
Jordan Liggitt 8d0e8f8cb8 Partition watchers by namespace/name scope
Kubernetes-commit: 1c2fa0c7f791cbe28ee54d957d6f8a1120d91018
2023-02-05 18:46:19 -05:00
Wojciech Tyczyński 4bbfd23c76 Minor cleanup in cacher
Kubernetes-commit: c0c925f6feb3ab961e088c5aad1376b6de68665d
2022-12-22 11:44:18 +01:00