Commit Graph

660 Commits

Author SHA1 Message Date
Lukasz Szaszkiewicz d8d65bebc5 storage/util: introduce AnnotateInitialEventsEndBookmark function
// AnnotateInitialEventsEndBookmark adds a special annotation to the given object
// which indicates that the initial events have been sent.
//
// Note that this function assumes that the obj's annotation
// field is a reference type (i.e. a map).

Kubernetes-commit: 47d9a47a08856613e2e6ae6aa8a1bdeb1e281f97
2023-08-24 12:19:10 +02:00
Lukasz Szaszkiewicz f75c503352 storage/factory: extend the Create method by newList and resourcePrefix params
Kubernetes-commit: ccabc01093a1344ebb27c32c946e9da3b8e91fd2
2023-07-28 09:53:01 +02:00
Lukasz Szaszkiewicz 52849e2b21 storage/cacher: duplicate logging when a watch is closed due to unresponsiveness.
Kubernetes-commit: 1327a57aaf8f5ffaea3d33204dbe64dccef4e0ef
2023-08-23 12:51:50 +02:00
Wojciech Tyczyński f56b9ee7f5 Graduate RemainingItemCount to GA
Kubernetes-commit: 4e2e059c7b205d2e4b246a262128223258a49498
2023-07-21 15:22:51 +02:00
Wojciech Tyczyński 3030f660a8 Graduate APIListChunking to GA
Kubernetes-commit: 6acfa3cb4ac876e46ead5ba4772ba18e480435ce
2023-07-21 11:35:21 +02:00
Lukasz Szaszkiewicz 7ecfbf7914 storage/etcd: simplify passing the transformer
Kubernetes-commit: a5600b6925169a7d99376d52472e5f8e8635082b
2023-08-21 12:27:41 +02:00
Lukasz Szaszkiewicz dc7ff6e034 storage/etcd: remove newWatcher function
Kubernetes-commit: 6161752ecbe434b8fb157dd3a30bb5c0cef7a23e
2023-08-21 12:13:12 +02:00
Lukasz Szaszkiewicz 6f2daefacf storage/util: move GetCurrentResourceVersionFromStorage
Kubernetes-commit: 9a253d896a096b4e1ffccf4b1f84e5cac1e1aad0
2023-07-26 15:53:13 +02:00
Lukasz Szaszkiewicz 2d9be35745 storage/etcd: no-op, refactor watcher.Watch method signature
Kubernetes-commit: f7e659db236286ca15707bf08acb08dc0ac4ab0e
2023-07-24 12:33:03 +02:00
Antonio Ojea 49f7deccb5 fix race on etcd client constructor for healthchecks
Change-Id: Id29b5b377989dcb5377316cfcdea367071a47365

Kubernetes-commit: 77b3bb0f69bfa4e5eb56ba484f724476304616cc
2023-08-08 13:55:14 +00:00
Rita Zhang 7ebae7f76f kmsv2: add metric for DEK cache filled
Signed-off-by: Rita Zhang <rita.z.zhang@gmail.com>

Kubernetes-commit: 3cbecf218dfea2e99ae95310ac03406d1d87a072
2023-08-09 12:28:01 -07:00
Damien Grisonnet 245d131967 apiserver/etcd3: fix segv during metric collection
Fix a segfault when collecting the storage size metrics when the getters
used to collect the data on etcd haven't been initialized properly. This
happens when the EtcdOptions are not applied which is the case for
aggregated apiservers that don't care about storage.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

Kubernetes-commit: c6efaf16c1ed07ce37485b7a272628f653cbf06f
2023-08-10 17:01:17 +02:00
Monis Khan 8e93c650b5 kmsv2: KDF based nonce extension
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: bf49c727ba10881d5378e9242f31dc00dede51be
2023-03-25 14:41:04 -04:00
Marek Siarkowicz e9acd0c76d Fix the semantic meaning of etcd server within component statuses and metrics.
Instead of numerating all the etcd endpoints known by apiserver, we will
group them by purpose. `etcd-0` will be the default etcd, `etcd-1` will
be the first resource override, `etcd-2` will be the second override and
so on.

Kubernetes-commit: 03aad1f823cb719fa6e6b6d33fefa2a2140cc760
2023-07-19 14:25:54 +02:00
Nilekh Chaudhari 3373784322 feat: improves metric and logging
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: 2c8288ac873b89ae6e351df5e805b1a825aae60d
2023-07-07 23:01:05 +00:00
Nilekh Chaudhari 36a1803532 chore: hashes keyID
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

Kubernetes-commit: 131216fa8f2dd13f2585e2010717733f4cb2c1e2
2023-06-29 20:32:27 +00:00
Marek Siarkowicz 573a8d6d05 Improve apiserver storage size metric to allow it's graduation
Change name to make it compliant with prometheus guidelines.
Calculate it on demand instead of periodic to comply with prometheus standards.
Replace "endpoint" with "server" label to make it semantically consistent with storage factory

Kubernetes-commit: 7a63997c8a1a9ba14f2bdc478fdf33cf88f48d80
2023-06-22 11:56:09 +02:00
Marek Siarkowicz 9002dac854 Implement ConsistentListFromCache feature gate
Request bookmark every 100ms when there is at least one request blocked on revision not present in watch cache.

Kubernetes-commit: 39bb8f4bb1d013937aceac6c387563ffe13545c5
2023-06-06 15:49:46 +02:00
Marek Siarkowicz 23cd6bbea9 Fix TestConditionalProgressRequester and TestWaitUntilFreshAndListTimeout flakes
Kubernetes-commit: c1decb6763d2abf76d96aee8641ad56a23e0ba52
2023-07-12 13:36:51 +02:00
Marek Siarkowicz cd751eb82e Implement conditionalProgressRequester that allows requesting watch progress notification if watch cache is not fresh
Kubernetes-commit: 98461be8ffa7383152c442414a16adb217e98080
2023-07-10 18:10:49 +02:00
Marek Siarkowicz df9896fd00 Test consistent List
Kubernetes-commit: b36fdd68b72cd5c64ea5be3917846067644da983
2023-07-03 17:22:22 +02:00
Wojciech Tyczyński b560936651 Generalize watch storage tests
Kubernetes-commit: 8266c4d934d42a5175a84bff10fda4bf36f13817
2023-06-27 18:12:30 +02:00
Monis Khan b90de05302 no-op: split transformer interface
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 9354e78289fde933f485c663df960b0bc6e24df5
2023-03-24 16:23:40 -04:00
Marek Siarkowicz eda3579943 Test all cases for LIST request cache bypass
Kubernetes-commit: e4cf103c9b98391e5e135c3991cd8f380bfe22fe
2023-06-21 14:12:04 +02:00
Marek Siarkowicz 72bea89ab7 Refactor shouldDelegateList/shouldListFromStorage to better explain decisions
Kubernetes-commit: 406899360b55b3a0e28970f369288a2bb8a804fe
2023-06-21 14:02:46 +02:00
Marek Siarkowicz 96d1438818 Refactor WithRequireLeader to make it part of the etcd store
Kubernetes-commit: a9af2de8fdd1f2799b2017b19027958c69ffb04d
2023-06-21 14:21:24 +02:00
Marek Siarkowicz fb965cd2c4 Make etcd component status consistent with health probes
Co-authored-by: Antonio Ojea <antonio.ojea.garcia@gmail.com>

Kubernetes-commit: a60314c47ef28dd04b086a2ad4b4bfd40b6f86a4
2023-06-05 16:26:12 +02:00
Madhav Jivrajani d79ec7e54a storage: Move cacher tests to reside with the cacher code
Doing this allows us to implement some more nuanced cacher manipulations
to be used in testing. For ex: implementing a test-only compaction method
for the watch cache.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 6d66fbc6b670f1120a9041873bb8d1a0655bbefc
2023-06-06 14:11:03 +05:30
Madhav Jivrajani 1fd6d59169 cacher: Move common testing utils to a single file
This commit prepares for when cacher tests are moved here
from the `tests` package. Tests in that package redeclare
some of the testing utils that exist here, so dedup-ing them.

This commit also adapts to any changes in test util signatures.
There are still some utils that can be reused but currently are
highly specific to some tests. (ex: watch_cache_test.go)

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 70978e4af619819787a4eb544ffd732aa7954d76
2023-06-06 14:08:25 +05:30
Madhav Jivrajani 90cfcdff83 storage: Add MadhavJivrajani as reviewer
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 7f87ec9406a748f2be0bca9e12b895c7dba486ba
2023-06-07 17:37:44 +05:30
Dr. Stefan Schimanski 8f40ac3bd1 k8s.io/apiserver: remove skewed completion from EtcdOptions
Kubernetes-commit: e9e4acb1dde69243a6e675e58833ae7936df9ce5
2023-06-02 20:25:31 +02:00
Eric Lin edd537cbe3 Do not copy bytes for cached serializations
Since cachingObject has the encoded data cached and they are not
supposed to change. It's memory efficient to just copy the slice
references.

Signed-off-by: Eric Lin <exlin@google.com>

Kubernetes-commit: 3085b57869a2a7bf5290ab97facaf17fedfa88a0
2023-05-31 19:12:32 +00:00
Madhav Jivrajani 3a79e5d43c storage: Implement TestWatchFromZero for cacher
There exists a storage test to test for rv=0 and production
of ADDED events. This commit adapts the test to be used for
the watch cache as well.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 4d85a1f00cb0f1350cf8495925be0e8bfed59a15
2023-05-30 19:46:07 +05:30
Monis Khan 95d403a421 kmsv2: ignore cache key expiration on reads
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 86a258b13a2a8dbb76cbb8be49cccbbf37b52b64
2023-05-17 13:26:08 -04:00
Anish Ramasekar aceaeaa13e update err status code check in transformation metrics
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 526d45416fb2748f3027ec8735de32d460bd25a8
2023-05-17 19:48:40 +00:00
Daniel Smith 6abce7684c lavalamp is taking a long break
Kubernetes-commit: 1ffe3f467e8b8033312b7c68943d58125fd27663
2023-05-11 16:43:38 +00:00
Lukasz Szaszkiewicz decaec94cd storage/etcd: skip SendInitialEvents if the request is backward compatible
otherwise an error will be returned.
backward compatibility is defined as RV = "" || RV = "O" and AllowWatchBookmark is set to false.
in that case we rely on 267eb25e60/staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go (L260)

Kubernetes-commit: f2de1a00b8fd7a22f9d0b2c1ace69be41d304f83
2023-05-08 13:04:31 +02:00
kkkkun f4c24327c3 Fix etcd3 config changed
Kubernetes-commit: 57969c1b52ce0c3c27ccbf02ecc52ebc1b8533d9
2023-05-06 19:40:40 +08:00
Wojciech Tyczyński d9c1a1d082 Refactor some watchcache tests
Kubernetes-commit: 1eca720dcc727b5deeeeb1164689d42c6cc316eb
2023-04-27 11:59:33 +02:00
Wojciech Tyczyński 53b8c6919c Refactor cacher.ListerWatcher code structure
Kubernetes-commit: 3f247e59edfd4083242ad7271d076a38291760ff
2023-04-25 19:03:20 +02:00
Wojciech Tyczyński 68a91d7597 Enable a bunch of generic storage tests for watchcache.
Kubernetes-commit: 88e2c34439016e5feddfed529698ed71e81904d9
2023-04-17 19:03:51 +02:00
Wojciech Tyczyński ca97e27fb8 Refactor watch bookmark tests to allow sharing between etcd3 and watchcache
Kubernetes-commit: 0297329795efc0f2bd24751e18f66b0aa79c222d
2023-04-17 14:22:52 +02:00
Wojciech Tyczyński 072d278e39 Generalize few watch tests from cacher
Kubernetes-commit: 45e836a968acc113cb03768cd8c730bea89bd332
2023-04-17 21:48:08 +02:00
Lukasz Szaszkiewicz 0df626818a cacher: do not popExpiredWatchers when the cacher hasn't dispatched any event
If the cacher hasn't seen any event (when lastProcessedResourceVersion is zero) and
the bookmarkTimer has ticked then we shouldn't popExpiredWatchers. This is
because the watchers wont' be re-added and will miss future bookmark events when
the cacher finally receives an event via the c.incoming chan.

Kubernetes-commit: 6db4cbfde7babfb34f5cd1059c769ec2d870f12a
2023-04-19 15:29:13 +02:00
Lukasz Szaszkiewicz 55dd2b5223 cacher avoid double locking (#117410)
* cacher: remove locking from watcherBookmarkTimeBuckets

it turns out that the watcherBookmarkTimeBuckets
is called from only three places/methods: startDispatching, finishDispatching and Watch.
All these methods acquire c.Lock() before touching watcherBookmarkTimeBuckets.

Thus we could remove explicit locking in
watcherBookmarkTimeBuckets since the access is already synced.

* cacher: rename watcherBookmarkTimeBuckets methods to indicate that proper synchronisation must be used

Kubernetes-commit: eab66a687b282266f0520b79166f7f55828ffd28
2023-04-19 15:13:11 +02:00
Tim Hockin 0165503c5a Replace uses of ObjectReflectDiff with cmp.Diff
ObjectReflectDiff is already a shim over cmp.Diff, so no actual output
or behavior changes

Kubernetes-commit: bc302fa4144d21a338683cd83701661f97be4aba
2023-03-23 11:34:03 -07:00
iyear b1805a9a4d apiserver/etcd: add request and request_error metrics (#117222)
* apiserver/etcd: add request and request_error metrics

Signed-off-by: iyear <ljyngup@gmail.com>

* apiserver/etcd: rename etcdRequestCounts metric

Signed-off-by: iyear <ljyngup@gmail.com>

---------

Signed-off-by: iyear <ljyngup@gmail.com>

Kubernetes-commit: 4f6714ef42dcdb1b35bb2d8581c1ef83d59f02d1
2023-04-12 22:02:45 +08:00
Lukasz Szaszkiewicz cea854521c cacher: prevent a potential deadlock
waitUntilWatchCacheFreshAndForceAllEvents must be called without
a read lock held otherwise the watchcache won't be able to make
progress (i.e. the watchCache.processEvent method that requries acquiring an exclusive lock)

the deadlock can happen only when the alpha watchlist feature flag is on
and the client specifically requests streaming.

Kubernetes-commit: 476e407ffd2ab393840d3f7a9fd01b71698738a3
2023-04-06 09:54:02 +02:00
Lukasz Szaszkiewicz 89471f9967 watchcache: currentCapacity acquires read lock
Kubernetes-commit: f138d29bf2df1a67883e6c16249eb4f26d34ec0c
2023-04-07 10:16:46 +02:00
Wojciech Tyczyński e4d6a4de7c Deflake TestGetListNonRecursive
Kubernetes-commit: 05f4e497f1d692031b31c27098abc1e838e8814b
2023-03-29 13:46:49 +02:00