Commit Graph

881 Commits

Author SHA1 Message Date
Marek Siarkowicz a559cb8be1 Unify references to group resource in metrics
Skip apiserver_storage_objects as it's a stable metrics that doesn't
have "group" label. As defined in KEP-1209 adding a label is considered
a breaking change so this change should be done via deprecation and
introduction of new metric.

Tested via:
```
kubectl get --raw /metrics | grep -i apiservice | grep -v APIServiceRegistrationController | grep -v aggregator_unavailable_apiservice | python3 -c "import sys; print('\n'.join([k+' ' + v for k,v in {a.split('{')[0]:'{'+a.split('{')[1].split('}')[0]+'}' for a in sys.stdin.readlines() if '#' not in a and '_sum' not in a and '_bucket' not in a}.items()]))"
```

Before:
```
apiserver_cache_list_fetched_objects_total {index="",resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_cache_list_returned_objects_total {resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_cache_list_total {index="",resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_longrunning_requests {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_request_body_size_bytes_count {resource="apiservices.apiregistration.k8s.io",verb="create"}
apiserver_request_duration_seconds_count {component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_sli_duration_seconds_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_total {code="201",component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_response_sizes_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="LIST",version="v1"}
apiserver_selfrequest_total {resource="apiservices",subresource="",verb="POST"}
apiserver_storage_events_received_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_evaluated_objects_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_fetched_objects_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_returned_objects_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_objects {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_dispatched_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_received_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_initializations_total {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_resource_version {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_events_sizes_count {group="apiregistration.k8s.io",kind="APIService",version="v1"}
apiserver_watch_events_total {group="apiregistration.k8s.io",kind="APIService",version="v1"}
etcd_request_duration_seconds_count {operation="listWithCount",type="/registry/apiregistration.k8s.io/apiservices/"}
etcd_requests_total {operation="listWithCount",type="/registry/apiregistration.k8s.io/apiservices/"}
watch_cache_capacity {resource="apiservices.apiregistration.k8s.io"}
```

After:
```
apiserver_cache_list_fetched_objects_total {group="apiregistration.k8s.io",index="",resource="apiservices"}
apiserver_cache_list_returned_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_cache_list_total {group="apiregistration.k8s.io",index="",resource="apiservices"}
apiserver_longrunning_requests {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_request_body_size_bytes_count {group="apiregistration.k8s.io",resource="apiservices",verb="create"}
apiserver_request_duration_seconds_count {component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_sli_duration_seconds_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_total {code="201",component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_response_sizes_count {component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_selfrequest_total {group="apiregistration.k8s.io",resource="apiservices",subresource="",verb="WATCH"}
apiserver_storage_events_received_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_evaluated_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_fetched_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_returned_objects_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_list_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_storage_objects {resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_dispatched_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_cache_events_received_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_cache_initializations_total {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_cache_resource_version {group="apiregistration.k8s.io",resource="apiservices"}
apiserver_watch_events_sizes_count {group="apiregistration.k8s.io",resource="apiservices",version="v1"}
apiserver_watch_events_total {group="apiregistration.k8s.io",resource="apiservices",version="v1"}
etcd_bookmark_counts {group="apiregistration.k8s.io",resource="apiservices"}
etcd_request_duration_seconds_count {group="apiregistration.k8s.io",operation="listWithCount",resource="apiservices"}
etcd_requests_total {group="apiregistration.k8s.io",operation="listWithCount",resource="apiservices"}
watch_cache_capacity {group="apiregistration.k8s.io",resource="apiservices"}
```

Kubernetes-commit: f712b01ddb55f6569b930ca714499051ba8cb311
2025-05-19 11:41:09 +02:00
Marek Siarkowicz 6764e31cac Separate getList and watch errors to prevent TestGetCacheBypass flakes
The TestGetCacheBypass requires watch cache to be initialized after
which it will want backend to return errors to check what requests are
cached and which are not.

The watch cache is marked as ready after a list succeeds, but before a watch request is executed.
If watch request fail it will immidietly flip back to unready.

Injecting error after watch cache was ready didn't guarantee that watch
request started, sometimes causing watch to be called after and fail.
This immidietly caused the watch cache to be again unready for the rest
of the test.

The fix is just to separate failure injection for List and Watch
responses.

Kubernetes-commit: 70e05132645d5c49cf0ac61a3ae8869dc60f0bde
2025-05-14 15:52:06 +02:00
Jordan Liggitt 067a2139bb Set non-experimental WatchProgressNotifyInterval config field
Kubernetes-commit: 6b0ebedccc19f3fbf877487cffa34c3694d0c275
2025-03-27 11:54:39 -04:00
Karl Isenberg 46dd96ca03 refactor: Stop using ioutil in apiserver
- The ioutl package is deprecated. Migrate to os package functions.

Kubernetes-commit: f93e4645c18c6f56bfddc158ef7b3f674b3c41dd
2025-05-08 11:28:52 -07:00
Ali Abbasi Alaei b6cfd80a0d pkg/storage/cacher/cacher_whitebox_test: deflake TestWatchNotHangingOnStartupFailure when ResilientWatchCacheInitialization is on
Kubernetes-commit: 2073ba2372b2cd1c53b327a50656a2cdd13decba
2025-05-01 16:55:49 -04:00
Wojciech Tyczyński fc69168d19 Fix etcd3 watcher flake
Kubernetes-commit: c3bb59d078a023153c6ea0c9a93e535b48f2557d
2025-04-24 11:32:49 +02:00
Wojciech Tyczyński 4f42467c04 Simplify etcd3 watcher
Kubernetes-commit: 6d6abaab7d1b8f288ed949a115cc769d83bf0fe2
2025-03-24 14:02:07 +01:00
Wojciech Tyczyński b9e86eb851 Fix race for sending errors in watch
Kubernetes-commit: c8c2844aaf1d04835624ff2d46417492e10dec11
2025-03-24 14:02:07 +01:00
Jordan Liggitt 1776f0c3f2 Parallelize cacher list tests
Kubernetes-commit: eca90dab3f553c5794e780c67e17ae75d9acb65b
2025-03-31 15:17:04 -04:00
Marek Siarkowicz e228aeaf39 Don't start etcd for skipped test
Kubernetes-commit: 1a15d582ae5fc84177f305d93ff473cca5de6f93
2025-03-31 21:06:54 +02:00
Marek Siarkowicz 29a5d82129 Stop cacher in TestWatchStreamSeparation to speed up shutdown
Kubernetes-commit: 75186095c58630fde0b3f89892c69c8ef91fffab
2025-03-31 21:28:21 +02:00
Marek Siarkowicz 30b60eb0a6 Fix flake, non-consistent list doesn't give any guarantees about staleness
Kubernetes-commit: 38d5cb368dd95b7f185dbba97fc3e193f48a83f2
2025-03-31 08:10:42 +02:00
Marek Siarkowicz aed144f141 Ensure that cacher is terminated in TestGetListRecursivePrefix
Kubernetes-commit: 9b5c4504ee49c366be3d7c806482ef4665dc5f70
2025-03-31 18:17:08 +02:00
Marek Siarkowicz 6811fdeb2d Unify should delegate list
Kubernetes-commit: 8fdd6fe4768d81da09f339c1dda831a05bb26f00
2025-03-20 11:08:37 +01:00
Marek Siarkowicz 5a059075db Serve LISTs with exact RV and continuations from cache
Kubernetes-commit: f82c9e56d928d1028d4b298578f275a2e5e69490
2024-07-03 21:36:51 +02:00
Marek Siarkowicz fa558b198a Create Snapshotter interface to fake the implementation
Kubernetes-commit: 3edeb60c089106229b582a8c6800388de433426a
2025-03-19 14:04:47 +01:00
Marek Siarkowicz 91724cfd41 Change precedence order for continue and legacy exact match
This doesn't matter for shouldDelegateList, but matters when picking
source of RV. RV from continue takes precedence.

Kubernetes-commit: 7da942ca7486310893d4f11f3af062957f953555
2025-03-19 13:58:43 +01:00
Marek Siarkowicz edd1d1f6ba Extend shouldDelegateList testing incorportating state of cacher
Kubernetes-commit: 929a9c0cad9b5c2c411a325fd4f356a2b5e01f13
2025-03-19 12:23:08 +01:00
Marek Siarkowicz 50f48ce4b3 Implement watchcache returning error from etcd that caused cache reinitialization
Kubernetes-commit: c09d87f79c90a5ebb0ef6a99abd13dec82b497e3
2025-03-18 16:48:33 +01:00
Marek Siarkowicz c59961a007 Test continue with negative RV for reading consistent RV
Kubernetes-commit: 6d21d8f2376b6e25064b10148b4f416e8d89a508
2025-03-17 12:10:54 +01:00
Marek Siarkowicz 4da0062093 Fix missing recursive in consistency check, enable panic on failed check and fix typo in logs
Kubernetes-commit: 4f1912abf2d139c57036c5333f62bd5bddd289fe
2025-03-17 19:24:16 +01:00
Marek Siarkowicz cc5ef43352 Extract delegator.Helper interface to allow making delegate decision based on cache state
Kubernetes-commit: 984b475e74904dd61c10b23472798a21496edc8f
2025-03-17 15:46:02 +01:00
Marek Siarkowicz 7a33f524c6 Use ValidateListOptions in watch cache
Kubernetes-commit: 9e7c080b863896ffbe9eff2a7edc63aa72ec30cf
2025-03-10 14:29:24 +01:00
Marek Siarkowicz 6f6da8e97b Add test cases for negative resource version in TestList
Kubernetes-commit: c4d77a07993302057441a886125c1c887e7869f1
2025-03-14 12:22:17 +01:00
Marek Siarkowicz a67992576e Test bypass for negative RV
Kubernetes-commit: 58d9b5c7b6467bb2cb14d49247cdada02e8f4a83
2025-03-14 14:34:43 +01:00
Marek Siarkowicz 3a2e1b53e5 Simplify bypass test by just testing shouldDelegateList function
Kubernetes-commit: d263344a9b400890409166c5823598a92bbb79fb
2025-03-14 14:26:44 +01:00
Marek Siarkowicz 6ace22f694 Fix flaky RunTestConsistentList
Noticed that cache might not nesseserly observe the write causing test
to flake. Fixed that changing the logic to require LessOrEqual of
writeRV instead of equal to writeRV. Also added comments explaining
edge cases.

Kubernetes-commit: 86169a7a1e09c120cadafc0213afbf9630f0d8af
2025-03-14 13:45:55 +01:00
Marek Siarkowicz f76f81aa6f Test recursive in TestGetListCacheBypass and separate overrides
Kubernetes-commit: 6d3bff83213b35797b23e89c90bfe939dd30c2cd
2025-03-04 20:07:41 +01:00
Marek Siarkowicz 3cb2448d98 Fix detecting consistent read when watchcache starts handling continue
Kubernetes-commit: 8f83f2446a5e2f11eb751fb56067c663b51cfd12
2025-03-12 18:37:10 +01:00
Marek Siarkowicz 36c1a58d7e Fix typo and pass the environment variable required to enable watchcache consistency checking in GCE tests
Kubernetes-commit: 8b0294daed236dcaf2b2f74ad4a38405118ebbaa
2025-03-13 11:55:23 +01:00
Marek Siarkowicz 6ed423348f Simplify shouldDelegateList
When ResourceVersionMatch is set to NotOlderThan, there is no need to handle continue or resourceVersion="".
The validation in apimachinery will not pass and return:
* "resourceVersionMatch is forbidden when continue is provided"
* "resourceVersionMatch is forbidden unless resourceVersion is provided"

Kubernetes-commit: a0cc02e264ead76dfb0ae75a505e4d2e54219def
2025-03-12 18:20:47 +01:00
Marek Siarkowicz 23e9b2c9d4 Implement consistency checking
Kubernetes-commit: e4d73c56cd055a6e3a23068bd70c424579df40fe
2025-02-27 17:53:06 +01:00
Marek Siarkowicz 73e72d16c5 Unify ListOptions validation between cache and etcd
Kubernetes-commit: ccb607f06b91496d02a3b94253261e03e3280630
2025-03-07 12:26:05 +01:00
Marek Siarkowicz 4305a14262 Separate compactWatchCache from increaseRV
Kubernetes-commit: 15cb82b3b3fa95ba5b914a5a02ac01d42eda892f
2025-02-25 14:44:50 +01:00
Marek Siarkowicz f422062046 Pass storage.ListOptions to WaitUntilFreshAndList
Kubernetes-commit: e6cf9dd1663a9a54fa673873e8e97641f69bcefd
2025-02-19 11:20:04 +01:00
Marek Siarkowicz 79ec45326d Run lists tests in subtests
Kubernetes-commit: 7a84e7630f91014fa3b632d8800e9a943003001c
2025-03-03 20:50:37 +01:00
Marek Siarkowicz 9e523945a2 Remove limit support from btree store
We cannot use limit as it would apply it before filtering, which is done
in cacher. Limit is not currently used, but let's remove it to be save,
until filtering is implemented in store.

Kubernetes-commit: 168c338f7cb44a08f3d1a7e1d0e72cac241e9a29
2025-02-25 19:17:30 +01:00
Marek Siarkowicz 3ad93853da Move GetCurrentResourceVersion to storage.Interface
Kubernetes-commit: fea89f25d1bfd7290bb0165f49a6d288ab5f717a
2025-02-24 19:49:45 +01:00
Marek Siarkowicz a4587170bd Move watch progress to separate package.
Kubernetes-commit: 740db0f01d9bceaa7bd6c0b6c5e5f96fc78c04dd
2025-02-25 13:53:17 +01:00
Marek Siarkowicz a64613cb08 Rename CacheProxy to CacheDelegator
Kubernetes-commit: 4c635ecf8250c7493481b9b8fb88e384de1b41db
2025-02-24 19:07:25 +01:00
Lukasz Szaszkiewicz 3aea7778be storage/cacher/ready: dynamically calculate the retryAfterSeconds
retryAfterSeconds is based on the time elapsed since the state (ready, unready) was last changed.

Kubernetes-commit: 04f0bd4e83bbc0a24b8a924333544be86b252c97
2024-10-14 08:17:14 +02:00
Marek Siarkowicz 09c0e7d0cd Add snapshotting of watch cache behind a feature gate
Kubernetes-commit: 2de2093bcef6544ef17c84116fbd751fc429820a
2025-02-19 17:03:48 +01:00
Marek Siarkowicz d522031d97 Ensure that btree threadedStoreIndexer implements orderedLister
Kubernetes-commit: 99881453838a63730fb34b6bd02775ae08320184
2025-02-19 11:47:26 +01:00
Antoni Zawodny efe7a1b26d Add watch cache capacity upper bound adjusting logic
Kubernetes-commit: 2173a0fafd448e55b7738b88fbbab392125dd975
2025-02-21 15:07:01 +01:00
xin.li afb79f83b3 fix wrong assertion on tests
Signed-off-by: xin.li <xin.li@daocloud.io>

Kubernetes-commit: bc4ae15d77beab23f321bf6547f82c04ba27c3fa
2025-02-22 12:39:01 +08:00
Marek Siarkowicz 476d81d4ba Test continuations and exact revision LISTs
Kubernetes-commit: 034285dc4774f2217ca4df4626ca44b8f5f4f261
2025-02-18 17:32:41 +01:00
Marek Siarkowicz 1e6e6f0a46 Refactor TestList and validate continuations to allow testing pagination and more exact RVs in the future
Kubernetes-commit: 764e13e27aedfd9e304e6014af23f20b5619216b
2025-02-19 17:10:58 +01:00
张可10140699 9da16fa551 Fix non-recursive list should also read RequestWatchProgress feature when consistent list from cache is enabled
Kubernetes-commit: bdf2e2d0646fcb6fa56289d85222e2de0b686244
2025-02-20 08:39:36 +08:00
Lukasz Szaszkiewicz 23e8a6cc2f cacher: decrease the running time of TestConsistentReadFallback
Kubernetes-commit: 601c0e359dc959bbfaf42c5d8a8a0a9a2175db74
2025-01-28 13:31:43 +01:00
Lukasz Szaszkiewicz 8a149c9296 cacher/cacher_whitebox_test: newTestCacherWithoutSyncing allow passing a clock
Kubernetes-commit: cfd6d9a2b4476d4256959dc2ebdd517f20191ea3
2025-01-27 19:51:35 +01:00