Commit Graph

466 Commits

Author SHA1 Message Date
Wojciech Tyczyński ae4e49a701 Fix deadlock in ready test
Kubernetes-commit: 3a91c76ffc7e3d7c8fec0041f825460f5e77b26e
2023-03-03 14:09:32 +01:00
Antonio Ojea d7aa6b3cbe wait again on pending state
Change-Id: I1ad79253a5a5d56a4d9611125825b1f7ad552be8

Kubernetes-commit: ca549e690955c7bdacfffa2954466d4f45204e51
2023-02-28 08:33:37 +00:00
Antonio Ojea e4baab2feb cacher allow context cancellation if not ready
Replace the sync.Cond variable with a channel so we can use the
context cancellation signal.

Co-authored-by: Wojciech Tyczy<C5><84>ski <wojtekt@google.com>
Change-Id: I2f75313a6337feee440ece4c1e873c32a12560dd

Kubernetes-commit: d4559ad44833022b9946b179b55a9dfbbd3db78c
2023-02-23 23:13:01 +00:00
Antonio Ojea 7e38414a60 apiserver cacher: don't accept requests if stopped
The cacher blocks requests until it is ready, however, the
ready variable doesn't differentiate if the cacher was stopped.

The cacher is using a condition variable based on sync.Cond to
handle the readiness, however, this was not taking into account
if it was not ready because it was waiting to be ready or it was
stopped.

Add a new condition to the condition variable to handle the
stop condition, and returning an error to signal the goroutines
that they should stop waiting and bail out.

Kubernetes-commit: 9d7e87dfe232be9ae5c0683a46d9f7bae5b87969
2022-03-03 12:01:59 +01:00
Jordan Liggitt b3ad9fb4e3 Generate and format files
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh

Replay of a9593d634c6a053848413e600dadbf974627515f

Kubernetes-commit: 15d9d196476d64482189f00f1cf1a2061aea5b35
2022-11-16 11:39:18 -05:00
Tim Allclair 03fe248033 Validate etcd paths
Kubernetes-commit: 39a167e9012076eee63458d30e7c9f8da8eb8095
2022-10-10 18:15:22 -07:00
Nic Cope e158b9091d Copy etcd client debug level logic from upstream
Replicated from https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/logger.go#L47

The logic of this function doesn't make a lot of sense to me, but
copying it will avoid any behaviour change.

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: 15bab875640db982980494b0cdce4c1e37fc12bc
2022-07-29 14:26:31 -07:00
Nic Cope dcc27cc581 Give etcd client logger a name
Logic copied from https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/client.go#L374

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: 364374dde361fd269bad5f57e5d53aae8cf85673
2022-07-29 14:24:17 -07:00
Nic Cope 1679f86577 Share a single etcd3 client logger across all clients
Currently the API server creates one etcd client per CRD. If clients
aren't provided a logger they'll each create their own. These loggers
can account for ~20% of API server memory consumption on a cluster with
hundreds of CRDs.

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: 4ab32877bc0e22dc7918db4976833297605ed023
2022-07-28 19:51:55 -07:00
Nic Cope a2f2f0453d Disable the etcd3 client logger
This logger is responsible for 20% of the API server's memory usage when
many CRDs are installed. See the below issue for more context.

https://github.com/kubernetes/kubernetes/issues/111476

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: c1c18295893282ae32e7be0d09f607680241712f
2022-07-27 14:44:49 -07:00
Abirdcfly d8cb3a961e fix: --chunk-size with selector returns missing result
Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Kubernetes-commit: 0519ec2d2aa27edbdcab545babe80c50d586b433
2022-06-20 00:54:57 +08:00
Anish Ramasekar e442eafb33 feat: prepare KMS data encryption for migration to AES-GCM
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
Co-authored-by: Monis Khan <mok@vmware.com>
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 90b42f91fd904b71fd52ca9ae55a5de73e6b779a
2022-03-16 17:54:10 +00:00
Steve Kuznetsov 42854c1734 pkg/storage/etcd3: be more precise in watch test
Previously, this test assumed that:
 - a global watch would return only an event for the key in question
 - only the delete event in question would be returned

Neither of these assumptions are correct for an etcd backend as long
as any other clients are interacting with the system. This commit
makes the watch more specific and extracts the correct event.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 2631c0a0f959bd67aa455045dce33e77150ab5f8
2022-03-23 12:59:53 -08:00
Abu Kashem 1b651c5994 add latency tracker for storage and transform
Kubernetes-commit: eca90856940e9251ecf3fde95c5e4d2d16f5ad68
2022-02-01 18:13:03 -05:00
Patrick Ohly ba3b8e9322 enhance and fix log calls
Some of these changes are cosmetic (repeatedly calling klog.V instead of
reusing the result), others address real issues:

- Logging a message only above a certain verbosity threshold without
  recording that verbosity level (if klog.V().Enabled() { klog.Info... }):
  this matters when using a logging backend which records the verbosity
  level.

- Passing a format string with parameters to a logging function that
  doesn't do string formatting.

All of these locations where found by the enhanced logcheck tool from
https://github.com/kubernetes/klog/pull/297.

In some cases it reports false positives, but those can be suppressed with
source code comments.

Kubernetes-commit: edffc700a43e610f641907290a5152ca593bad79
2022-02-16 12:17:47 +01:00
Steve Kuznetsov 25c879eec5 pkg/storage/etcd3: update test to time out, not panic
Without these select statements, this test runs until the package-global
timeout and causes a panic. This change makes the test fail faster and
more legibly.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: fc33d0176a5afb81927430d075165152f953c54e
2022-03-23 12:22:35 -08:00
Maciej Borsz f4edb394b0 Implement page size progressing for rare selectors.
Kubernetes-commit: a2ad9f9e4aba6aae6657a3189bdced6dbc8ba4b5
2022-03-07 15:44:59 +00:00
Steve Kuznetsov edb0c0eba4 pkg/storage/etcd3: use correct diff in tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 8903372e8b8d32f8b244f5087e3f2907b86f03bd
2022-03-23 12:33:57 -08:00
Steve Kuznetsov de7922ba23 pkg/storage/etcd3: refactor "too large" resourceVersions
When tests attempt to validate behavior in the case that a client asks
for a resource version that is "too large" for the underlying storage,
the previous implementation would simply add 1 to the latest revision
seen. This is only appropriate for storage backends that
  a) provide a continuous monotonic logical clock
  b) have no other events occurring while the test runs

For instance, when using a singe etcd backend as a shared fixture for
these tests, adding 1 to a previously-seen revision is not suffcient to
ensure that the resulting revision is "too large". By instead using the
largest possible integer value, we can be certain of this.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: b973cdc57cc6ee57684455cdb76db13a8c82cefa
2022-03-23 12:27:16 -08:00
Daniel Smith 7b0bc4f4b3 remove unneeded references
Kubernetes-commit: 2831f9a343ec405efce60d09da482a654971018e
2022-03-17 18:35:00 +00:00
Daniel Smith cbbb5800c3 find and replace
Kubernetes-commit: e405ae9ec8592bf028e351bab44de420908bac0d
2022-03-17 18:50:38 +00:00
astraw99 b0232c830b Fix function name of IsExist err
Kubernetes-commit: 79471695c2a16cb7c1fb17adff7765f07e2c9057
2022-03-19 23:06:51 +08:00
Steve Kuznetsov 3d5978d57e storage: etcd: clarify lease manager configurations
When the original commit created the lease manager, this comment was
added to set the default test reuse time to 1s. Even at that time, the
comment claimed it was setting 10s. Instead of using this value, though,
new tests that did not call `testSetup()` started to use the default
configuration for production. This commit clarifies the intent of this
comment, moves it next to the code block that it actually applies to,
and makes use of this test-specific logic everywhere.

x-ref: f230b000db

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 6aa37eb06247fb95a6a4ef61cbd50885e52055a0
2022-02-18 08:10:57 -08:00
Wojciech Tyczyński 58b4aa562f Fix potential race in dispatching watch event
Kubernetes-commit: 779f157ecfb24d0ee944f18e481bfa8cc8c94f6f
2022-02-22 17:14:15 +01:00
Wojciech Tyczyński 0b8c9f5ebf Avoid perfoming deep copies in watchcache if not needed
Kubernetes-commit: 0cb3a02bbede2f547ddd20e23e2bbc782ac3956c
2022-02-21 15:18:25 +01:00
Steve Kuznetsov 1501fbd930 storage: etcd: use cmp.Diff for comparisons
This commit simply modernizes the comparisons made in the storage tests
to use `cmp.Diff()` so that pointer comparisons and length checks do not
have to be made by hand. We also get nice diffs in the test output this
way instead of large pasted blobs.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: dfdd486f09321e9105fa747a8d1ac5a9a2a7a94a
2022-02-17 07:55:49 -08:00
Steve Kuznetsov fb315d8c17 store: etcd: TestWatch: use cmp.Diff
Modernize the comparisons used in the watch tests to use `cmp.Diff()` for
readability.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: d17a19b39d2dbdaf2cbbaad46de403d6d7ce0602
2022-02-18 06:51:36 -08:00
Steve Kuznetsov 557ecdf10d store: etcd: TestWatch: use sub-tests
This was the last test to not use sub-tests, so we can also remove the
indices that the expectation functions take as parameters now.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 9f7bb4264e0b79cbe7979c09f0e4c75a434a27bb
2022-02-18 06:50:05 -08:00
Wojciech Tyczyński b028a2ea8d Unify merge GetToList and List storage methods to single GetList
Kubernetes-commit: f3d56369f71f5106eeec8300e77bd69b172733da
2022-02-10 10:15:29 +01:00
Wojciech Tyczyński 9f8f72fea9 Change storage.Interface to utilize recurisve parameter
Kubernetes-commit: 3016b3d8f868a3041c7cb86695de09d7ab27cf3b
2022-02-10 09:58:31 +01:00
Steve Kuznetsov 9eba72652e storage: etcd: TestList: improve RV choices
In this test, the current implementation uses a nebulous "RV 1" for some
queries. The intent of this absolute choice is to probe etcd at a
version before any writes ocurred for the test. The particular test
fixture for etcd that is used starts at revision 1, so 1 is used.

This choice is hard to understand the meaning of for readers, though,
and is not valid for any other etcd fixture used for the tests. In order
to improve readability of the test as well as to make it more resilient
to the underlying store, this change updates the test to read the
revision of the underlying storage before making any writes and using
that revision when querying the storage in the tests.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: d2b42b6369ab8db9d0aa0b58dcdf6548ff489d70
2022-02-17 08:36:30 -08:00
Steve Kuznetsov 4074cf3cb4 storage: etcd: TestWatchError: improve readability
This test, as written, is *extremely* cryptic and hard to parse. Add a
comment and stop intentionally ignoring an error that only needs to be
ignored if we're being cryptic.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 50eed81923495f5ee1ac44436676ddbaf2a380fe
2022-02-17 15:46:05 -08:00
Steve Kuznetsov 67dd3e671d storage: etcd: pass contexts correctly in TestConsistentList
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 7272aca69d2602b0bcc97bb5cdc8139ba4687ceb
2022-02-16 13:18:27 -08:00
Steve Kuznetsov af1cb1cefe storage: transformers: pass a context.Context
When an envelope transformer calls out to KMS (for instance), it will be
very helpful to pass a `context.Context` to allow for cancellation. This
patch does that, while passing the previously-expected additional data
via a context value.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 27312feb9983c18d1daf00afba788727d024cdd0
2022-02-17 07:29:44 -08:00
Steve Kuznetsov dbc2925b64 storage: etcd: shorten sub-test names
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 4e166ff74d64042cbeabf72d908ffbf0243d4715
2022-02-16 14:10:43 -08:00
Steve Kuznetsov eaea50a0de storage: etcd: TestConditionalDelete: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 7729691f5c08dd3aa05e3019190653648a52aba2
2022-02-16 13:29:16 -08:00
Steve Kuznetsov ce3e7f36a8 storage: etcd: TestUnconditionalDelete: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: dbb00694f69a1f061d15fc3440a39c8ea360977d
2022-02-16 13:27:52 -08:00
Steve Kuznetsov a1c2b74171 storage: etcd: TestGuaranteedUpdate: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: b6a66252174c59335d82e5b4990b930c8265a69c
2022-02-16 13:26:00 -08:00
Steve Kuznetsov cbb630c2dd storage: etcd: TestGetToList: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: f1ded3b0c34c70c7a993d3c979571d96a29d56bc
2022-02-16 13:23:49 -08:00
Steve Kuznetsov ba213514c9 storage: etcd: remove a duplicate test case
This test case was a duplicate of the previous one.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 921e7525c074750a47818fdf89a4fe5c0b058f0f
2022-02-16 13:13:03 -08:00
Wojciech Tyczyński 46eb50ac00 Cleanup apiserver storage selflink references where possible
Kubernetes-commit: 9b2908ea3b98edbb1a8bcaf4e3e429283b2debfc
2022-01-19 21:13:25 +01:00
Madhav Jivrajani ffc1b941d4 cacher: Minor cleanup and refactor of code and tests
* Remove linter warnings.
* Cancel contexts to avoid leaks.
* Rename a few XXXThreadUnsafe to XXXLocked to
  maintain consistency.
  * A few are still called XXXThreadUnsafe mainly
    because those are safe to be called from the
    perspective that only one gorotuine will access
    them - not really called under a lock.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: c3081b48759db1f05a446f2acca7e05c4511ce2e
2022-02-10 13:44:34 +05:30
Madhav Jivrajani 4f2ee5cf42 cacher: Use watchCacheInterval to reduce lock contention
- Modify GetAllEventsSinceThreadUnsafe to return a watchCacheInterval
- Modify Watch() to compute a watchCacheInterval rather than a slice
  of all "initEvents" and pass this interval to process()
- Use interval::Next() to obtain events to process rather than obtain
  them all at once
- Modify tests accordingly to use interval
- On invalidation, stop processing and stop the watch.
- Make indexValidator injectable for testing
- Add unit test for verifying the behaviour of stopping the watch.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 7f2aa7ad3a61a52d0a780f904b291d063399c28a
2021-10-05 15:52:09 +05:30
Madhav Jivrajani 74fdb4a93b cacher: watchCacheInterval to reduce lock contention
watchCacheInterval serves as an abstraction over a source
of watchCacheEvents. It maintains a window of events over
an underlying source and these events can be served using
the exposed Next() API. The main intent for doing things
this way is to introduce an upper bound of memory usage
for starting a watch and reduce the maximum possible time
interval for which the lock would be held while events are
copied over.

The source of events for the interval is typically either
the watchCache circular buffer, if events being retrieved
need to be for resource versions > 0 or the underlying
implementation of Store, if resource version = 0.

Furthermore, an interval can be either valid or invalid at
any given point of time. The notion of validity makes sense
only in cases where the window of events in the underlying
source can change over time - i.e. for watchCache circular
buffer. When the circular buffer is full and an event needs
to be popped off, watchCache::startIndex is incremented. In
this case, an interval tracking that popped event is valid
only if it has already been copied to its internal buffer.
However, for efficiency we perform that lazily and we mark
an interval as invalid iff we need to copy events from the
watchCache and we end up needing events that have already
been popped off. This translates to the following condition:
  watchCacheInterval::startIndex >= watchCache::startIndex.
When this condition becomes false, the interval is no longer
valid and should not be used to retrieve and serve elements
from the underlying source.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 347607e97139959f33024a691d0561b1479aeeef
2021-12-05 23:17:00 +05:30
Davanum Srinivas c56bc7b872 OWNERS cleanup - Jan 2021 Week 1
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: 9682b7248fb69733c2a0ee53618856e87b067f16
2022-01-03 10:59:47 -05:00
Patrick Ohly ec795ae204 avoid klog Info calls without verbosity
In the following code pattern, the log message will get logged with v=0 in JSON
output although conceptually it has a higher verbosity:

   if klog.V(5).Enabled() {
       klog.Info("hello world")
   }

Having the actual verbosity in the JSON output is relevant, for example for
filtering out only the important info messages. The solution is to use
klog.V(5).Info or something similar.

Whether the outer if is necessary at all depends on how complex the parameters
are. The return value of klog.V can be captured in a variable and be used
multiple times to avoid the overhead for that function call and to avoid
repeating the verbosity level.

Kubernetes-commit: 9eaa2dc554e0c3d4485d4c916dfdbc2f517db2e0
2021-12-11 12:10:21 +01:00
Wojciech Tyczyński 067689be43 Graduate EfficientWatchResumption feature to GA
Kubernetes-commit: b7257a4d6a3f2382f4753322ec6d3458410b04f9
2021-12-10 10:20:51 +01:00
Wojciech Tyczyński b4935b065f Slightly improve reliability of storage List tests.
Kubernetes-commit: 8ca6c5922b9a0bafa8c83617d6930b981430334e
2021-12-10 12:05:32 +01:00
Davanum Srinivas b840d63feb Cleanup OWNERS files (No Activity in the last year)
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: 497e9c1971c9e7d0193bc6d11503ec4ad527f1d5
2021-12-10 15:18:50 -05:00
Wojciech Tyczyński a55c2c834e Unify List and GetToList functions of etcd3
Kubernetes-commit: 27db8e34cfc930972beb8b5a33c443e134bcd614
2021-11-22 13:30:25 +01:00