This commit allows injecting errors for the
Watch() method of the dummy storage impl.
As a consequence of this, a race is introduced
between when the injected error is written and
read whenever a Watch() is invoked using the
dummy storage. This commit adds locking in order
to mitigate this.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: 2593671337ad449f51b9dcc0b63aa190dd07ab68
Each of these scripts is basically identical, and all were too brittle.
Now they should be more resilient and easier to manage. The script
still needs to be updated if we add new ones, which I do not love.
More cleanup to follow.
Kubernetes-commit: e0ecccff3f5148cc167117ac73233b4edc1640d8
This change updates the API server code to load the encryption
config once at start up instead of multiple times. Previously the
code would set up the storage transformers and the etcd healthz
checks in separate parse steps. This is problematic for KMS v2 key
ID based staleness checks which need to be able to assert that the
API server has a single view into the KMS plugin's current key ID.
Signed-off-by: Monis Khan <mok@microsoft.com>
Kubernetes-commit: f507bc255382b2e2095351053bc17e74f7100d35
- Moves kms proto apis to the staging repo
- Updates generate and verify kms proto scripts to check staging repo
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
Kubernetes-commit: c3794e2377016b1c18b1dcb63dc61d686c8ebcbf
return the last request error, instead of last error received
The rate limit allows 1 event per healthcheck timeout / 2
Kubernetes-commit: 510a85c53a5138babb1650fadd328e6f34baa03b
Use GroupResource instead of object reflection when recording the
following metrics:
- etcd_request_duration_seconds
- etcd_bookmark_counts
Add GroupResource to logs and traces where only reflection-based typing
was previously used.
Both of these changes allow us to disginguish between different CRDs,
all of which are represented as *unstructured.Unstructured.
Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>
Kubernetes-commit: 305fa2add60ad507417304d865f001006d5175fe
Use the group resource instead of objectType in watch cache metrics,
because all CustomResources are grouped together as
*unstructured.Unstructured, instead of 1 entry per type.
Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>
Kubernetes-commit: d08b69e8d35a5aa73a178c508f9b0e1ad74b882d
All CustomResources are treated as *unstructured.Unstructured, leading
the watch cache to log anything related to CRs as Unstructured. This
change uses the schema.GroupResource instead of object type for all type
related log messages in the watch cache, resulting in distinct output
for each CR type.
Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>
Kubernetes-commit: 397533a4c2df9639ff4422c907d06fae195a1835
- add feature gate
- add encrypted object and run generated_files
- generate protobuf for encrypted object and add unit tests
- move parse endpoint to util and refactor
- refactor interface and remove unused interceptor
- add protobuf generate to update-generated-kms.sh
- add integration tests
- add defaulting for apiVersion in kmsConfiguration
- handle v1/v2 and default in encryption config parsing
- move metrics to own pkg and reuse for v2
- use Marshal and Unmarshal instead of serializer
- add context for all service methods
- check version and keyid for healthz
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
Kubernetes-commit: f19f3f409938ff9ac8a61966e47fbe9c6075ec90
Replicated from https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/logger.go#L47
The logic of this function doesn't make a lot of sense to me, but
copying it will avoid any behaviour change.
Signed-off-by: Nic Cope <nicc@rk0n.org>
Kubernetes-commit: c1aa7a0fe73cbcab8e70f7b73a845ae9394f9a71
Currently the API server creates one etcd client per CRD. If clients
aren't provided a logger they'll each create their own. These loggers
can account for ~20% of API server memory consumption on a cluster with
hundreds of CRDs.
Signed-off-by: Nic Cope <nicc@rk0n.org>
Kubernetes-commit: 0c81eabb853e581abbcb37ebf094af3316e1012e
This logger is responsible for 20% of the API server's memory usage when
many CRDs are installed. See the below issue for more context.
https://github.com/kubernetes/kubernetes/issues/111476
Signed-off-by: Nic Cope <nicc@rk0n.org>
Kubernetes-commit: 0e5401c93940126beac45264aa056507b0950075
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: a9593d634c6a053848413e600dadbf974627515f
v1.43.0 marked grpc.WithInsecure() deprecated so this commit moves to use
what is the recommended replacement:
grpc.WithTransportCredentials(insecure.NewCredentials())
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Kubernetes-commit: 2c8bfad9106039aa15233b5bf7282b25a7b7e0a0
expiredBookmarkWatchers allows us to schedule the next bookmark event after dispatching not before as it was previously.
It opens a new functionality in which a watcher might decide to change when the next bookmark should be delivered based on some internal state.
Kubernetes-commit: 0576f6a011cba8f0c8550fd3dd31111376c9dcd0
Using a Pod type in a GetList() call in a test
can panic at worst and error out at best. Here,
neither happened because the error condition
being tested for (cacher being stopped or not)
gets returned before the list pointer can be
enforced.
This commit changes the above to use PodList.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: 487761f4e2543114db158f0d59e598dedc481882
Our tests are mostly error based and explicit error typing allows
us to test against error types directly. Having made this change also
makes it obvious that our test coverage was lacking in two branches,
specifically, we were previously not testing empty start keys nor were
we testing for invalid start RVs.
Kubernetes-commit: 213e380a2e48830db6c71d2da5485d4226d95625
The means by which we encode and decode the continue token during a
paginated LIST call is not specific to etcd3. In order to allow for a
generic suite of tests against any storage.Interface implementation, we
need this logic to live outside of the etcd3 package, or import cycles
will exist.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: eb3aa5be10393968d8083c79f5958501fc029e8d
This test resource only exists to test the versioning logic, but it
cannot live in the general testing package, or import cycles will exist.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: fa2a6d633984aaa36ef988ecf4410c20e27897ea
The means by which we extract and parse the version of an API object is
not specific to etcd3. In order to allow for a generic suite of tests
against any storage.Interface imlpementation, we need this logic to live
outside of the etcd3 package, or import cycles will exist.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 3939f3003e9605c06f65e64d1fc6f94b294f9d97
The cacher blocks requests until it is ready, however, the
ready variable doesn't differentiate if the cacher was stopped.
The cacher is using a condition variable based on sync.Cond to
handle the readiness, however, this was not taking into account
if it was not ready because it was waiting to be ready or it was
stopped.
Add a new condition to the condition variable to handle the
stop condition, and returning an error to signal the goroutines
that they should stop waiting and bail out.
Kubernetes-commit: 2cb3a56e83ae33464edb174b1b6373ba50600759
There's no reason to create the watch using the underlying watcher.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 774870611c1d2b405d67eb73ade628ced5e0e994
Different callers to this test may need to do different backend-specific
validation on the stored data, so we allow them a callback for this.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 6d25e96cedaad249fe75aac4b1fe08bb69829a61
There is no functional difference between checking for an empty key
using the database client and doing so with the storage interface. Using
the latter allows this test to be more portable.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: f894f8196d9266915424e2cefc9e4eb480ae6f5e
These can't live in `_test.go` files, or others won't be able to re-use
them as you can't import from test files.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 0bfeb728786726cd28b4c64d48874d45a2724165
This commit simply factors the test functionality into functions that
accept `storage.Interface`.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 117f674cab1f598251483811fb769046dc3458a3
It is not possible for the nil-check to ever return anything different
from what the explicit boolean used to, but this is only something that
a reader can come to the conclusion on if they very, very carefuly read
the code. Instead of having this implicit flow that is difficult to
follow, let's keep the boolean.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 809fd64b289add1b378b45c748c23b7278c366f1
Previous work by liggitt in 01760927b82 improved the boilerplate
required to run an embedded etcd server for tests as well as set up the
`*etcd3.store{}` for testing. A number of tests were not ported to use the
new helpers, though, either due to custom setup or due to inconsistent
use of setup options. A follow-up by stevekuznetsov in 6aa37eb0624
removed much of the inconsistency, meaning that most callers to
`newStore()` were simply using the default boilerplate and options that
`testSetup()` used.
This patch moves all users to testSetup(), adding options as necessary
to enable some fringe setup use-cases. With a unified setup, new tests
will not copy boilerplate they do not need and it will be immediately
obvious when reading a test if the client or storage setup is *not*
default, improving readability.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 138faa3799341d02df9fc4bedc1371d338c34887
We must ensure that we notice if the etcd behavior on linearized reads
changes.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: ed5fd905f2b42e9919d99c40a1cb25014f0a7f89
In a number of tests, the underlying storage backend interaction will
return the revision (logical clock underpinning the MVCC implementation)
at the call-time of the RPC. Previously, the tests validated that this
returned revision was exactly equal to some previously seen revision.
This assertion is only true in systems where no other events are
advancing the logical clock. For instance, when using a single etcd
cluster as a shared fixture for these tests, the assertion is not valid
any longer. By checking that the returned revision is no older than the
previously seen revision, the validation logic is correct in all cases.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: eba25cdbbcc5d35e707516194f64d8ed363c2773
Previously, this test assumed that:
- a global watch would return only an event for the key in question
- only the delete event in question would be returned
Neither of these assumptions are correct for an etcd backend as long
as any other clients are interacting with the system. This commit
makes the watch more specific and extracts the correct event.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 2631c0a0f959bd67aa455045dce33e77150ab5f8
Some of these changes are cosmetic (repeatedly calling klog.V instead of
reusing the result), others address real issues:
- Logging a message only above a certain verbosity threshold without
recording that verbosity level (if klog.V().Enabled() { klog.Info... }):
this matters when using a logging backend which records the verbosity
level.
- Passing a format string with parameters to a logging function that
doesn't do string formatting.
All of these locations where found by the enhanced logcheck tool from
https://github.com/kubernetes/klog/pull/297.
In some cases it reports false positives, but those can be suppressed with
source code comments.
Kubernetes-commit: edffc700a43e610f641907290a5152ca593bad79
Without these select statements, this test runs until the package-global
timeout and causes a panic. This change makes the test fail faster and
more legibly.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: fc33d0176a5afb81927430d075165152f953c54e
When tests attempt to validate behavior in the case that a client asks
for a resource version that is "too large" for the underlying storage,
the previous implementation would simply add 1 to the latest revision
seen. This is only appropriate for storage backends that
a) provide a continuous monotonic logical clock
b) have no other events occurring while the test runs
For instance, when using a singe etcd backend as a shared fixture for
these tests, adding 1 to a previously-seen revision is not suffcient to
ensure that the resulting revision is "too large". By instead using the
largest possible integer value, we can be certain of this.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: b973cdc57cc6ee57684455cdb76db13a8c82cefa
When the original commit created the lease manager, this comment was
added to set the default test reuse time to 1s. Even at that time, the
comment claimed it was setting 10s. Instead of using this value, though,
new tests that did not call `testSetup()` started to use the default
configuration for production. This commit clarifies the intent of this
comment, moves it next to the code block that it actually applies to,
and makes use of this test-specific logic everywhere.
x-ref: f230b000db
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 6aa37eb06247fb95a6a4ef61cbd50885e52055a0
This commit simply modernizes the comparisons made in the storage tests
to use `cmp.Diff()` so that pointer comparisons and length checks do not
have to be made by hand. We also get nice diffs in the test output this
way instead of large pasted blobs.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: dfdd486f09321e9105fa747a8d1ac5a9a2a7a94a
Modernize the comparisons used in the watch tests to use `cmp.Diff()` for
readability.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: d17a19b39d2dbdaf2cbbaad46de403d6d7ce0602
This was the last test to not use sub-tests, so we can also remove the
indices that the expectation functions take as parameters now.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 9f7bb4264e0b79cbe7979c09f0e4c75a434a27bb
In this test, the current implementation uses a nebulous "RV 1" for some
queries. The intent of this absolute choice is to probe etcd at a
version before any writes ocurred for the test. The particular test
fixture for etcd that is used starts at revision 1, so 1 is used.
This choice is hard to understand the meaning of for readers, though,
and is not valid for any other etcd fixture used for the tests. In order
to improve readability of the test as well as to make it more resilient
to the underlying store, this change updates the test to read the
revision of the underlying storage before making any writes and using
that revision when querying the storage in the tests.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: d2b42b6369ab8db9d0aa0b58dcdf6548ff489d70
This test, as written, is *extremely* cryptic and hard to parse. Add a
comment and stop intentionally ignoring an error that only needs to be
ignored if we're being cryptic.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 50eed81923495f5ee1ac44436676ddbaf2a380fe
When an envelope transformer calls out to KMS (for instance), it will be
very helpful to pass a `context.Context` to allow for cancellation. This
patch does that, while passing the previously-expected additional data
via a context value.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 27312feb9983c18d1daf00afba788727d024cdd0
This test case was a duplicate of the previous one.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Kubernetes-commit: 921e7525c074750a47818fdf89a4fe5c0b058f0f
* Remove linter warnings.
* Cancel contexts to avoid leaks.
* Rename a few XXXThreadUnsafe to XXXLocked to
maintain consistency.
* A few are still called XXXThreadUnsafe mainly
because those are safe to be called from the
perspective that only one gorotuine will access
them - not really called under a lock.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: c3081b48759db1f05a446f2acca7e05c4511ce2e
- Modify GetAllEventsSinceThreadUnsafe to return a watchCacheInterval
- Modify Watch() to compute a watchCacheInterval rather than a slice
of all "initEvents" and pass this interval to process()
- Use interval::Next() to obtain events to process rather than obtain
them all at once
- Modify tests accordingly to use interval
- On invalidation, stop processing and stop the watch.
- Make indexValidator injectable for testing
- Add unit test for verifying the behaviour of stopping the watch.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: 7f2aa7ad3a61a52d0a780f904b291d063399c28a
watchCacheInterval serves as an abstraction over a source
of watchCacheEvents. It maintains a window of events over
an underlying source and these events can be served using
the exposed Next() API. The main intent for doing things
this way is to introduce an upper bound of memory usage
for starting a watch and reduce the maximum possible time
interval for which the lock would be held while events are
copied over.
The source of events for the interval is typically either
the watchCache circular buffer, if events being retrieved
need to be for resource versions > 0 or the underlying
implementation of Store, if resource version = 0.
Furthermore, an interval can be either valid or invalid at
any given point of time. The notion of validity makes sense
only in cases where the window of events in the underlying
source can change over time - i.e. for watchCache circular
buffer. When the circular buffer is full and an event needs
to be popped off, watchCache::startIndex is incremented. In
this case, an interval tracking that popped event is valid
only if it has already been copied to its internal buffer.
However, for efficiency we perform that lazily and we mark
an interval as invalid iff we need to copy events from the
watchCache and we end up needing events that have already
been popped off. This translates to the following condition:
watchCacheInterval::startIndex >= watchCache::startIndex.
When this condition becomes false, the interval is no longer
valid and should not be used to retrieve and serve elements
from the underlying source.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: 347607e97139959f33024a691d0561b1479aeeef
In the following code pattern, the log message will get logged with v=0 in JSON
output although conceptually it has a higher verbosity:
if klog.V(5).Enabled() {
klog.Info("hello world")
}
Having the actual verbosity in the JSON output is relevant, for example for
filtering out only the important info messages. The solution is to use
klog.V(5).Info or something similar.
Whether the outer if is necessary at all depends on how complex the parameters
are. The return value of klog.V can be captured in a variable and be used
multiple times to avoid the overhead for that function call and to avoid
repeating the verbosity level.
Kubernetes-commit: 9eaa2dc554e0c3d4485d4c916dfdbc2f517db2e0
Split process() function into processEvents() and process().
This is done in anticipation of GetAllEventsSinceThreadUnsafe()
returning an entity using which events can be constructed and
not the events itself.
Subsequently, this commit also moves updating resource version
for initEvents from Watch() to the processEvents() func.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: aab7cd3d8a66f425022ca5b2a2bd0d3019efe526
This PR enables unaryClientInterceptor in conjunction with Prometheus interceptor.
Previously it was simply overwritten by the Prometheus interceptor.
As a result etcd client didn't attempt to retry certain errors.
The unaryClientInterceptor is important because it knows how to retry all sorts of errors from the etcd cluster. It will make the API server more resilient to failures - end users won't see certain errors.
The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72
Kubernetes-commit: 83171562b0954b2e19eb69943f01a44779cc7a8f
This is a Config specialized for a GroupResource.
It will support generating new resource-specific metrics.
Kubernetes-commit: 85bcd243aa3c8769a5904a1aea44ce704f5e7174
instead of using a goroutine refreshing the budget, obtain
the value from the last time the budget was accessed.
Kubernetes-commit: dd2c38306000eeb1720afc8346165a6caab09259
Madhav Jivrajani