Commit Graph

75 Commits

Author SHA1 Message Date
Maciej Borsz f4edb394b0 Implement page size progressing for rare selectors.
Kubernetes-commit: a2ad9f9e4aba6aae6657a3189bdced6dbc8ba4b5
2022-03-07 15:44:59 +00:00
Steve Kuznetsov de7922ba23 pkg/storage/etcd3: refactor "too large" resourceVersions
When tests attempt to validate behavior in the case that a client asks
for a resource version that is "too large" for the underlying storage,
the previous implementation would simply add 1 to the latest revision
seen. This is only appropriate for storage backends that
  a) provide a continuous monotonic logical clock
  b) have no other events occurring while the test runs

For instance, when using a singe etcd backend as a shared fixture for
these tests, adding 1 to a previously-seen revision is not suffcient to
ensure that the resulting revision is "too large". By instead using the
largest possible integer value, we can be certain of this.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: b973cdc57cc6ee57684455cdb76db13a8c82cefa
2022-03-23 12:27:16 -08:00
astraw99 b0232c830b Fix function name of IsExist err
Kubernetes-commit: 79471695c2a16cb7c1fb17adff7765f07e2c9057
2022-03-19 23:06:51 +08:00
Steve Kuznetsov 3d5978d57e storage: etcd: clarify lease manager configurations
When the original commit created the lease manager, this comment was
added to set the default test reuse time to 1s. Even at that time, the
comment claimed it was setting 10s. Instead of using this value, though,
new tests that did not call `testSetup()` started to use the default
configuration for production. This commit clarifies the intent of this
comment, moves it next to the code block that it actually applies to,
and makes use of this test-specific logic everywhere.

x-ref: f230b000db

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 6aa37eb06247fb95a6a4ef61cbd50885e52055a0
2022-02-18 08:10:57 -08:00
Steve Kuznetsov 1501fbd930 storage: etcd: use cmp.Diff for comparisons
This commit simply modernizes the comparisons made in the storage tests
to use `cmp.Diff()` so that pointer comparisons and length checks do not
have to be made by hand. We also get nice diffs in the test output this
way instead of large pasted blobs.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: dfdd486f09321e9105fa747a8d1ac5a9a2a7a94a
2022-02-17 07:55:49 -08:00
Wojciech Tyczyński b028a2ea8d Unify merge GetToList and List storage methods to single GetList
Kubernetes-commit: f3d56369f71f5106eeec8300e77bd69b172733da
2022-02-10 10:15:29 +01:00
Wojciech Tyczyński 9f8f72fea9 Change storage.Interface to utilize recurisve parameter
Kubernetes-commit: 3016b3d8f868a3041c7cb86695de09d7ab27cf3b
2022-02-10 09:58:31 +01:00
Steve Kuznetsov 9eba72652e storage: etcd: TestList: improve RV choices
In this test, the current implementation uses a nebulous "RV 1" for some
queries. The intent of this absolute choice is to probe etcd at a
version before any writes ocurred for the test. The particular test
fixture for etcd that is used starts at revision 1, so 1 is used.

This choice is hard to understand the meaning of for readers, though,
and is not valid for any other etcd fixture used for the tests. In order
to improve readability of the test as well as to make it more resilient
to the underlying store, this change updates the test to read the
revision of the underlying storage before making any writes and using
that revision when querying the storage in the tests.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: d2b42b6369ab8db9d0aa0b58dcdf6548ff489d70
2022-02-17 08:36:30 -08:00
Steve Kuznetsov 67dd3e671d storage: etcd: pass contexts correctly in TestConsistentList
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 7272aca69d2602b0bcc97bb5cdc8139ba4687ceb
2022-02-16 13:18:27 -08:00
Steve Kuznetsov af1cb1cefe storage: transformers: pass a context.Context
When an envelope transformer calls out to KMS (for instance), it will be
very helpful to pass a `context.Context` to allow for cancellation. This
patch does that, while passing the previously-expected additional data
via a context value.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 27312feb9983c18d1daf00afba788727d024cdd0
2022-02-17 07:29:44 -08:00
Steve Kuznetsov dbc2925b64 storage: etcd: shorten sub-test names
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 4e166ff74d64042cbeabf72d908ffbf0243d4715
2022-02-16 14:10:43 -08:00
Steve Kuznetsov eaea50a0de storage: etcd: TestConditionalDelete: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 7729691f5c08dd3aa05e3019190653648a52aba2
2022-02-16 13:29:16 -08:00
Steve Kuznetsov ce3e7f36a8 storage: etcd: TestUnconditionalDelete: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: dbb00694f69a1f061d15fc3440a39c8ea360977d
2022-02-16 13:27:52 -08:00
Steve Kuznetsov a1c2b74171 storage: etcd: TestGuaranteedUpdate: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: b6a66252174c59335d82e5b4990b930c8265a69c
2022-02-16 13:26:00 -08:00
Steve Kuznetsov cbb630c2dd storage: etcd: TestGetToList: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: f1ded3b0c34c70c7a993d3c979571d96a29d56bc
2022-02-16 13:23:49 -08:00
Steve Kuznetsov ba213514c9 storage: etcd: remove a duplicate test case
This test case was a duplicate of the previous one.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 921e7525c074750a47818fdf89a4fe5c0b058f0f
2022-02-16 13:13:03 -08:00
Wojciech Tyczyński 46eb50ac00 Cleanup apiserver storage selflink references where possible
Kubernetes-commit: 9b2908ea3b98edbb1a8bcaf4e3e429283b2debfc
2022-01-19 21:13:25 +01:00
Wojciech Tyczyński b4935b065f Slightly improve reliability of storage List tests.
Kubernetes-commit: 8ca6c5922b9a0bafa8c83617d6930b981430334e
2021-12-10 12:05:32 +01:00
Mike Spreitzer 31ec50ec43 Plumb the schema.GroupResource into etcd3 Store struct
Kubernetes-commit: 6f8019aae95db85552381f7e7066a52fef61be7a
2021-09-13 16:54:34 -04:00
Jordan Liggitt 29227fd4ee Avoid spurious calls to update/delete validation
Kubernetes-commit: 561ce035b5b7d577ddb5bb30657bc1f8b399e109
2021-08-05 17:21:40 -04:00
Jordan Liggitt 4bb06f3c4d Simplify running embedded etcd server in tests
Kubernetes-commit: 01760927b82cf28bc6ba5b6364fda3c9f8c5c0b1
2021-06-15 12:07:46 -04:00
Jordan Liggitt 8492f8d104 Drop use of deprecated clientv3.SetLogger
Kubernetes-commit: 52b629efbc8be30901192880b8005e7362f9a69f
2021-05-19 09:10:05 -04:00
Jordan Liggitt 6365bf9124 Switch to go.etcd.io/etcd/client/v3
Kubernetes-commit: 2979c3325e3ac4023dd19c435f3a995074328751
2021-03-23 11:01:52 -04:00
Ling Samuel 59be6d306d api-server add --lease-max-object-count
Signed-off-by: Ling Samuel <lingsamuelgrace@gmail.com>

Kubernetes-commit: c8db72c38c34a965800bd9d570c495c45f142186
2020-12-07 10:45:44 +08:00
Wenjia 15c145e78b Remove unused expectedRV in store_test TestGet
expectedRV is defined in tests struct  but not set in test cases, removing the unnecessary checking

Kubernetes-commit: f8f36c672776bb00b2d53c5d49e92b1bfb608516
2021-01-21 11:27:57 -08:00
pacoxu b74ce63ea5 fix static check for vendor/k8s.io/apiserver/pkg/storage/etcd3/
Kubernetes-commit: aaa7084a4a548795c987b29dffb48749531b60a4
2021-01-07 11:21:50 +08:00
Ling Samuel bd7da186cc apiserver add metric etcd_lease_object_counts
Signed-off-by: Ling Samuel <lingsamuelgrace@gmail.com>

Kubernetes-commit: 7e9fe39cd7c3ee67fcc29e6ca0836917c5a4e9e1
2020-12-07 10:45:44 +08:00
wojtekt ddcc69d4f7 Use current state from watchcache to avoid etcd get for deletions
Kubernetes-commit: 7bab6a9c6ed7d4e54da3e010a7bf0a30786d28f9
2020-11-03 17:30:26 +01:00
wojtekt 8c25869e43 Add suggestion to storage interface Delete method
Kubernetes-commit: c2d61896f45967cdea15072b7a12faeb635b5dc5
2020-11-03 13:21:00 +01:00
Ling Samuel f673ce24f7 apiserver add --lease-reuse-duration-seconds to config lease reuse duration
Signed-off-by: Ling Samuel <lingsamuelgrace@gmail.com>

Kubernetes-commit: c99567005db15d220f2b792e98af735c5aa24f6e
2020-12-02 17:35:10 +08:00
wojtekt 942f220e85 Remove variadic argument from storage interface
Kubernetes-commit: 8b98305858b107369f2c9b9fd8ef1c5b0da078c0
2020-11-02 15:52:51 +01:00
wojtekt e40efde837 Implement etcd3 progress-notify feature in etcd3 layer
Kubernetes-commit: 56e72841b6005740453828a9f4f7a9a1b9a831f6
2020-08-31 12:40:41 +02:00
wojtekt a5000473c1 Pipe newFunc to etcd3 storage layer
Kubernetes-commit: fbd65a265a47ffe081aaac2f794a55034333d11d
2020-08-31 11:58:45 +02:00
Abu Kashem bd49a32a3f count of etcd object should be limited to the specified resource
Currently count includes keys from different resource(s) if the keys
are a prefix of the specified resource/key.

Consider the following keys:
A: <storage-prefix>//foo.bar.io/machines
B: <storage-prefix>//foo.bar.io/machinesets
If we ask for the count of key A, the result will also include the
keys from key B since key B shares the same prefix as key A.

Append a separator to mark the end of the key, this will exclude all
other keys from a different resource that is a prefix of the specified
key.

Kubernetes-commit: 7e445867aa4d37a67591faf6e5508abaea69d216
2020-09-11 17:07:09 -04:00
wojtekt 6e6045833c Fix bug for inconsistent lists served from etcd
Kubernetes-commit: a5b60c352d4457d96a90ddf1479f0833ef36f7d4
2020-08-14 14:51:56 +02:00
Joe Betz ee219411ed Add ResourceVersionMatch parameter to make Resource Version semantics consistent for list
Kubernetes-commit: e214f2408b59c745c199645547948a8ad2a87ac2
2020-05-29 10:44:26 -07:00
Joe Betz 5b75cf0ff9 Add GetOptions and ListOptions to storage interface
Kubernetes-commit: 4c99949ae6b4a74e4ffda0caba86c151c23d579e
2020-05-29 10:48:33 -07:00
Monis Khan f5c1e085ed storage: confirm that paging and predicate filtering work together
This change adds the TestListContinuationWithFilter test which
confirms that paging with a predicate that does not match everything
results in the correct amount of calls to TransformFromStorage and
KV.Get.  The partial result of each paging call is also asserted.

Signed-off-by: Monis Khan <mok@vmware.com>

Kubernetes-commit: 002c75442d768d2bcc51047667354ff16bbfa2e8
2020-02-28 15:35:22 -05:00
Wenjia Zhang e518fe04f6 Resolve uncompatibility from update: etcd CAFile -> TrustedCAFIle
Kubernetes-commit: 9ead9373f350c7ad438257a9e8b8977a67b900eb
2019-10-23 11:15:43 -07:00
Wenjia Zhang 41da2ced4c Replace github.com/coreos/etcd by go.etcd.io/etcd
Kubernetes-commit: 3b274fad2a719dc1fd0eaef6c55a0d344db10242
2019-10-15 22:38:43 -07:00
Joe Betz 1b75e5416d Use same 'minimum resource version' semantics both when watch cache is enabled and disabled
Kubernetes-commit: 1406f7adf20a9ceaf5b8270ec9a61dd9e450ae8a
2018-12-04 16:55:59 -08:00
wojtekt 6d99094b88 Move APIObjectVersioner
Kubernetes-commit: fd819f8fdc2528249efc94c18da434b61fe87925
2019-06-28 13:58:08 +02:00
wojtekt 01c4fe2028 Cleanup util file
Kubernetes-commit: 762518fd47bb058d11b26c60d9cf9f9a9c5b3c05
2019-06-28 13:30:43 +02:00
Chao Xu d403146da2 Protect remainingItemCount behind a feature flag. Also updating the API doc
Kubernetes-commit: 1ad5cb5bb16c83bc7f0a3b78ad196a665a45996d
2019-05-31 15:32:26 -07:00
Jordan Liggitt 2c3ab14d64 Set expected in-memory version when decoding unstructured objects from etcd
Kubernetes-commit: 932553a08c5142fc7751555697e28bdc1f2eff09
2019-06-04 22:12:34 -04:00
Jordan Liggitt 00e2435130 Make GetRemainingItemCount/SetRemainingItemCount use pointers
Kubernetes-commit: 145e8c46cf81ff16b65ff729c392ced262984847
2019-05-27 12:03:26 -04:00
Chao Xu bba40a550f Make ListMeta.RemainingItemCount a pointer (*int64) to make sure it's omitted
when serialized to proto.

The SetRemainingItemCount() and GetRemainingItemCount() still takes and
returns an int64 to make developers life easier.

Kubernetes-commit: e28a1072d94d947f38db7abc4c66426b8f057b17
2019-05-24 17:35:00 -07:00
Chao Xu 5c9333e4d3 Run deleteValidation at the storage layer so that it will be retried on
conflict.

Adding unit test verify that deleteValidation is retried.

adding e2e test verifying the webhook can intercept configmap and custom
resource deletion, and the existing object is sent via the
admissionreview.OldObject.

update the admission integration test to verify that the existing object
is passed to the deletion admission webhook as oldObject, in case of an
immediate deletion and in case of an update-on-delete.

Kubernetes-commit: 7bb4a3bace048cb9cd93d0221a7bf7c4accbf6be
2019-04-09 13:49:16 -07:00
Chao Xu 034ed25bbe Add RemainingItemCount to ListMeta
Kubernetes-commit: 58c18309a84f9e0fe05b92c202616e4ecf8062f7
2019-04-01 14:44:19 -07:00
Jordan Liggitt 80029a760c Deprecate and remove use of alpha metadata.initializers field, remove IncludeUninitialized options
Kubernetes-commit: 17aa60686ebe5fd04b4fe6f442dc36a8d70b6730
2019-01-17 09:19:28 -05:00