Commit Graph

113 Commits

Author SHA1 Message Date
Steve Kuznetsov ce3e7f36a8 storage: etcd: TestUnconditionalDelete: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: dbb00694f69a1f061d15fc3440a39c8ea360977d
2022-02-16 13:27:52 -08:00
Steve Kuznetsov a1c2b74171 storage: etcd: TestGuaranteedUpdate: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: b6a66252174c59335d82e5b4990b930c8265a69c
2022-02-16 13:26:00 -08:00
Steve Kuznetsov cbb630c2dd storage: etcd: TestGetToList: use sub-tests
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: f1ded3b0c34c70c7a993d3c979571d96a29d56bc
2022-02-16 13:23:49 -08:00
Steve Kuznetsov ba213514c9 storage: etcd: remove a duplicate test case
This test case was a duplicate of the previous one.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 921e7525c074750a47818fdf89a4fe5c0b058f0f
2022-02-16 13:13:03 -08:00
Wojciech Tyczyński 46eb50ac00 Cleanup apiserver storage selflink references where possible
Kubernetes-commit: 9b2908ea3b98edbb1a8bcaf4e3e429283b2debfc
2022-01-19 21:13:25 +01:00
Wojciech Tyczyński b4935b065f Slightly improve reliability of storage List tests.
Kubernetes-commit: 8ca6c5922b9a0bafa8c83617d6930b981430334e
2021-12-10 12:05:32 +01:00
Mike Spreitzer 31ec50ec43 Plumb the schema.GroupResource into etcd3 Store struct
Kubernetes-commit: 6f8019aae95db85552381f7e7066a52fef61be7a
2021-09-13 16:54:34 -04:00
Jordan Liggitt 29227fd4ee Avoid spurious calls to update/delete validation
Kubernetes-commit: 561ce035b5b7d577ddb5bb30657bc1f8b399e109
2021-08-05 17:21:40 -04:00
Jordan Liggitt 4bb06f3c4d Simplify running embedded etcd server in tests
Kubernetes-commit: 01760927b82cf28bc6ba5b6364fda3c9f8c5c0b1
2021-06-15 12:07:46 -04:00
Jordan Liggitt 8492f8d104 Drop use of deprecated clientv3.SetLogger
Kubernetes-commit: 52b629efbc8be30901192880b8005e7362f9a69f
2021-05-19 09:10:05 -04:00
Jordan Liggitt 6365bf9124 Switch to go.etcd.io/etcd/client/v3
Kubernetes-commit: 2979c3325e3ac4023dd19c435f3a995074328751
2021-03-23 11:01:52 -04:00
Ling Samuel 59be6d306d api-server add --lease-max-object-count
Signed-off-by: Ling Samuel <lingsamuelgrace@gmail.com>

Kubernetes-commit: c8db72c38c34a965800bd9d570c495c45f142186
2020-12-07 10:45:44 +08:00
Wenjia 15c145e78b Remove unused expectedRV in store_test TestGet
expectedRV is defined in tests struct  but not set in test cases, removing the unnecessary checking

Kubernetes-commit: f8f36c672776bb00b2d53c5d49e92b1bfb608516
2021-01-21 11:27:57 -08:00
pacoxu b74ce63ea5 fix static check for vendor/k8s.io/apiserver/pkg/storage/etcd3/
Kubernetes-commit: aaa7084a4a548795c987b29dffb48749531b60a4
2021-01-07 11:21:50 +08:00
Ling Samuel bd7da186cc apiserver add metric etcd_lease_object_counts
Signed-off-by: Ling Samuel <lingsamuelgrace@gmail.com>

Kubernetes-commit: 7e9fe39cd7c3ee67fcc29e6ca0836917c5a4e9e1
2020-12-07 10:45:44 +08:00
wojtekt ddcc69d4f7 Use current state from watchcache to avoid etcd get for deletions
Kubernetes-commit: 7bab6a9c6ed7d4e54da3e010a7bf0a30786d28f9
2020-11-03 17:30:26 +01:00
wojtekt 8c25869e43 Add suggestion to storage interface Delete method
Kubernetes-commit: c2d61896f45967cdea15072b7a12faeb635b5dc5
2020-11-03 13:21:00 +01:00
Ling Samuel f673ce24f7 apiserver add --lease-reuse-duration-seconds to config lease reuse duration
Signed-off-by: Ling Samuel <lingsamuelgrace@gmail.com>

Kubernetes-commit: c99567005db15d220f2b792e98af735c5aa24f6e
2020-12-02 17:35:10 +08:00
wojtekt 942f220e85 Remove variadic argument from storage interface
Kubernetes-commit: 8b98305858b107369f2c9b9fd8ef1c5b0da078c0
2020-11-02 15:52:51 +01:00
wojtekt e40efde837 Implement etcd3 progress-notify feature in etcd3 layer
Kubernetes-commit: 56e72841b6005740453828a9f4f7a9a1b9a831f6
2020-08-31 12:40:41 +02:00
wojtekt a5000473c1 Pipe newFunc to etcd3 storage layer
Kubernetes-commit: fbd65a265a47ffe081aaac2f794a55034333d11d
2020-08-31 11:58:45 +02:00
Abu Kashem bd49a32a3f count of etcd object should be limited to the specified resource
Currently count includes keys from different resource(s) if the keys
are a prefix of the specified resource/key.

Consider the following keys:
A: <storage-prefix>//foo.bar.io/machines
B: <storage-prefix>//foo.bar.io/machinesets
If we ask for the count of key A, the result will also include the
keys from key B since key B shares the same prefix as key A.

Append a separator to mark the end of the key, this will exclude all
other keys from a different resource that is a prefix of the specified
key.

Kubernetes-commit: 7e445867aa4d37a67591faf6e5508abaea69d216
2020-09-11 17:07:09 -04:00
wojtekt 6e6045833c Fix bug for inconsistent lists served from etcd
Kubernetes-commit: a5b60c352d4457d96a90ddf1479f0833ef36f7d4
2020-08-14 14:51:56 +02:00
Joe Betz ee219411ed Add ResourceVersionMatch parameter to make Resource Version semantics consistent for list
Kubernetes-commit: e214f2408b59c745c199645547948a8ad2a87ac2
2020-05-29 10:44:26 -07:00
Joe Betz 5b75cf0ff9 Add GetOptions and ListOptions to storage interface
Kubernetes-commit: 4c99949ae6b4a74e4ffda0caba86c151c23d579e
2020-05-29 10:48:33 -07:00
Monis Khan f5c1e085ed storage: confirm that paging and predicate filtering work together
This change adds the TestListContinuationWithFilter test which
confirms that paging with a predicate that does not match everything
results in the correct amount of calls to TransformFromStorage and
KV.Get.  The partial result of each paging call is also asserted.

Signed-off-by: Monis Khan <mok@vmware.com>

Kubernetes-commit: 002c75442d768d2bcc51047667354ff16bbfa2e8
2020-02-28 15:35:22 -05:00
Wenjia Zhang e518fe04f6 Resolve uncompatibility from update: etcd CAFile -> TrustedCAFIle
Kubernetes-commit: 9ead9373f350c7ad438257a9e8b8977a67b900eb
2019-10-23 11:15:43 -07:00
Wenjia Zhang 41da2ced4c Replace github.com/coreos/etcd by go.etcd.io/etcd
Kubernetes-commit: 3b274fad2a719dc1fd0eaef6c55a0d344db10242
2019-10-15 22:38:43 -07:00
Joe Betz 1b75e5416d Use same 'minimum resource version' semantics both when watch cache is enabled and disabled
Kubernetes-commit: 1406f7adf20a9ceaf5b8270ec9a61dd9e450ae8a
2018-12-04 16:55:59 -08:00
wojtekt 6d99094b88 Move APIObjectVersioner
Kubernetes-commit: fd819f8fdc2528249efc94c18da434b61fe87925
2019-06-28 13:58:08 +02:00
wojtekt 01c4fe2028 Cleanup util file
Kubernetes-commit: 762518fd47bb058d11b26c60d9cf9f9a9c5b3c05
2019-06-28 13:30:43 +02:00
Chao Xu d403146da2 Protect remainingItemCount behind a feature flag. Also updating the API doc
Kubernetes-commit: 1ad5cb5bb16c83bc7f0a3b78ad196a665a45996d
2019-05-31 15:32:26 -07:00
Jordan Liggitt 2c3ab14d64 Set expected in-memory version when decoding unstructured objects from etcd
Kubernetes-commit: 932553a08c5142fc7751555697e28bdc1f2eff09
2019-06-04 22:12:34 -04:00
Jordan Liggitt 00e2435130 Make GetRemainingItemCount/SetRemainingItemCount use pointers
Kubernetes-commit: 145e8c46cf81ff16b65ff729c392ced262984847
2019-05-27 12:03:26 -04:00
Chao Xu bba40a550f Make ListMeta.RemainingItemCount a pointer (*int64) to make sure it's omitted
when serialized to proto.

The SetRemainingItemCount() and GetRemainingItemCount() still takes and
returns an int64 to make developers life easier.

Kubernetes-commit: e28a1072d94d947f38db7abc4c66426b8f057b17
2019-05-24 17:35:00 -07:00
Chao Xu 5c9333e4d3 Run deleteValidation at the storage layer so that it will be retried on
conflict.

Adding unit test verify that deleteValidation is retried.

adding e2e test verifying the webhook can intercept configmap and custom
resource deletion, and the existing object is sent via the
admissionreview.OldObject.

update the admission integration test to verify that the existing object
is passed to the deletion admission webhook as oldObject, in case of an
immediate deletion and in case of an update-on-delete.

Kubernetes-commit: 7bb4a3bace048cb9cd93d0221a7bf7c4accbf6be
2019-04-09 13:49:16 -07:00
Chao Xu 034ed25bbe Add RemainingItemCount to ListMeta
Kubernetes-commit: 58c18309a84f9e0fe05b92c202616e4ecf8062f7
2019-04-01 14:44:19 -07:00
Jordan Liggitt 80029a760c Deprecate and remove use of alpha metadata.initializers field, remove IncludeUninitialized options
Kubernetes-commit: 17aa60686ebe5fd04b4fe6f442dc36a8d70b6730
2019-01-17 09:19:28 -05:00
Jordan Liggitt bd604a62aa Remove deprecated --etcd-quorum-read flag
Kubernetes-commit: cff79c542130831f4a212099974570244a0c9586
2018-10-08 11:04:28 -04:00
Mike Danese f8e0930b86 storage: propagate TransformFromStorage errors from List
Like we do everywhere else we use TranformFromStorage. The current
behavior is causing all service account tokens to be regenerated,
invalidating old service account tokens and unrecoverably breaking apps
that are using InClusterConfig or exported service account tokens.

If we are going to break stuff, let's just break the Lists so that
misconfiguration of encryption config or checkpoint corruption are
obvious.

Kubernetes-commit: e7bda4431da05b55b4e8f66ed308d4ed90efd2df
2018-10-03 13:30:29 -07:00
Chao Xu cc84cfddc9 support continueToken for inconsistent list
Kubernetes-commit: 0a7286c6b21a858f7397a0835776cb5900d98e87
2018-08-10 10:27:40 -07:00
David Eads cb5cac48ee make package name match all the import aliases
Kubernetes-commit: d3bd0eb1d5cefc25e4476d8dc086ebd90439ef4e
2018-08-01 10:01:32 -04:00
Mikhail Mazurskiy 0f7bbcadfb Add missing error handling in schema-related code
Kubernetes-commit: bfe313d5f351dfae086a85a97e7103183173e5b5
2018-06-03 14:59:58 +10:00
Cong Ding f230b000db etcd: reuse leases for keys in a time window
Reuse leases for keys in a time window, to reduce the overhead to etcd
caused by using massive number of leases

Fixes #47532

Kubernetes-commit: 163529bc202054d991f0ce2e21738cc18ffd6022
2018-05-30 17:27:00 -07:00
wojtekt 5d15c5549a Fix incorrectly set resource version in List
Kubernetes-commit: a3578c864ed7db5a73daa88839eed11d86af209b
2018-05-22 15:32:25 +02:00
hzxuzhonghu 6ab99203b7 Replace "golang.org/x/net/context" with "context"
Kubernetes-commit: 70e45eccf27726f0e63dd1024924ccc7e2cd35a0
2018-02-28 12:20:22 +08:00
halfcrazy 6f8c3a80da fix typo in package apiserver
Kubernetes-commit: 0da91a8577ddfdeaff985cbb6c0da69d5a2ffc81
2018-02-01 03:04:33 +08:00
YuxiJin-tobeyjin 57076f11ae Should use Fatalf while need to format the output
Kubernetes-commit: 10751e54e6f01ed771a8fb75abd90346ee4ab501
2017-10-25 16:20:32 +08:00
Jordan Liggitt 8e603ae547 Recheck if transformed data is stale when doing live lookup during update
Kubernetes-commit: 070089c6bfccc18ba6039eee45b982633087c87e
2018-01-17 01:21:52 -05:00
Clayton Coleman 5f039fac0c Avoid intermediate List allocations as items added to the list
Pick a reasonable middle ground between allocating larger chunks of
memory (2048 * ~500b for pod slices) and having many small allocations
as the list is resized by preallocating capacity based on the expected
list size. At worst, we'll allocate a 1M slice for pods and only add
a single pod to it (if the selector is very specific).

Kubernetes-commit: ce0dc76901bd1ce36ca20c5cf96b89088d0e95a2
2017-10-09 22:16:13 -04:00
Clayton Coleman f24913f06e Promote continuation token schema to v1
Change the filtering logic to require a leading path and clean that
afterwards.

Kubernetes-commit: ac8808b792f624fa04aa7c589bd5aca1b9afde39
2017-09-24 18:09:54 -04:00
Clayton Coleman 2f9b480e69 Fill partial pages on the server rather than forcing client to
The etcd3 storage now attempts to fill partial pages to prevent clients
having to make more round trips (latency from server to etcd is lower
than client to server). The server makes repeated requests to etcd of
the current page size, then uses the filter function to eliminate any
matches. After this change the apiserver will always return full pages,
but we leave the language in place that clients must tolerate it.

Reduces tail latency of large filtered lists, such as viewing pods
assigned to a node.

Kubernetes-commit: da7124e5e5c0385dd5bcfc72ef035effc7708913
2017-09-24 18:06:57 -04:00
Kubernetes Publisher 0f62a50c16 etcd3 store: retry w/live object on conflict
In GuaranteedUpdate, if it was called with a suggestion (e.g. via the
watch cache), and the suggested object is stale, perform a live lookup
and then retry the update.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>

Kubernetes-commit: bf33df16b52508974ddedacd814010cfe0fb79f0
2017-09-22 11:42:06 +00:00
Clayton Coleman 460257fd61 Server side implementation of paging for etcd3
Add a feature gate in the apiserver to control whether paging can be
used. Add controls to the storage factory that allow it to be disabled
per resource. Use a JSON encoded continuation token that can be
versioned. Create a 410 error if the continuation token is expired.

Adds GetContinue() to ListMeta.

Kubernetes-commit: 8952a0cb722b77459cf2701632a30f5b264f5aba
2017-09-03 14:04:12 +00:00
Clayton Coleman a9bfd91dd9 Do not persist SelfLink into etcd storage
This behavior regressed in an earlier release. Clearing the self link
ensures that a new version is always written and reduces the size of the
stored object by a small amount. Add tests to verify that Create and
Update result in no SelfLink stored in etcd.

Kubernetes-commit: 461c3701f0915acbf49c339f5321fa86879a963e
2017-07-16 04:08:42 +00:00
Clayton Coleman 2f829d739b GuaranteedUpdate must write if stored data is not canonical
An optimization added to the GuaranteedUpdate loop changed the
comparison of the current objects serialization against the stored data,
instead comparing to the in memory object, which defeated the mechanism
we use to migrate stored data.

This commit preserves that optimization but correctly verifies the in
memory serialization against the on disk serialization by fetching the
latest serialized data. Since most updates are not no-ops, this should
not regress the performance of the normal path.

Kubernetes-commit: b851614adfe2b39941d518485480ff527fa4f0c1
2017-07-04 08:39:44 +00:00
Clayton Coleman 5fa08b8c5e Allow initialization of resources
Add support for creating resources that are not immediately visible to
naive clients, but must first be initialized by one or more privileged
cluster agents. These controllers can mark the object as initialized,
allowing others to see them.

Permission to override initialization defaults or modify an initializing
object is limited per resource to a virtual subresource "RESOURCE/initialize"
via RBAC.

Initialization is currently alpha.

Kubernetes-commit: 331eea67d8000e5c4b37e2234a90903c15881c2f
2017-06-13 20:47:30 +00:00
Clayton Coleman f27996225a Update etcd3 storage to leverage storage/value interfaces
Adds context argument which must be set for AES GCM authenticated data
to be passed.

Kubernetes-commit: a73990a33f95713f026ee7ae9ae6741255aaf8e4
2017-05-17 17:27:53 +00:00
Cao Shufeng 1e6400df1f delete etcd socket file for unit tests
This change clean up the environment for etcd3 unit test.
Without this change, "make test" will leave some socket files in
workspace. And these socket files make hack/verify-generated-protobuf.sh
fails.

Kubernetes-commit: 4e9dcf3da536a9b956eb092fe9dbc5b2081cfb9d
2017-03-31 20:37:15 +00:00
Jordan Liggitt a22f78080a Preserve custom etcd prefix compatibility for etcd3
Kubernetes-commit: 6853e4d71ea128ff955fad32972ad9edcb376dfb
2017-03-18 19:56:09 +00:00
Clayton Coleman 74dc1360f1 Allow ValueTransformer to indicate resource is stale
Allows a transformer (such as an encrypter) to force an update if a new
key is in use, thus allowing simple writes to the REST layer to
trivially migrate keys.
2017-02-13 07:36:41 -05:00
Clayton Coleman 3d8615c851 Allow values to be wrapped prior to serialization in etcd3 2017-02-13 07:36:41 -05:00
Dr. Stefan Schimanski 4a2fe15be7 Mechanical move: pkg/storage -> k8s.io/apiserver 2017-02-13 07:36:41 -05:00