Commit Graph

37 Commits

Author SHA1 Message Date
Marek Siarkowicz 1ebb228a1f Make object transformation concurrent to remove watch cache scalability issue for conversion webhook
Test by enabling consistent list from cache in storage version migrator stress test that uses
conversion webhook that bottlenects events comming to watch cache.

Set concurrency to 10, based on maximum/average transform latency when
running stress test. In my testing max was about 60-100ms, while average
was 6-10ms.

Kubernetes-commit: bb686f203308481bcd7808f767171cdef27e12a0
2024-07-22 11:24:37 +02:00
Lukasz Szaszkiewicz 90ba08909c storage/etcd: the watcher supports the API streaming
Kubernetes-commit: ca562fd280a9dd5db952eb8c14b93fcf6668cc49
2023-08-22 14:02:39 +02:00
wackxu f40bea647b paginate initial list inside the storage watcher
Signed-off-by: wackxu <xushiwei5@huawei.com>

Kubernetes-commit: f5d6c65186d63647a46400762f849d500e6ac591
2023-07-28 16:43:34 +08:00
Lukasz Szaszkiewicz 206231e0d0 storage/etcd3: error when progressNotify option set and newFunc was provided for a registry
Kubernetes-commit: 1988c31fc8115bf9eec1adf2bffed3fd677d1a9f
2023-08-28 17:50:42 +02:00
Lukasz Szaszkiewicz 7ecfbf7914 storage/etcd: simplify passing the transformer
Kubernetes-commit: a5600b6925169a7d99376d52472e5f8e8635082b
2023-08-21 12:27:41 +02:00
Lukasz Szaszkiewicz dc7ff6e034 storage/etcd: remove newWatcher function
Kubernetes-commit: 6161752ecbe434b8fb157dd3a30bb5c0cef7a23e
2023-08-21 12:13:12 +02:00
Lukasz Szaszkiewicz 2d9be35745 storage/etcd: no-op, refactor watcher.Watch method signature
Kubernetes-commit: f7e659db236286ca15707bf08acb08dc0ac4ab0e
2023-07-24 12:33:03 +02:00
Marek Siarkowicz cd751eb82e Implement conditionalProgressRequester that allows requesting watch progress notification if watch cache is not fresh
Kubernetes-commit: 98461be8ffa7383152c442414a16adb217e98080
2023-07-10 18:10:49 +02:00
Marek Siarkowicz 96d1438818 Refactor WithRequireLeader to make it part of the etcd store
Kubernetes-commit: a9af2de8fdd1f2799b2017b19027958c69ffb04d
2023-06-21 14:21:24 +02:00
Jordan Liggitt 8ea1930d95 Recognize etcd/grpc cancel errors correctly
Kubernetes-commit: 267eb25e60955fe8e438c6311412e7cf7d028acb
2023-03-08 15:51:25 -05:00
Wojciech Tyczyński 2df16457e7 Add metrics for number of events received from etcd
Kubernetes-commit: 85359d74ed0fe2cdb8e2d41cb5a3b501a2c20ed8
2023-02-28 11:16:13 +01:00
Wojciech Tyczyński 499bbb88dc Refactor WatchError test to make it generic
Kubernetes-commit: b3f9272d57a71a9c50b38754b7309ddf68adb93f
2022-10-28 11:35:58 +02:00
Andy Goldstein 8587d29223 etcd3: include GroupResource in logs/metrics
Use GroupResource instead of object reflection when recording the
following metrics:

- etcd_request_duration_seconds
- etcd_bookmark_counts

Add GroupResource to logs and traces where only reflection-based typing
was previously used.

Both of these changes allow us to disginguish between different CRDs,
all of which are represented as *unstructured.Unstructured.

Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>

Kubernetes-commit: 305fa2add60ad507417304d865f001006d5175fe
2022-08-25 13:55:55 -04:00
Steve Kuznetsov af1cb1cefe storage: transformers: pass a context.Context
When an envelope transformer calls out to KMS (for instance), it will be
very helpful to pass a `context.Context` to allow for cancellation. This
patch does that, while passing the previously-expected additional data
via a context value.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 27312feb9983c18d1daf00afba788727d024cdd0
2022-02-17 07:29:44 -08:00
Jordan Liggitt 6365bf9124 Switch to go.etcd.io/etcd/client/v3
Kubernetes-commit: 2979c3325e3ac4023dd19c435f3a995074328751
2021-03-23 11:01:52 -04:00
wojtekt 5290a24345 Implement support for watch initialization in P&F
Kubernetes-commit: 0cc217647ca8be0820973b970124a072c27b6575
2021-05-07 12:49:06 +02:00
Maciej Borsz 770eb2bc44 Update watcher.go
Kubernetes-commit: 532e35ba31e4df0a2df7dd5f2930035ce9379559
2021-05-07 16:07:34 +02:00
wojtekt e40efde837 Implement etcd3 progress-notify feature in etcd3 layer
Kubernetes-commit: 56e72841b6005740453828a9f4f7a9a1b9a831f6
2020-08-31 12:40:41 +02:00
AkashSarda e0d29c69be Migrate API server logs to Structured logging framewrok
Kubernetes-commit: af60e7b34d59d635740880a03e7e2587ec9ba041
2020-06-17 21:06:35 +00:00
Davanum Srinivas 5879417a28 switch over k/k to use klog v2
Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: 442a69c3bdf6fe8e525b05887e57d89db1e2f3a5
2020-04-17 15:25:06 -04:00
Michael Gasch 0cdbc006ea Add etcd WithRequireLeader option to API watches
Watches against etcd in the API server can hang forever if the etcd
cluster loses quorum, e.g. the majority of nodes crashes. This fix
improves responsiveness (detection and reaction time) of API server
watches against etcd in some rare (but still possible) edge cases so
that watches are terminated with `"etcdserver: no leader"
(ErrNoLeader)`.

Implementation behavior described by jingyih:

```
The etcd server waits until it cannot find a leader for 3 election
timeouts to cancel existing streams. 3 is currently a hard coded
constant. The election timeout defaults to 1000ms.

If the cluster is healthy, when the leader is stopped, the leadership
transfer should be smooth. (leader transfers its leadership before
stopping). If leader is hard killed, other servers will take an election
timeout to realize leader lost and start campaign.
```

For further details, discussion and validation see
https://github.com/kubernetes/kubernetes/issues/89488#issuecomment-606491110
and https://github.com/etcd-io/etcd/issues/8980.

Closes: https://github.com/kubernetes/kubernetes/issues/89488

Signed-off-by: Michael Gasch <mgasch@vmware.com>

Kubernetes-commit: 70c9f770d7aa2194bfd3f58fe01756a7d200b866
2020-03-31 12:01:53 +02:00
Ted Yu 006e524c38 Log mvcc revision compaction as warning
Kubernetes-commit: 9fec783d805ebdb05f1936dffd9621702eb0f711
2019-10-31 07:24:17 -07:00
danielqsj 76c4ab8bb5 unify alias of api errors under pkg and staging
Kubernetes-commit: 5bc0e26c1902e7e28abfc828de19a2dbb3e492c2
2019-11-12 16:26:59 +08:00
Wenjia Zhang 41da2ced4c Replace github.com/coreos/etcd by go.etcd.io/etcd
Kubernetes-commit: 3b274fad2a719dc1fd0eaef6c55a0d344db10242
2019-10-15 22:38:43 -07:00
Joe Betz db6e870285 Add webhook converter integration test suite
Kubernetes-commit: e23769f018ff50763656f99edd907e91e8bfff3e
2019-05-08 22:42:41 -07:00
Ryan McNamara cae815db01 Error when etcd3 watch finds delete event with nil prevKV
Kubernetes-commit: 504380682fce0b60318d64999554ccd518b8fc63
2019-04-16 12:55:15 -07:00
Davanum Srinivas 2710b17b80 Move from glog to klog
- Move from the old github.com/golang/glog to k8s.io/klog
- klog as explicit InitFlags() so we add them as necessary
- we update the other repositories that we vendor that made a similar
change from glog to klog
  * github.com/kubernetes/repo-infra
  * k8s.io/gengo/
  * k8s.io/kube-openapi/
  * github.com/google/cadvisor
- Entirely remove all references to glog
- Fix some tests by explicit InitFlags in their init() methods

Change-Id: I92db545ff36fcec83afe98f550c9e630098b3135

Kubernetes-commit: 954996e231074dc7429f7be1256a579bedd8344c
2018-11-09 13:49:10 -05:00
Jordan Liggitt 1f9c7bdd99 Quiet verbose apiserver logs
Kubernetes-commit: 862f8567bb4b1d4e9f63330cf3a516af61ae7cc5
2018-06-11 22:32:46 -04:00
hzxuzhonghu 6ab99203b7 Replace "golang.org/x/net/context" with "context"
Kubernetes-commit: 70e45eccf27726f0e63dd1024924ccc7e2cd35a0
2018-02-28 12:20:22 +08:00
Di Xu 7a46c8324d remove FilterFunc and use SelectionPredicate everywhere
Kubernetes-commit: 3ae7bdd211e3df1350446b6def142b4d31c75e52
2017-11-22 16:57:06 +08:00
Clayton Coleman 460257fd61 Server side implementation of paging for etcd3
Add a feature gate in the apiserver to control whether paging can be
used. Add controls to the storage factory that allow it to be disabled
per resource. Use a JSON encoded continuation token that can be
versioned. Create a 410 error if the continuation token is expired.

Adds GetContinue() to ListMeta.

Kubernetes-commit: 8952a0cb722b77459cf2701632a30f5b264f5aba
2017-09-03 14:04:12 +00:00
Clayton Coleman 5fa08b8c5e Allow initialization of resources
Add support for creating resources that are not immediately visible to
naive clients, but must first be initialized by one or more privileged
cluster agents. These controllers can mark the object as initialized,
allowing others to see them.

Permission to override initialization defaults or modify an initializing
object is limited per resource to a virtual subresource "RESOURCE/initialize"
via RBAC.

Initialization is currently alpha.

Kubernetes-commit: 331eea67d8000e5c4b37e2234a90903c15881c2f
2017-06-13 20:47:30 +00:00
Monis Khan 6794013a5b Panic server on watch errors in test environment
This change makes it so that errors during watch decoding panic the
server if it is in a test environment.  This allows us to catch coder
errors related to storing incompatible types at the same location in
etcd.

Signed-off-by: Monis Khan <mkhan@redhat.com>

Kubernetes-commit: a13f026fd012859f04467e6007e2cafe4a788927
2017-06-13 20:47:29 +00:00
Clayton Coleman f27996225a Update etcd3 storage to leverage storage/value interfaces
Adds context argument which must be set for AES GCM authenticated data
to be passed.

Kubernetes-commit: a73990a33f95713f026ee7ae9ae6741255aaf8e4
2017-05-17 17:27:53 +00:00
Clayton Coleman 74dc1360f1 Allow ValueTransformer to indicate resource is stale
Allows a transformer (such as an encrypter) to force an update if a new
key is in use, thus allowing simple writes to the REST layer to
trivially migrate keys.
2017-02-13 07:36:41 -05:00
Clayton Coleman 3d8615c851 Allow values to be wrapped prior to serialization in etcd3 2017-02-13 07:36:41 -05:00
Dr. Stefan Schimanski 4a2fe15be7 Mechanical move: pkg/storage -> k8s.io/apiserver 2017-02-13 07:36:41 -05:00