Commit Graph

533 Commits

Author SHA1 Message Date
David Ashpole aa161f2fc0 migrate apiserver utiltrace usage to component-base/tracing
Kubernetes-commit: de26b9023f2872c5cd7e15fad5dd5ab649222c13
2022-10-20 18:15:38 +00:00
Tim Allclair bd7c7f52c2 Consolidate AuditContext
Kubernetes-commit: f1d684b7b60b39b7dc1eb4156307c593f0ba74e1
2022-07-12 11:53:57 -07:00
Wojciech Tyczyński 3afe8ebf5f Minor cleanup of etcd3 tests
Kubernetes-commit: e04fe81dfcff189ef5162b9b26b55760f613a010
2022-10-26 13:05:49 +02:00
Wojciech Tyczyński 73a664638b Refactor pagination tests
Kubernetes-commit: 6c8ce894e18a2e2cae20edd2f6e79db9407ebce3
2022-10-26 12:18:21 +02:00
Wojciech Tyczyński dec6f492e8 Refactor GetReads
Kubernetes-commit: 8472e1bc13b0fa8aa3a67e5fcf2f13bfd7974cd3
2022-10-26 11:30:32 +02:00
Wojciech Tyczyński 5a8fdf8480 Move GuaranteedUpdateChecksData test to generic package
Kubernetes-commit: afc5ded83931cdb8a2643486d6834eb8d9b04588
2022-10-25 14:31:20 +02:00
Wojciech Tyczyński d9a3685d6f Minor cleanup of storage tests
Kubernetes-commit: 5344bc5e1b7e50d5ff359c714d522b421b28bd12
2022-10-25 14:09:36 +02:00
Wojciech Tyczyński 1960fa837f Move GuaranteedUpdate test to generic test package
Kubernetes-commit: 012676acc3dd2b235bd601e4105294d2715f155b
2022-10-25 14:04:00 +02:00
Han Kang e9f139f163 swap name for storage metric for underlying storage db
Change-Id: Ic7cd9bf5ef29d9c51a728ef37c94d77172fbf678

Kubernetes-commit: 3a2d1bab26bfb9e7d887207dec0c51645c7017fe
2022-10-24 15:27:05 -04:00
Oscar Utbult 9d3d7b483a grammar: replace all occurrences of "the the" with "the"
Kubernetes-commit: e4f776f23098ecf942cafa898777195adbc800f1
2022-09-17 22:58:01 +02:00
lixiaobing1 c144979a82 replace WithInsecure() with WithTransportCredentials()
Kubernetes-commit: 7892175acdb329d44cf1f34230f78e608b3cb736
2022-10-15 16:41:53 +08:00
Monis Khan f4ac8fc8bb prefixTransformers: allocate after transformation length is known
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 59e1a32fc8ed35e328a3971d3a1d640ffc28ff55
2022-10-17 16:56:40 -04:00
Monis Khan be9579fc15 k8s.io/apiserver/pkg/storage/value: allow encryption-at-rest approval
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: b68bc0678d5876e4c11c7d5289f777b6e37c4917
2022-10-17 17:07:56 -04:00
Harsha Narayana 1da54ec21a kmsv2: enable logging for kmsv2 enc/dec operations
Kubernetes-commit: 79d741f1f8efcfc75cecd22898c7b6b689449f0a
2022-08-31 22:08:55 +05:30
Monis Khan 8d68e6f323 Load encryption config once
This change updates the API server code to load the encryption
config once at start up instead of multiple times.  Previously the
code would set up the storage transformers and the etcd healthz
checks in separate parse steps.  This is problematic for KMS v2 key
ID based staleness checks which need to be able to assert that the
API server has a single view into the KMS plugin's current key ID.

Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: f507bc255382b2e2095351053bc17e74f7100d35
2022-08-29 17:25:48 -04:00
Anish Ramasekar 1411f0e151 kmsv2: validate annotations key and size
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: d1fb258ff2d009f202cff3fdd25e6fd2bbda08ef
2022-09-14 21:58:17 +00:00
Monis Khan c602291fa1 encryption config: no-op refactor to prepare for single loading
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: db850931a8699e780dd794e1763fd0e54b4239b5
2022-08-29 17:25:48 -04:00
Anish Ramasekar c027ae3881 Add staging directory for kms
- Moves kms proto apis to the staging repo
- Updates generate and verify kms proto scripts to check staging repo

Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: c3794e2377016b1c18b1dcb63dc61d686c8ebcbf
2022-08-23 20:22:09 +00:00
Anish Ramasekar ec520ccd91 [KMS]: update envelope caching unit tests
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 92dce5de71f752c8c136ec7c7417a73d50317cf5
2022-09-15 18:01:48 +00:00
viveksahu26 6cb2fda0ab fix warnings or linter errors
Signed-off-by: viveksahu26 <vivekkumarsahu650@gmail.com>

Kubernetes-commit: 386bc4a7fa3e975cb247d300fdc5ad1b14a15605
2022-09-05 19:10:43 +05:30
Antonio Ojea 1239036585 rate limit /healthz etcd healthchecks
return the last request error, instead of last error received
The rate limit allows 1 event per healthcheck timeout / 2

Kubernetes-commit: 510a85c53a5138babb1650fadd328e6f34baa03b
2022-08-29 11:09:58 +02:00
Antonio Ojea d2581bb0e0 fix etcd unit tests
stop leaking goroutines

reduce etcd test duration

Kubernetes-commit: dd6d3d95cdeb0e165e8365212d85d0f3b972d3e8
2022-08-28 23:13:45 +02:00
Antonio Ojea fb26cb4f0c don't serialize etcd healthchecks
Kubernetes-commit: 5a67248115ec0bc762c0351a73810117ac0bb814
2022-08-29 17:57:01 +02:00
Monis Khan 70b4742ce2 kms: fix go routine leak in gRPC connection
Signed-off-by: Monis Khan <mok@microsoft.com>

Kubernetes-commit: 4e68e9b5ad70ae074b3fb20f0fb2ba25d0792274
2022-08-24 01:51:19 +00:00
Andy Goldstein 8587d29223 etcd3: include GroupResource in logs/metrics
Use GroupResource instead of object reflection when recording the
following metrics:

- etcd_request_duration_seconds
- etcd_bookmark_counts

Add GroupResource to logs and traces where only reflection-based typing
was previously used.

Both of these changes allow us to disginguish between different CRDs,
all of which are represented as *unstructured.Unstructured.

Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>

Kubernetes-commit: 305fa2add60ad507417304d865f001006d5175fe
2022-08-25 13:55:55 -04:00
Andy Goldstein 7eb011f596 watch cache: metrics: objectType -> group resource
Use the group resource instead of objectType in watch cache metrics,
because all CustomResources are grouped together as
*unstructured.Unstructured, instead of 1 entry per type.

Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>

Kubernetes-commit: d08b69e8d35a5aa73a178c508f9b0e1ad74b882d
2022-08-11 15:51:21 -04:00
Andy Goldstein ecf3a57374 watch cache: log GroupResource, not objectType
All CustomResources are treated as *unstructured.Unstructured, leading
the watch cache to log anything related to CRs as Unstructured. This
change uses the schema.GroupResource instead of object type for all type
related log messages in the watch cache, resulting in distinct output
for each CR type.

Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>

Kubernetes-commit: 397533a4c2df9639ff4422c907d06fae195a1835
2022-08-11 15:45:04 -04:00
Anish Ramasekar bdd7082eed chore(kms): remove unused plugin name and migrate from deprecated `io/ioutil` pkg
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 7db7a63959162d743f771183bf4e88e82afef868
2022-08-23 22:55:22 +00:00
Anish Ramasekar 225e26ac4a Implement KMS v2alpha1
- add feature gate
- add encrypted object and run generated_files
- generate protobuf for encrypted object and add unit tests
- move parse endpoint to util and refactor
- refactor interface and remove unused interceptor
- add protobuf generate to update-generated-kms.sh
- add integration tests
- add defaulting for apiVersion in kmsConfiguration
- handle v1/v2 and default in encryption config parsing
- move metrics to own pkg and reuse for v2
- use Marshal and Unmarshal instead of serializer
- add context for all service methods
- check version and keyid for healthz

Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: f19f3f409938ff9ac8a61966e47fbe9c6075ec90
2022-06-29 20:51:35 +00:00
Sally O'Malley 4f9e133507 kubelet tracing
Signed-off-by: Sally O'Malley <somalley@redhat.com>
Co-authored-by: David Ashpole <dashpole@google.com>

Kubernetes-commit: 47e7d8034ff3be8e198dde6a671d05a11c30e333
2021-10-10 09:17:27 -04:00
Nic Cope 9b243e9d90 Copy etcd client debug level logic from upstream
Replicated from https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/logger.go#L47

The logic of this function doesn't make a lot of sense to me, but
copying it will avoid any behaviour change.

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: c1aa7a0fe73cbcab8e70f7b73a845ae9394f9a71
2022-07-29 14:26:31 -07:00
Nic Cope a105c2570c Give etcd client logger a name
Logic copied from https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/client.go#L374

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: f54d2606336e2e8130339d2a0bc04fac6906aa78
2022-07-29 14:24:17 -07:00
Nic Cope 463756f91d Share a single etcd3 client logger across all clients
Currently the API server creates one etcd client per CRD. If clients
aren't provided a logger they'll each create their own. These loggers
can account for ~20% of API server memory consumption on a cluster with
hundreds of CRDs.

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: 0c81eabb853e581abbcb37ebf094af3316e1012e
2022-07-28 19:51:55 -07:00
Nic Cope ef17269e10 Disable the etcd3 client logger
This logger is responsible for 20% of the API server's memory usage when
many CRDs are installed. See the below issue for more context.

https://github.com/kubernetes/kubernetes/issues/111476

Signed-off-by: Nic Cope <nicc@rk0n.org>

Kubernetes-commit: 0e5401c93940126beac45264aa056507b0950075
2022-07-27 14:44:49 -07:00
Wojciech Tyczyński 6ab5ae374a Fix draining cacher tests
Kubernetes-commit: a530a6898a36d43eef766f041eab4e4b6d5adedb
2022-07-28 10:20:23 +02:00
Lukasz Szaszkiewicz 684cf11215 cacher: add support for consistent streaming
design details https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/3157-watch-list#design-details

Kubernetes-commit: debace151cfffc1b5d805341efcbf5f0062e95ed
2022-07-05 15:48:30 +02:00
Marcel Zięba 3c024b4916 Add option to retry internal api error in reflector.
Kubernetes-commit: 0b2b6489de8f75d5299f54180617601126bb8878
2022-07-25 08:02:54 +00:00
Maciej Wyrzuc cb0bb2af35 Add additional etcd check to readyz with 2 seconds timeout.
Kubernetes-commit: b42045a64fd07fb948660839b6c7c14440bee9df
2022-07-25 13:08:50 +00:00
Davanum Srinivas 7e94033a61 Generate and format files
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh

Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Kubernetes-commit: a9593d634c6a053848413e600dadbf974627515f
2022-07-19 20:54:13 -04:00
Mikko Ylinen 12a8b7fef3 grpc: move to use grpc.WithTransportCredentials()
v1.43.0 marked grpc.WithInsecure() deprecated so this commit moves to use
what is the recommended replacement:

grpc.WithTransportCredentials(insecure.NewCredentials())

Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>

Kubernetes-commit: 2c8bfad9106039aa15233b5bf7282b25a7b7e0a0
2022-05-11 12:13:28 +03:00
Wojciech Tyczyński 2049cfbb9e Adjust watch channel sizes in watchcache
Kubernetes-commit: 0db5c05bdb8bbc510307a48cbade712583bb009e
2022-04-28 11:56:41 +02:00
Lukasz Szaszkiewicz 27a7c443bd cacher: with expiredBookmarkWatchers
expiredBookmarkWatchers allows us to schedule the next bookmark event after dispatching not before as it was previously.
It opens a new functionality in which a watcher might decide to change when the next bookmark should be delivered based on some internal state.

Kubernetes-commit: 0576f6a011cba8f0c8550fd3dd31111376c9dcd0
2022-07-15 15:28:50 +02:00
Artur Żyliński e34c622d49 Add audit-id to storage traces
Refactor GetAuditIDTruncated to use context instead of request

Kubernetes-commit: b1e12b01b6c578da3eb593805b48e9d4a69efe54
2022-06-20 17:09:32 +02:00
zk ad01c712d9 Update etcdRequestLatency metrics bucket size
Kubernetes-commit: fb372d07982463984dfdab7cf24d937b91b0bb83
2022-01-09 23:45:09 +08:00
Madhav Jivrajani 218ed2fae0 cacher: Use PodList type for use in GetList
Using a Pod type in a GetList() call in a test
can panic at worst and error out at best. Here,
neither happened because the error condition
being tested for (cacher being stopped or not)
gets returned before the list pointer can be
enforced.

This commit changes the above to use PodList.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>

Kubernetes-commit: 487761f4e2543114db158f0d59e598dedc481882
2022-07-15 12:22:04 +05:30
Anish Ramasekar c6c1465ed7 Add KMS v2alpha1 API
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>

Kubernetes-commit: 907545445ab8b4e34c1068ab9828a930c30cbfc4
2022-05-24 23:43:09 +00:00
Abirdcfly 6be1a93b55 fix: --chunk-size with selector returns missing result
Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Kubernetes-commit: fba297629d82866078faa6dc076cb5f81633e557
2022-06-20 00:54:57 +08:00
Han Kang a414002089 cleanup deprecated metrics and usages
Kubernetes-commit: f223b900907b71431d7b6ceefa1642bb44fd9d84
2022-06-01 11:55:14 -07:00
Han Kang de98becc1d add explicit typing for continue tests
Our tests are mostly error based and explicit error typing allows
us to test against error types directly. Having made this change also
makes it obvious that our test coverage was lacking in two branches,
specifically, we were previously not testing empty start keys nor were
we testing for invalid start RVs.

Kubernetes-commit: 213e380a2e48830db6c71d2da5485d4226d95625
2022-05-31 10:23:07 -07:00
Steve Kuznetsov 03363cd42b storage: move some list tests to generic package
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

Kubernetes-commit: 162450c01c9004ded53bb77f651f9a3f30920b30
2022-05-12 15:16:07 -07:00