Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
deprecate insecure http flags and remove already deprecated flags
**What this PR does / why we need it**:
1. deprecate `insecure-bind-address` `insecure-port` flags
2. remove flags `public-address-override` `address` `port` They are mark deprecated in #36604, which is more than a year ago.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58951
**Special notes for your reviewer**:
**Release note**:
```release-note
Deprecate insecure flags `--insecure-bind-address`, `--insecure-port` and remove `--public-address-override`.
```
Kubernetes-commit: 2f175bc43279a1a4552610a42af0a006f4c6fba2
Automatic merge from submit-queue (batch tested with PRs 58955, 58968, 58971, 58963, 58298). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove dead testing code
Removes a deprecated etcd server init function and stops starting an etc server during unit tests that we never use.
Kubernetes-commit: 52c3ff3ab84391cc9658ba1501ad0e70aff5edcc
Automatic merge from submit-queue (batch tested with PRs 58955, 58968, 58971, 58963, 58298). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove --tls-ca-file which had no effect
The flag has had no effect since we started configuring loopback based on the direct cert. I've marked it deprecated this release so we don't break launch scripts right away, but its coming. I think we should remove it in 1.11.
```release-note-action-required
kube-apiserver flag --tls-ca-file has had no effect for some time. It is now deprecated and slated for removal in 1.11. If you are specifying this flag, you must remove it from your launch config before ugprading to 1.11.
```
Kubernetes-commit: d8f32b291751a707d1682003c0c695a530572afa
Automatic merge from submit-queue (batch tested with PRs 56995, 58498, 57426, 58902, 58863). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
cleanup enable/disable api resources code
**What this PR does / why we need it**:
After #57228, `runtime-config` flag has stop support enable/disable resources of a specific groupVersion,
so this pr does some clean work about this.
Mainly delete unused code in `k8s.io/apiserver/pkg/server/storage/resource_config.go`
**Special notes for your reviewer**:
/assign @deads2k @sttts
**Release note**:
```release-note
NONE
```
/kind cleanup
Kubernetes-commit: 5ab8f8cea957ca71b3da4436b6bbcd0abf76546b
Automatic merge from submit-queue (batch tested with PRs 56995, 58498, 57426, 58902, 58863). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
flag precedence redo
Changes the Kubelet configuration flag precedence order so that flags take precedence over config from files/ConfigMaps.
This should fix the re-parse issue with #56097 that led to revert.
Fixes#56171.
In order to prevent global flags (registered in 3rd party libs, etc.) from leaking into the command's help text, this PR turns off Cobra's flag parsing in the `kubelet` command and re-implements help and usage funcs for the Kubelet. Cobra's default funcs automatically merge all global flags into the command's flagset, which results in incorrect help text. I tried to keep the formatting as close as possible to the what the Kubelet currently produces.
Diff between Kubelet's help text on `upstream/master` vs `mtaufen/kc-flags-precedence-redo`, which shows a leaked flag being removed, but no change to the formatting:
```
diff --git a/upstream.master.help b/mtaufen.kc-flags-precedence-redo.help
index 798a030..0797869 100644
--- a/upstream.master.help
+++ b/mtaufen.kc-flags-precedence-redo.help
@@ -30,7 +30,6 @@ Flags:
--authorization-mode string Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (default "AlwaysAllow")
--authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 5m0s)
--authorization-webhook-cache-unauthorized-ttl duration The duration to cache 'unauthorized' responses from the webhook authorizer. (default 30s)
- --azure-container-registry-config string Path to the file containing Azure container registry configuration information.
--bootstrap-checkpoint-path string <Warning: Alpha feature> Path to to the directory where the checkpoints are stored
--bootstrap-kubeconfig string Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.
--cadvisor-port int32 The port of the localhost cAdvisor endpoint (set to 0 to disable) (default 4194)
```
Ultimately, I think we should implement a common lib that K8s components can use to generate clean help text, as the global flag leakage problem affects all core k8s binaries. I would like to do so in a future PR, to keep this PR simple. We could base the help text format on the default values returned from `Command.HelpTemplate` and `Command.UsageTemplate`. Unfortunately, the template funcs used to process these defaults are private to Cobra, so we'd have to re-implement these, or avoid using them.
```release-note
NONE
```
Kubernetes-commit: cf92d921d907f2c3fdf635b690849fd663705638
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use SSH tunnel for webhook communication iff the webhook is deployed as a service
**What this PR does / why we need it**:
We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services.
```
I0119 17:41:18.678436 1 ssh.go:400] [4cdf44753cc3705d: localhost:10258] Dialing...
W0119 17:41:18.678483 1 ssh.go:424] SSH tunnel not found for address "localhost", picking random node
I0119 17:41:18.679810 1 ssh.go:402] [4cdf44753cc3705d: localhost:10258] Dialed in 1.398691ms.
W0119 17:41:18.679928 1 admission.go:256] Failed calling webhook, failing closed xxx: failed calling admission webhook "xxx": Post xxx: ssh: rejected: connect failed (Connection refused)
I0119 17:41:18.680346 1 wrap.go:42] POST /api/v1/namespaces/kube-system/pods: (5.725588ms) 500
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # https://github.com/kubernetes/kubernetes/issues/58779
**Special notes for your reviewer**:
**Release note**:
```release-note
kube-apiserver is changed to use SSH tunnels for webhook iff the webhook is not directly routable from apiserver's network environment.
```
/assign @lavalamp @caesarxuchao @cheftako
Kubernetes-commit: ac495f169b71d51dc9a63c2ff76cf9f596578853
Automatic merge from submit-queue (batch tested with PRs 55792, 58342). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add a metric to track usage of inflight request limit.
This one is tricky. The goal is to know how 'loaded' given apiserver is before we start dropping the load, to so we need to somehow expose 'fullness' of channels.
Sadly this metric is pretty volatile so it's not clear how to do this correctly. I decided to do pre-aggregation to smoothen the metric a bit. In the current implementation the metric publishes maximum "usage" of the inflight is previous second.
If you have any ideas please share.
@smarterclayton @lavalamp @wojtek-t @liggitt @deads2k @caesarxuchao @sttts @crassirostris @hulkholden
```release-note
NONE
```
Kubernetes-commit: a73c96d7b28e8c04a5bd5f68af04eecea3ec14ff
Automatic merge from submit-queue (batch tested with PRs 58518, 58771, 58101, 56829). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Resource version parsing should all be in one place
This is 100% refactoring that ought to be a no-op at run time. It puts resource version parsing in a single file. Doing this because I want to follow up with a change which will make it obvious to users of the system our rules about resource version. Don't want to mix that with this refactor (on the off chance it gets rolled back).
Part of: #58112
```release-note
NONE
```
Kubernetes-commit: d9ded43bbeed4ff0d1708b39e9fabc47b8f596cb
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Never let cluster-scoped resources skip webhooks
Fix#57964
This allows user write webhooks for cluster-scoped custom resources.
We still need to figure out how to selectively exempt cluster-scoped resources from webhooks to avoid bootstrapping deadlocks. For now, if a deadlock occurs, users can work around by first deleting the webhook configuration, then rebooting the webhook, then re-enabling the webhook configuration.
```release-note
Bug fix: webhooks now do not skip cluster-scoped resources
Action required: Before upgrading your Kubernetes clusters, double check if you had configured webhooks for cluster-scoped objects (e.g., nodes, persistentVolume), these webhooks will start to take effect. Delete/modify the configs if that's not desirable.
```
Kubernetes-commit: 0b8f3a2240c7444196e321ded215eafd7a2c96db
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
trace patch operations
Just like `update`, `create`, `get` and `delete` operations.
**Release note**:
```release-note
NONE
```
Kubernetes-commit: a5e33195e4418eac8fde3a9db33f4fbf0f73d87f
Automatic merge from submit-queue (batch tested with PRs 58412, 56132, 58506, 58542, 58394). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
don't stop informer delivery on error
If an informer delivery fails today, we stop delivering to it entirely. The pull updates the code to skip that particular notification, delay, and continue delivery with the next time.
/assign derekwaynecarr
/assign ncdc
/assign ash2k
@derekwaynecarr This would change the "the controller isn't doing anything?!" to "the controller missed my (individual) resource!"
```release-note
NONE
```
Kubernetes-commit: 71426ba59fd4a37e5da7deac6298ab33101bb5b6
Automatic merge from submit-queue (batch tested with PRs 58547, 57228, 58528, 58499, 58618). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add TLS min version flag
Adds a flag for controlling the minimum TLS level allowed.
/assign liggitt
@kubernetes/sig-node-pr-reviews @k8s-mirror-api-machinery-pr-reviews
```release-note
--tls-min-version on kubelet and kube-apiserver allow for configuring minimum TLS versions
```
Kubernetes-commit: 3550551b9f68641f55fc16b6d31eb27ac39b1914
Automatic merge from submit-queue (batch tested with PRs 58547, 57228, 58528, 58499, 58618). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
pass down runtime-config through apiserver chain
**What this PR does / why we need it**:
>kube-apiserver is actually a delegation chain of aggregator+kube+apiextensions. Let's pass down runtime-config through the chain, each layer removing the groups it knows about.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#57189
**Special notes for your reviewer**:
make a new pkg `k8s.io/apiserver/pkg/server/resourceconfig`, and
1. move resourceconfig related code to it.
2. abstract generic used function `MergeAPIResourceConfigs`, put it here.
**Release note**:
```release-note
NONE
```
Kubernetes-commit: f0b73194945d81bb78c23e94db9f1075ce13e642
Automatic merge from submit-queue (batch tested with PRs 58547, 57228, 58528, 58499, 58618). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Send correct resource version for delete events from watch cache
Fixes#58545
the watch cache filtering is returning the previous object content intact, including resource version. this is the logic the watch cache uses:
```go
switch {
case curObjPasses && !oldObjPasses:
watchEvent = watch.Event{Type: watch.Added, Object: event.Object.DeepCopyObject()}
case curObjPasses && oldObjPasses:
watchEvent = watch.Event{Type: watch.Modified, Object: event.Object.DeepCopyObject()}
case !curObjPasses && oldObjPasses:
watchEvent = watch.Event{Type: watch.Deleted, Object: event.PrevObject.DeepCopyObject()}
}
```
when processing a delete event, we should be sending the old object's content *but* with the event's resource version set in it. corresponding logic exists in the uncached stores:
77ac663df4/staging/src/k8s.io/apiserver/pkg/storage/etcd/etcd_watcher.go (L401-L403)77ac663df4/staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go (L373-L378)
```release-note
Fixes an issue where the resourceVersion of an object in a DELETE watch event was not the resourceVersion of the delete itself, but of the last update to the object. This could cause clients receiving the watch event to revert to an old "last observed" resource version, and disrupt their ability to re-establish watches properly.
```
Kubernetes-commit: 619305f1015758f5ffc858a5b3e26d1a779c4278
Automatic merge from submit-queue (batch tested with PRs 58446, 58459, 58340). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add apiserver metric for number of requests dropped by 'max-inflight-requests' filters.
Useful for figuring out on which dimension master is overloaded.
cc @sttts @lavalamp @deads2k @timothysc @hulkholden
Kubernetes-commit: 07ad1f7176e09f0814dba7011596991de2032953
Automatic merge from submit-queue (batch tested with PRs 58496, 58078, 58123). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
refactor admission flag
**What this PR does / why we need it**:
Refactor admission control flag, finally make cluster admins not care about orders in this flag.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Add `--enable-admission-plugin` `--disable-admission-plugin` flags and deprecate `--admission-control`.
Afterwards, don't care about the orders specified in the flags.
```
Kubernetes-commit: f9bb978ad65a924620e1f362059f46731fe454ee
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use Fatalf instead of Fatal while formatting the output is needed
**What this PR does / why we need it**:
Should use Fatalf instead of Fatal while formatting the output is needed
**Release note**:
NONE
```release-note
```
Kubernetes-commit: 7dadeee5e8b84d41c5249a2ca6a8fb64f48c8e77
Automatic merge from submit-queue (batch tested with PRs 57868, 58284, 56370, 58400, 58439). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix decoding of admission config file
Fixes#584261a552bbe14 (diff-eb9532eb476083e1ab31da9dd6f83eceR41) attempted to use a locally constructed scheme, but the name `scheme` was shadowed by a function arg.
Attempts to run the apiserver with a structured plugin config file would fail to decode (since the passed scheme didn't know about the AdmissionConfiguration type), then fall back to treating the file as a legacy config, and silently continue without correct config
```release-note
kube-apiserver: fixes loading of `--admission-control-config-file` containing AdmissionConfiguration apiserver.k8s.io/v1alpha1 config object
```
Kubernetes-commit: c7526fbb67711c5a36eb957b2e315e0efb815c5b
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update vendor spf13/cobra to enforce required flags
**What this PR does / why we need it**:
spf13/cobra#502 has enforced checking flags that marked as required, an error will be raised if unset.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:fixes #54855
xref #48400fixeskubernetes/kubectl#121
**Special notes for your reviewer**:
/assign @liggitt @eparis
**Release note**:
```release-note
kubectl now enforces required flags at a more fundamental level
```
Kubernetes-commit: 048757b8a51333f59d3112d2b228d2f0102a4afc
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Support for custom tls cipher suites in api server and kubelet
**What this PR does / why we need it**:
This pull request aims to solve the problem of users not able to set custom cipher suites in the api server.
Several users have requested this given that some default ciphers are vulnerable.
There is a discussion in #41038 of how to implement this. The options are:
- Setting a fixed list of ciphers, but users will have different requirements so a fixed list would be problematic.
- Letting the user set them by parameter, this requires adding a new parameter that could be pretty long with the list of all the ciphers.
I implemented the second option, if the ciphers are not passed by parameter, the Go default ones will be used (same behavior as now).
**Which issue this PR fixes**
fixes#41038
**Special notes for your reviewer**:
The ciphers in Go tls config are constants and the ones passed by parameters are a comma-separated list. I needed to create the `type CipherSuitesFlag` to support that conversion/mapping, because i couldn't find any way to do this type of reflection in Go.
If you think there is another way to implement this, let me know.
If you want to test it out, this is a ciphers combination i tested without the weak ones:
```
TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
```
If this is merged i will implement the same for the Kubelet.
**Release note**:
```release-note
kube-apiserver and kubelet now support customizing TLS ciphers via a `--tls-cipher-suites` flag
```
Kubernetes-commit: b7100f1ee7231617891a100dd34b3490a1f578e4
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Recheck if transformed data is stale when doing live lookup during update
Fixes#49565
Caching storage can pass in a cached object to `GuaranteedUpdate` as a hint for the current object.
If the hint is identical to the data we want to persist, before short-circuiting as a no-op update, we force a live lookup.
We should check two things on the result of that live lookup before short-circuiting as a no-op update:
1. the bytes we want to persist still match the transformed bytes read from etcd
2. the state read from etcd didn't report itself as stale. this would mean the transformer used to read the data would not be the transformer used to write it, and "no-op" writes should still be performed, since transformation will make the underlying content actually different.
After a live lookup, we checked byte equality, but not the stale indicator. This meant that key rotation or encrypted->decrypted, and decrypted->encrypted updates are broken.
Introduced in #54780 and picked back to 1.8 in #55294
```release-note
Fixed encryption key and encryption provider rotation
```
Kubernetes-commit: 4257f7595abbe7299cbbdaaae4fa36559892f654
Kubernetes Publisher