Automatic merge from submit-queue (batch tested with PRs 67756, 64149, 68076, 68131, 68120). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
kube-ctrl-mgr: enable secure port 10257
This PR enables authn+authz (delegated to the kube-apiserver) and the secure port 10257 for the kube-controller-manager. In addition, the insecure port is disabled.
Moreover, it adds integration test coverage for the `--port` and `--secure-port` flags, plus the testserver infrastructure to tests flags in general inside integration tests.
```release-note
Enable secure serving on port 10257 to kube-controller-manager (configurable via `--secure-port`). Delegated authentication and authorization have to be configured like for aggregated API servers.
```
Kubernetes-commit: 5d4b0f81bf809d517e880aea935264f5497c1e3c
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Refactor admission webhook client code to a apiserver/pkg/util package
As part of #67006 This refactoring enable us to share code between admission webhooks and CRD conversion webhooks.
@deads2k @lavalamp @sttts @kubernetes/sig-api-machinery-misc
Kubernetes-commit: 14eb029fba5717e2dfe3434a682f9aec4752c44d
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Fix grammar in secure-port flag help
The phrasing made it difficult to understand the message.
```release-note
NONE
```
Kubernetes-commit: 95e85a5dfb0d766479a1708a3363192af51c8419
Automatic merge from submit-queue (batch tested with PRs 67764, 68034, 67836). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
apiserver: make not-found external-apiserver-authn configmap non-fatal
As client-ca and requestheader-client-ca is optional in the external-apiserver-authentication config file and components like kube-controller-manager and kube-scheduler won't need that anyway, we better make it non-fatal if the configmap is not found in the cluster.
Consumer counter-part PR to https://github.com/kubernetes/kubernetes/pull/67694.
```release-note
Don't let aggregated apiservers fail to launch if the external-apiserver-authentication configmap is not found in the cluster.
```
Kubernetes-commit: 55859a60fe09c412e183c92ad265cf0d52fbe3d8
This is the old behaviour and we did not intent to change it due to enabled authn/z in general.
As the kube-apiserver this sets the "system:unsecured" user info.
Kubernetes-commit: 8aa0eefce8fbd801a38da46c8704f2d74996e5cd
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Promote AdvancedAuditing to GA
**What this PR does / why we need it**:
Removes deprecated legacy code used for basic audit logging in favor of advanced audit logging.
```release-note
Promote AdvancedAuditing to GA, replacing the previous (legacy) audit logging mechanisms.
```
Kubernetes-commit: 58ead1c1e8b450e75956bf17153427aa7b3966f2
Automatic merge from submit-queue (batch tested with PRs 66577, 67948, 68001, 67982). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
apiserver: forward panic in WithTimeout filter
```release-note
Return apiserver panics as 500 errors instead terminating the apiserver process.
```
Without this PR a panic in a HTTP handler will not be caught in the Go routine started by the timeout filter. Uncaught panics terminate the process.
This is a strong condidate to be backported to 1.11, 1.10 and 1.9.
Kubernetes-commit: ca8f267cc86c183a23668e24a59a0eaa973daf93
Automatic merge from submit-queue (batch tested with PRs 66960, 67545). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
delegated authn/z: optionally opt-out of mandatory authn/authz kubeconfig
This adds `RemoteKubeConfigFileOptional` field to the delegated authn/z option structs. If set to true, the authn/z kubeconfig file flags are optional. If no kubeconfig is given, all token requests are considered to be anonymous and no client CA is looked up in the cluster.
Prerequisite for https://github.com/kubernetes/kubernetes/pull/64149 and https://github.com/kubernetes/kubernetes/pull/67069.
Kubernetes-commit: 1b3a2dd0830ca0e02d5b95d2ecc0161d0c93a0c7
Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Size http2 buffers to allow concurrent streams
http/2 requests from a given client multiplex over a single connection via streams, chopped up into frames.
The amount of data the client is allowed to send for a given stream and for the overall connection before acknowledgement is determined by the server's MaxUploadBufferPerStream and MaxUploadBufferPerConnection settings respectively, both defaulting to 1MB.
The number of concurrent streams a client is allowed to send over a single connection is determined by the server's MaxConcurrentStreams setting, defaulting to 250.
We observed a starvation issue with the kube aggregator's proxy client if handling of a POST through the aggregator to a backend server exceeded the 1MB buffer size AND the backend server required a second POST request through the aggregator to be handled before it could drain the first request's body.
Logically, if concurrent streams are allowed in a single connection, the connection buffer should be MaxUploadBufferPerStream*MaxConcurrentStreams to allow individual streams to make progress even when one stream is blocked.
This PR shrinks the `MaxUploadBufferPerStream` size to 256kb (which is still large enough to allow all the resources we saw in our test clusters to be sent in a single frame), and grows the MaxUploadBufferPerConnection to accomodate concurrent streams.
I'm also opening a golang issue, [reproducer](https://gist.github.com/liggitt/00239c99b4c148ac1b23e57f86b3af93), and fix for the defaults for this
```release-note
adjusted http/2 buffer sizes for apiservers to prevent starvation issues between concurrent streams
```
Kubernetes-commit: 9edf196c017c12ade054f6b5ecdc7ffcb66ee94d
Automatic merge from submit-queue (batch tested with PRs 67894, 64097). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
HPA metrics specificity improvements
**What this PR does / why we need it**:
Improves available specificity for HPA metrics by adding metric selector fields for metrics of Pods and Objects.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Implements this KEP: https://github.com/kubernetes/community/pull/2055
**Special notes for your reviewer**:
Need to add/update tests?
**Release note**:
```release-note
Introduces autoscaling/v2beta2 and custom_metrics/v1beta2, which implement metric selectors for Object and Pods metrics, as well as allowing AverageValue targets on Objects, similar to External metrics.
```
/assign @DirectXMan12
Kubernetes-commit: fdb5707194d56cbbd0da11c5be3a2a5653e714c9
Automatic merge from submit-queue (batch tested with PRs 54935, 67768, 67896, 67787). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
expose generic storage factory primitives
The `StorageFactory` is central to the idea of a generic and decorated storage configuration for a given resource. This pull exposes the "standard" one from the generic apiserver to allow easier composition without duplication.
@kubernetes/sig-api-machinery-pr-reviews
/assign @yue9944882 @sttts
```release-note
NONE
```
Kubernetes-commit: 5abe4ccec8250e8a1401e7d612c2f4635eba3e11
Automatic merge from submit-queue (batch tested with PRs 54935, 67768, 67896, 67787). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
apiserver: unify handling of unspecified options in authn/z options
Use nil pointer for unspecified, not some kind of half zero-valued structs.
Kubernetes-commit: 5afa23f2c4d1c4501ed722e43dab4b1f64b0fddd
Automatic merge from submit-queue (batch tested with PRs 67362, 67256, 67809). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Dry run integration
Implement an integration test for dry-run. Also, this turns on the knob to allow dry-run requests, so let's be careful.
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 7b96c818df5da720f68ea8ba1a1fc44386ad21a8
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
delegated authz: add AlwaysAllowPaths to option struct (defaulting to /healthz)
Add `AlwaysAllowPaths` field to delegated authz. These http paths are excluded from the authz chain.
Prerequisite for https://github.com/kubernetes/kubernetes/pull/64149 and https://github.com/kubernetes/kubernetes/pull/67069.
```release-note
Added --authorization-always-allow-paths to components doing delegated authorization to exclude certain HTTP paths like /healthz from authorization.
```
Kubernetes-commit: 5ed26a348b017c3ece8ac468d15770ddf8b922ae
Automatic merge from submit-queue (batch tested with PRs 67707, 62420). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
svcacct: support jose.OpaqueSigner and push errors to token generator creation
```release-note
NONE
```
Kubernetes-commit: 166a1356e83b1c271c0d58aa4d3fb69bab713072
Automatic merge from submit-queue (batch tested with PRs 67576, 66936). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Support dry run in admission webhooks
**What this PR does / why we need it**:
Follow up to https://github.com/kubernetes/kubernetes/pull/66391
- [x] add DryRun to ```admission.k8s.io/v1beta1.AdmissionReview```
- [x] add DryRunnable to ```admissionregistration.k8s.io/v1beta1.(Valid|Mut)atingWebhookConfiguration```
- [x] add dry run support to (Valid|Mut)atingAdmissionWebhook
Includes all the api-changes outlined by https://github.com/kubernetes/community/pull/2387
/sig api-machinery
**Release note**:
```release-note
To address the possibility dry-run requests overwhelming admission webhooks that rely on side effects and a reconciliation mechanism, a new field is being added to admissionregistration.k8s.io/v1beta1.ValidatingWebhookConfiguration and admissionregistration.k8s.io/v1beta1.MutatingWebhookConfiguration so that webhooks can explicitly register as having dry-run support. If a dry-run request is made on a resource that triggers a non dry-run supporting webhook, the request will be completely rejected, with "400: Bad Request". Additionally, a new field is being added to the admission.k8s.io/v1beta1.AdmissionReview API object, exposing to webhooks whether or not the request being reviewed is a dry-run.
```
Kubernetes-commit: 5a16163c87fe2a90916a51b52771a668bcaf2a0d
Automatic merge from submit-queue (batch tested with PRs 67378, 67675, 67654). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add some unit tests in staging/src/k8s.io/apiserver/pkg/server/options
**What this PR does / why we need it**:
add some unit tests in staging/src/k8s.io/apiserver/pkg/server/options
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 5930ea30c196678864a29d5cc1bc3710048bef5b
Automatic merge from submit-queue (batch tested with PRs 67298, 67518, 67635, 67673). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix unstructured metadata accessors to respect omitempty semantics
Fixes#67541Fixes#48211Fixes#49075
Follow up of #67562
`ObjectMeta` has fields with `omitempty` json tags. This means that when the fields have zero values, they should not be persisted in the object.
Before this PR, some of the metadata accessors for unstructured objects did not respect these semantics i.e they would persist a field even if it had a zero value.
This PR updates the accessors so that the field is removed from the unstructured object map if it contains a zero value.
/sig api-machinery
/kind bug
/area custom-resources
/cc sttts liggitt yue9944882 roycaihw
/assign sttts liggitt
**Release note**:
```release-note
NONE
```
Kubernetes-commit: ce8a6285dfa0bcc96e6f8ef0e21ef40f958930c7
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Log real file's name and line
**What this PR does / why we need it**:
Have correct location of emission in the logs
**Release note**:
pkg/kubectl/util/logs & staging/src/k8s.io/apiserver/pkg/util/logs
use `glog.info(...)` but this function is not made to be wrapped because
the underlying mechanism use a fixed call trace length to determine
where the log has been emited.
This results is having `logs.go:49` in the logs which is in the body
of the wrapper function and thus useless.
Instead use `glog.infoDepth(1, ...)` which tells the underlying mechanism
to go back 1 more level in the call trace to determine where the log
has been emitted.
Kubernetes-commit: 70bc7f687707c7df4d182a14f2089b6e87992db1
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Cleanup apiserver errors
- delete unused methods (is this safe, or might they be used in another project outside kubernetes/kubernetes?)
- use the standard library `http.Error` function (functional change adds a newline)
```release-note
NONE
```
/sig api-machinery
/kind cleanup
Kubernetes-commit: 30e4f528ed30a70bdb0c14b5cfe49d00a78194c2
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Prevent resourceVersion updates for custom resources on no-op writes
Fixes partly https://github.com/kubernetes/kubernetes/issues/67541
For ObjectMeta pruning, we round trip by marshalling and unmarshalling. If the ObjectMeta contained any strings with `""` (or other fields with empty values) _and_ the respective fields are `omitempty`, those fields will be lost in the round trip process.
This makes ObjectMeta after the no-op write different from the one before the write.
Resource version is incremented every time data is written to etcd. Writes to etcd short-circuit if the bytes being written are identical to the bytes already present.
So this ends up incrementing the `resourceVersion` even on no-op writes. This PR updates the `BeforeUpdate` function such that omitempty fields have values set only if they are non-zero so that they produce an unstructured object that matches ObjectMeta omitempty semantics.
/sig api-machinery
/kind bug
/area custom-resources
/assign sttts liggitt
**Release note**:
```release-note
Prevent `resourceVersion` updates for custom resources on no-op writes.
```
Kubernetes-commit: 816f2a48689247c8344d911995b0f97c721ad481
Automatic merge from submit-queue (batch tested with PRs 55600, 67386). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update Annotations description about audit.Event
ref: https://github.com/kubernetes/kubernetes/pull/58679#discussion_r209375632
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
/assign @liggitt
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 4c5e6156525b96b72961b86ff5bd82c44ea0cd96
Kubernetes Publisher
