Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Promote AdvancedAuditing to GA
**What this PR does / why we need it**:
Removes deprecated legacy code used for basic audit logging in favor of advanced audit logging.
```release-note
Promote AdvancedAuditing to GA, replacing the previous (legacy) audit logging mechanisms.
```
Kubernetes-commit: 58ead1c1e8b450e75956bf17153427aa7b3966f2
Automatic merge from submit-queue (batch tested with PRs 66577, 67948, 68001, 67982). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
apiserver: forward panic in WithTimeout filter
```release-note
Return apiserver panics as 500 errors instead terminating the apiserver process.
```
Without this PR a panic in a HTTP handler will not be caught in the Go routine started by the timeout filter. Uncaught panics terminate the process.
This is a strong condidate to be backported to 1.11, 1.10 and 1.9.
Kubernetes-commit: ca8f267cc86c183a23668e24a59a0eaa973daf93
Automatic merge from submit-queue (batch tested with PRs 66960, 67545). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
delegated authn/z: optionally opt-out of mandatory authn/authz kubeconfig
This adds `RemoteKubeConfigFileOptional` field to the delegated authn/z option structs. If set to true, the authn/z kubeconfig file flags are optional. If no kubeconfig is given, all token requests are considered to be anonymous and no client CA is looked up in the cluster.
Prerequisite for https://github.com/kubernetes/kubernetes/pull/64149 and https://github.com/kubernetes/kubernetes/pull/67069.
Kubernetes-commit: 1b3a2dd0830ca0e02d5b95d2ecc0161d0c93a0c7
Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Size http2 buffers to allow concurrent streams
http/2 requests from a given client multiplex over a single connection via streams, chopped up into frames.
The amount of data the client is allowed to send for a given stream and for the overall connection before acknowledgement is determined by the server's MaxUploadBufferPerStream and MaxUploadBufferPerConnection settings respectively, both defaulting to 1MB.
The number of concurrent streams a client is allowed to send over a single connection is determined by the server's MaxConcurrentStreams setting, defaulting to 250.
We observed a starvation issue with the kube aggregator's proxy client if handling of a POST through the aggregator to a backend server exceeded the 1MB buffer size AND the backend server required a second POST request through the aggregator to be handled before it could drain the first request's body.
Logically, if concurrent streams are allowed in a single connection, the connection buffer should be MaxUploadBufferPerStream*MaxConcurrentStreams to allow individual streams to make progress even when one stream is blocked.
This PR shrinks the `MaxUploadBufferPerStream` size to 256kb (which is still large enough to allow all the resources we saw in our test clusters to be sent in a single frame), and grows the MaxUploadBufferPerConnection to accomodate concurrent streams.
I'm also opening a golang issue, [reproducer](https://gist.github.com/liggitt/00239c99b4c148ac1b23e57f86b3af93), and fix for the defaults for this
```release-note
adjusted http/2 buffer sizes for apiservers to prevent starvation issues between concurrent streams
```
Kubernetes-commit: 9edf196c017c12ade054f6b5ecdc7ffcb66ee94d
Automatic merge from submit-queue (batch tested with PRs 67894, 64097). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
HPA metrics specificity improvements
**What this PR does / why we need it**:
Improves available specificity for HPA metrics by adding metric selector fields for metrics of Pods and Objects.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Implements this KEP: https://github.com/kubernetes/community/pull/2055
**Special notes for your reviewer**:
Need to add/update tests?
**Release note**:
```release-note
Introduces autoscaling/v2beta2 and custom_metrics/v1beta2, which implement metric selectors for Object and Pods metrics, as well as allowing AverageValue targets on Objects, similar to External metrics.
```
/assign @DirectXMan12
Kubernetes-commit: fdb5707194d56cbbd0da11c5be3a2a5653e714c9
Automatic merge from submit-queue (batch tested with PRs 54935, 67768, 67896, 67787). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
expose generic storage factory primitives
The `StorageFactory` is central to the idea of a generic and decorated storage configuration for a given resource. This pull exposes the "standard" one from the generic apiserver to allow easier composition without duplication.
@kubernetes/sig-api-machinery-pr-reviews
/assign @yue9944882 @sttts
```release-note
NONE
```
Kubernetes-commit: 5abe4ccec8250e8a1401e7d612c2f4635eba3e11
Automatic merge from submit-queue (batch tested with PRs 54935, 67768, 67896, 67787). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
apiserver: unify handling of unspecified options in authn/z options
Use nil pointer for unspecified, not some kind of half zero-valued structs.
Kubernetes-commit: 5afa23f2c4d1c4501ed722e43dab4b1f64b0fddd
Automatic merge from submit-queue (batch tested with PRs 67362, 67256, 67809). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Dry run integration
Implement an integration test for dry-run. Also, this turns on the knob to allow dry-run requests, so let's be careful.
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 7b96c818df5da720f68ea8ba1a1fc44386ad21a8
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
delegated authz: add AlwaysAllowPaths to option struct (defaulting to /healthz)
Add `AlwaysAllowPaths` field to delegated authz. These http paths are excluded from the authz chain.
Prerequisite for https://github.com/kubernetes/kubernetes/pull/64149 and https://github.com/kubernetes/kubernetes/pull/67069.
```release-note
Added --authorization-always-allow-paths to components doing delegated authorization to exclude certain HTTP paths like /healthz from authorization.
```
Kubernetes-commit: 5ed26a348b017c3ece8ac468d15770ddf8b922ae
Automatic merge from submit-queue (batch tested with PRs 67707, 62420). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
svcacct: support jose.OpaqueSigner and push errors to token generator creation
```release-note
NONE
```
Kubernetes-commit: 166a1356e83b1c271c0d58aa4d3fb69bab713072
Automatic merge from submit-queue (batch tested with PRs 67576, 66936). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Support dry run in admission webhooks
**What this PR does / why we need it**:
Follow up to https://github.com/kubernetes/kubernetes/pull/66391
- [x] add DryRun to ```admission.k8s.io/v1beta1.AdmissionReview```
- [x] add DryRunnable to ```admissionregistration.k8s.io/v1beta1.(Valid|Mut)atingWebhookConfiguration```
- [x] add dry run support to (Valid|Mut)atingAdmissionWebhook
Includes all the api-changes outlined by https://github.com/kubernetes/community/pull/2387
/sig api-machinery
**Release note**:
```release-note
To address the possibility dry-run requests overwhelming admission webhooks that rely on side effects and a reconciliation mechanism, a new field is being added to admissionregistration.k8s.io/v1beta1.ValidatingWebhookConfiguration and admissionregistration.k8s.io/v1beta1.MutatingWebhookConfiguration so that webhooks can explicitly register as having dry-run support. If a dry-run request is made on a resource that triggers a non dry-run supporting webhook, the request will be completely rejected, with "400: Bad Request". Additionally, a new field is being added to the admission.k8s.io/v1beta1.AdmissionReview API object, exposing to webhooks whether or not the request being reviewed is a dry-run.
```
Kubernetes-commit: 5a16163c87fe2a90916a51b52771a668bcaf2a0d
Automatic merge from submit-queue (batch tested with PRs 67378, 67675, 67654). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add some unit tests in staging/src/k8s.io/apiserver/pkg/server/options
**What this PR does / why we need it**:
add some unit tests in staging/src/k8s.io/apiserver/pkg/server/options
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 5930ea30c196678864a29d5cc1bc3710048bef5b
Automatic merge from submit-queue (batch tested with PRs 67298, 67518, 67635, 67673). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix unstructured metadata accessors to respect omitempty semantics
Fixes#67541Fixes#48211Fixes#49075
Follow up of #67562
`ObjectMeta` has fields with `omitempty` json tags. This means that when the fields have zero values, they should not be persisted in the object.
Before this PR, some of the metadata accessors for unstructured objects did not respect these semantics i.e they would persist a field even if it had a zero value.
This PR updates the accessors so that the field is removed from the unstructured object map if it contains a zero value.
/sig api-machinery
/kind bug
/area custom-resources
/cc sttts liggitt yue9944882 roycaihw
/assign sttts liggitt
**Release note**:
```release-note
NONE
```
Kubernetes-commit: ce8a6285dfa0bcc96e6f8ef0e21ef40f958930c7
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Log real file's name and line
**What this PR does / why we need it**:
Have correct location of emission in the logs
**Release note**:
pkg/kubectl/util/logs & staging/src/k8s.io/apiserver/pkg/util/logs
use `glog.info(...)` but this function is not made to be wrapped because
the underlying mechanism use a fixed call trace length to determine
where the log has been emited.
This results is having `logs.go:49` in the logs which is in the body
of the wrapper function and thus useless.
Instead use `glog.infoDepth(1, ...)` which tells the underlying mechanism
to go back 1 more level in the call trace to determine where the log
has been emitted.
Kubernetes-commit: 70bc7f687707c7df4d182a14f2089b6e87992db1
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Cleanup apiserver errors
- delete unused methods (is this safe, or might they be used in another project outside kubernetes/kubernetes?)
- use the standard library `http.Error` function (functional change adds a newline)
```release-note
NONE
```
/sig api-machinery
/kind cleanup
Kubernetes-commit: 30e4f528ed30a70bdb0c14b5cfe49d00a78194c2
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Prevent resourceVersion updates for custom resources on no-op writes
Fixes partly https://github.com/kubernetes/kubernetes/issues/67541
For ObjectMeta pruning, we round trip by marshalling and unmarshalling. If the ObjectMeta contained any strings with `""` (or other fields with empty values) _and_ the respective fields are `omitempty`, those fields will be lost in the round trip process.
This makes ObjectMeta after the no-op write different from the one before the write.
Resource version is incremented every time data is written to etcd. Writes to etcd short-circuit if the bytes being written are identical to the bytes already present.
So this ends up incrementing the `resourceVersion` even on no-op writes. This PR updates the `BeforeUpdate` function such that omitempty fields have values set only if they are non-zero so that they produce an unstructured object that matches ObjectMeta omitempty semantics.
/sig api-machinery
/kind bug
/area custom-resources
/assign sttts liggitt
**Release note**:
```release-note
Prevent `resourceVersion` updates for custom resources on no-op writes.
```
Kubernetes-commit: 816f2a48689247c8344d911995b0f97c721ad481
Automatic merge from submit-queue (batch tested with PRs 55600, 67386). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update Annotations description about audit.Event
ref: https://github.com/kubernetes/kubernetes/pull/58679#discussion_r209375632
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
/assign @liggitt
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 4c5e6156525b96b72961b86ff5bd82c44ea0cd96
For ObjectMeta pruning, we round trip through marshalling and
unmarshalling. If the ObjectMeta contained any strings with "" (or other
fields with empty values) _and_ the respective fields are omitempty,
those fields will be lost in the round trip process.
This makes ObjectMeta after the no-op write different from the one
before the write.
Resource version is incremented every time data is written to etcd.
Writes to etcd short-circuit if the bytes being written are identical
to the bytes already present. So this ends up incrementing the
resourceVersion even on no-op writes.
The zero values are set in BeforeCreate and BeforeUpdate. This commit
updates BeforeUpdate such that zero values are only set when the
object does not have a zero value for the respective field.
Kubernetes-commit: d691748aa64376e4b90ae2c01aca14d1be2b442c
Automatic merge from submit-queue (batch tested with PRs 67041, 66948). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
50342: Establish '406 Not Acceptable' response for protobuf serializa…
…tion 'errNotMarshalable'
- Added metav1.Status() that enforces '406 Not Acceptable' response if
protobuf serialization is not fully supported for the API resource type.
- JSON and YAML serialization are supposed to be more completely baked
in, so serialization involving those, and general errors with seralizing
protobuf, will return '500 Internal Server Error'.
- If serialization failure occurs and original HTTP status code is
error, use the original status code, else use the serialization failure
status code.
- Write encoded API responses to intermediate buffer
- Use apimachinery/runtime::Encode() instead of
apimachinery/runtime/protocol::Encode() in
apiserver/endpoints/handlers/responsewriters/writers::SerializeObject()
- This allows for intended encoder error handling to fully work, facilitated by
apiserver/endpoints/handlers/responsewriters/status::ErrorToAPIResponse() before officially
writing to the http.ResponseWriter
- The specific part that wasn't working by ErrorToAPIResponse() was the
HTTP status code set. A direct call to
http.ResponseWriter::WriteHeader(statusCode) was made in
SerializeObject() with the original response status code, before
performing the encode. Once this
method is called, it can not again update the status code at a later
time, with say, an erro status code due to encode failure.
- Updated relevant apiserver unit test to reflect the new behavior
(TestWriteJSONDecodeError())
- Add build deps from make update for protobuf serializer
**What this PR does / why we need it**:
This PR fixes a bug that was blocking extensible error handling in the case that serializing response data fails, and implements a '406 Not Acceptable' status code response if protobuf marshal definitions are not implemented for an API resource type. See commit message for further details.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#50342
**Special notes for your reviewer**:
**Release note**:
```release-note
Fixed a bug that was blocking extensible error handling when serializing API responses error out. Previously, serialization failures always resulted in the status code of the original response being returned. Now, the following behavior occurs:
- If the serialization type is application/vnd.kubernetes.protobuf, and protobuf marshaling is not implemented for the requested API resource type, a '406 Not Acceptable is returned'.
- If the serialization type is 'application/json':
- If serialization fails, and the original status code was an failure (e.g. 4xx or 5xx), the original status code will be returned.
- If serialization fails, and the original status code was not a failure (e.g. 2xx), the status code of the serialization failure will be returned. By default, this is '500 Internal Server Error', because JSON serialization is our default, and not supposed to be implemented on a type-by-type basis.
```
Kubernetes-commit: 363e34124da0ee8c23064e51ed18ef280fba64a7
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update github.com/imdario/mergo to v0.3.5
**What this PR does / why we need it**:
Updates github.com/imdario/mergo library to v0.3.5. We were pinned because of a functionality change in the dependency, however, a new function was introduced with similar functionality to the old.
There is apparently some Debian packaging issues (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878254) because of this. I'm still not clear what those are, unless they are forcing the library to update, as opposed to using our `vendor/`.
That said, this will allow for some of those vendor conflicts to resolve for anyone else who is using client-go, so that's at least worthwhile.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
fixes#27543, fixes https://github.com/kubernetes/client-go/issues/431
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 6b4135267911b6c10ed536308d29d2a7c453eef6
Automatic merge from submit-queue (batch tested with PRs 67062, 67169, 67539, 67504, 66876). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix Proto Generator to not assign types to packages they don't belong to
Before this PR, types were assign to the first package they defined or being used. This result in wrong import statements. The change also fixes what #67158 tries to fix as the package path for protos will never empty.
An step forward from #67158
/assign @sttts @smarterclayton
Kubernetes-commit: 97215fe29dbef2e42a2531caa1786b3f02555d5d
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kube-{apiserver,ctrl-mgr}: unify into DeprecatedInsecureServingOptions
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Kubernetes-commit: b9544382baf4a0a366dd01e05e585f8556993754
Kubernetes Publisher
Zhenguo Niu
wojtekt
Dr. Stefan Schimanski
Jordan Liggitt
David Eads
Mehdy Bohlool
fisherxu
Chao Xu
hangaoshuai
jennybuckley
Nikhita Raghunath
Patrick Barker
