Automatic merge from submit-queue (batch tested with PRs 64741, 65484). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
enable etcdv3 client prometheus metics
**What this PR does / why we need it**:
Provide a way to enable etcdv3 client prometheus metics
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64740
**Special notes for your reviewer**:
**Release note**:
```release-note
Enable etcdv3 client prometheus metics
```
Kubernetes-commit: 1570d3851cce541f3e7a9eb9ae6f242255bf33e6
Automatic merge from submit-queue (batch tested with PRs 65593, 65678). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
simplify httplog.LogOf
**What this PR does / why we need it**:
LogOf returns the logger hiding in w， otherwise returns a passthroughLogger, and it never panics.
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 83a8e42d035a6371fb94c8cb13a9664cc0bcacf1
Automatic merge from submit-queue (batch tested with PRs 65518, 65624, 65380, 65390, 65586). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
cleanup: remove deadcode
/kind cleanup
remove deadcode
remove unnecessary conversion
```release-note
NONE
```
Kubernetes-commit: 16dbb95c51532e8a937a3ccb90223e8568b130c3
Automatic merge from submit-queue (batch tested with PRs 65319, 64513, 65474, 65601, 65634). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix apiserver metrics
Kubernetes-commit: 4f465fac5b395b2395b65c91035b3803e8631e59
Automatic merge from submit-queue (batch tested with PRs 65319, 64513, 65474, 65601, 65634). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use same rvParse in watchcache and move cacheStorage in separate dir
**What this PR does / why we need it**:
Use same rvParse in watchcache and move cacheStorage in separate dir.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 7b46e884ae439c53439ce55779b34530df6fa3db
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
apiserver: do not print feature gates for glog v=0
**What this PR does / why we need it**:
Demand verbosity level > 0 for glog Infof() calls when
setting feature gates in pkg/util/feature_gate.go.
Without this, regular calls to things like `kubeadm token generate` would also print `feature_gate.go:230] feature gates: &{map[]}`.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#953
**Special notes for your reviewer**:
i doubt there is a particular reason to not use verbosity level here?
is `v=1` sufficient here?
/area apiserver
/area kubeadm
/kind cleanup
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/cc @kubernetes/sig-api-machinery-pr-reviews
**Release note**:
```release-note
NONE
```
Kubernetes-commit: f0dcdd76229127df8364aa84bbe9f93313644f38
Automatic merge from submit-queue (batch tested with PRs 64575, 65120, 65463, 65434, 65522). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add support for Linux Abstract Socket Namespace for KMS provider plugin.
**What this PR does / why we need it**:
Currently, kube-apiserver and kms-plugin interact via a Unix Domain Socket. The current implementation, assumes that such a Domain Socket is supported via a socket file, which is in turn is supported via a volume shared between kube-apiserver and kms-plugin containers.
However, Linux supports Abstract Socket Namespace, where a socket does not need to be back-up by a file. In golang, such sockets are created by prefixing a socket's name with @.
Benefits of using Linux Abstract Socket Namespace:
1. Don't need to worry about possible collisions with existing files.
2. Simpler configuration of master's manifest - no need to setup a shared volume between kube-apiserver and kms-plugin containers.
3. Don't need to remember to unlink the socket when KMS Plugin shuts down.
4. Creates a possibility to run KMS Plugin without access to file system.
This PR adds the ability to define a KMS endpoint as: unix:///@kms-provider.sock
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 75c8b56dcbbaae8da9735ec3a41c4f8e36116ce6
Automatic merge from submit-queue (batch tested with PRs 65453, 65523, 65513, 65560). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Print type information when unknown watch error
Got
```
E0628 00:23:07.106285 1 watch.go:274] unable to encode watch object: expected pointer, but got invalid kind
```
on a production system and had no way to debug what type was being sent.
@liggitt re: the message I sent you
Kubernetes-commit: c4ca6638c397e90a486d363a10b3e6411fa6a0e4
Got
```
E0628 00:23:07.106285 1 watch.go:274] unable to encode watch object: expected pointer, but got invalid kind
```
on a production system and had no way to debug what type was being sent.
Kubernetes-commit: 307849baef076d8ee61a3b9649f9260a765f7ac0
Automatic merge from submit-queue (batch tested with PRs 65404, 65323, 65468). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Collapse the list and watch resource version parse
**What this PR does / why we need it**:
Collapse the list and watch resource version parse, as discuss in [#64513](https://github.com/kubernetes/kubernetes/pull/64513#issuecomment-399380988)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: f9a1cb9b63f155d53102d33993cb73357d84c981
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add user-agent to audit-logging
**What this PR does / why we need it**:
Add User-Agent to audit event.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64791
**Special notes for your reviewer**:
**Release note**:
```release-note
Add user-agent to audit-logging.
```
Kubernetes-commit: 1f4f0123edc973d06ceb15d517eb4c555b944427
Automatic merge from submit-queue (batch tested with PRs 64122, 64936, 65288, 65383). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update to rules_go 0.12.1 and gazelle 0.12.0 and perform related cleanups
**What this PR does / why we need it**: my initial intent was to simply update to rules_go 0.12.1 and gazelle 0.12.0.
A few internal changes / deprecations meant that I finally needed to clean up some technical debt. This also fixes#64122.
I've attempted to keep the steps as separate commits to make it easier to review:
1. Disable gazelle proto rule generation; legacy proto rules are deprecated, and we don't (currently) build protos at build time anyway, instead generating them with `hack/update-generated-protobuf.sh` and then checking them in. We can revisit this in the future if we'd like.
2. Remove the legacy `go_default_library_protos` filegroups using [buildozer](https://github.com/bazelbuild/buildtools/tree/master/buildozer). We don't use these, anyway.
3. Update the rules_go bazel workspace dependency to 0.12.1.
4. Vendor gazelle 0.12.0 and update BUILD files with `hack/update-bazel.sh`. This causes a lot of diffs, because `select()`s are no longer used in `srcs` attributes, external tests are folded into non-external tests, and vendored targets get an `importmap` attribute.
5. Set `gazelle:prefix` on `staging/src/BUILD` to make gazelle treat these correctly(ish). This allows us to remove the sed rewrite hack in `hack/update-bazel.sh`.
6. Explicitly set `# gazelle:importmap_prefix k8s.io/kubernetes/vendor` on `vendor/`, so that all vendored dependencies get the right importmap. gazelle 0.12.0 uses the bazel workspace name + `vendor/` as a prefix, which doesn't work with native go. Newer gazelle will use the go prefix (https://github.com/bazelbuild/bazel-gazelle/pull/207), but it's not released yet. Setting this correctly now also fixes later `BUILD` churn.
7. Re-run `hack/update-bazel.sh`. This causes a bunch of diffs, since anything under `staging/src` now uses the `staging/src/` path instead of `vendor/`. (Both would work for bazel, but gazelle uses the former, since `vendor/` uses symlinks.) Also `importmap`s under `vendor/` are fixed.
8. Reformat a few files (using [buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier)) to make later diffs easier to read.
9. Rework the `go_genrule` rules to use the new `go_genrule` from https://github.com/kubernetes/repo-infra/pull/72, which is more bazely, since it uses the rules_go `go_path` rule instead of lots of shell.
10. Remove the deprecated `go_prefix` rule from the root BUILD.bazel file.
11. Set `# gazelle:importmap_prefix k8s.io/kubernetes/vendor` on `staging/src` as well, which ensures that these repos are treated as vendored dependencies. (It's basically the bazel-y way of doing the `vendor/k8s.io` symlinks.)
12. Run `hack/update-bazel.sh` one last time to fix all of the `importmap`s under `staging/src`.
Note re: point 6 above - we're pretty much ignoring the `vendor/k8s.io` symlinks entirely now under bazel. Using the `gazelle:prefix` directive ensures these get mapped into the right go importpath, and the `go_path` rule installs these correctly now too.
**Special notes for your reviewer**: this should not be submitted before https://github.com/kubernetes/repo-infra/pull/72, obviously.
**Release note**:
```release-note
NONE
```
/assign @BenTheElder @fejta @thockin
cc @cblecker @jayconrod
Kubernetes-commit: 1ad1c8c7f80d99b9625924b2102a04a555162bfb
Automatic merge from submit-queue (batch tested with PRs 61330, 64793, 64675, 65059, 65368). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
legacy api endpoints only support v1 ever
The legacy API endpoint should only ever have a v1. This removes flexibility we don't need or want.
@kubernetes/sig-api-machinery-pr-reviews
@sttts
```release-note
NONE
```
Kubernetes-commit: 3a266a1b3f0f3d8beec2a093fd91c6ad21a1c435
Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
apiserver: fix typo introduced in #57366
fix typo introduced in #57366
```release-note
NONE
```
Kubernetes-commit: 1308d9a3800e37a1fe2f82ee17edf452c9115faa
Automatic merge from submit-queue (batch tested with PRs 65152, 65199, 65179, 64598, 65216). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix MaxAge default audit log option
Fix typo in flag default.
```release-note
NONE
```
Kubernetes-commit: 01bc75c291e2ef2a812ff41a65f0de1245c60e55
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
etcd: reuse leases for keys in a time window
Reuse leases for keys in a time window, to reduce the overhead to etcd
caused by using massive number of leases
Fixes#47532
```release-note
NONE
```
Kubernetes-commit: 8ba32978b78cc6306181f7282e9721282a932571
Automatic merge from submit-queue (batch tested with PRs 65116, 61718, 65140, 65128, 65099). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
apiserver: add context to authn/authz kubeconfig errors
Before this the user only saw messages about in-cluster config, but didn't know which.
Kubernetes-commit: d7db405b307ecac66e58f205b64f20d080525d40
Automatic merge from submit-queue (batch tested with PRs 64140, 64898, 65022, 65037, 65027). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use actual etcd client for /healthz/etcd checks
* avoids redialing etcd on every health check (which makes slow DNS a false-positive healthz failure)
* ensures etcd TLS setup is correct (errors verifying the etcd API or sending client credentials manifest as healthz failures)
* ensures the etcd cluster is actually responsive
fixes#64909
```release-note
Etcd health checks by the apiserver now ensure the apiserver can connect to and exercise the etcd API
```
Kubernetes-commit: 9d97913e75059a2ab7ed5488c9dd42bff2a23836
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Quiet verbose apiserver logs
In our production environments, these three lines accounted for ~10 logged lines per second at --v=2
Something that verbose per-request should be at a higher verbosity log level
/cc @eparis
/sig api-machinery
Kubernetes-commit: 83ad4d9e96e77a87434a43e475a653c3c64f48ff
Automatic merge from submit-queue (batch tested with PRs 64285, 63660, 63661, 63662, 64883). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix up legacy printer table adapter
As server-side printing is more widespread, downstream consumers will start leaning on this adapter to transform client-side printers to server-side printers.
This PR:
* Set the name format correctly on column 0
* Tolerates case-differences in the column 0 name (many old printers used NAME)
* Set ListMeta continue/resourceVersion/selfLink correctly when adapting a legacy printer
/cc @smarterclayton
```release-note
NONE
```
Kubernetes-commit: 56a2d2f5786c9746b9dd349340eb1269043df50d
Automatic merge from submit-queue (batch tested with PRs 65256, 64236, 64919, 64879, 57932). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix CRD OpenAPI schema
fixes#65243
depends on https://github.com/kubernetes/kube-openapi/pull/84
without this PR, kubectl complains about creating this CRD with a validation schema (which worked in 1.10):
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: resources.mygroup.example.com
spec:
group: mygroup.example.com
version: v1alpha1
scope: Namespaced
names:
plural: resources
singular: resource
kind: Kind
listKind: KindList
validation:
openAPIV3Schema:
properties:
spec:
type: array
items:
type: number
```
> error: error validating "/Users/jliggitt/projects/snippets/crd/crd.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.items): unknown field "type" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrArray, ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.items): missing required field "Schema" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrArray, ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.items): missing required field "JSONSchemas" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrArray]; if you choose to ignore these errors, turn validation off with --validate=false
that is because the types used to serialize JSONSchema require custom marshaling/unmarshaling, and the OpenAPI generator was not informed of that, so it produced this:
```json
{
"io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrArray": {
"description": "JSONSchemaPropsOrArray represents a value that can either be a JSONSchemaProps or an array of JSONSchemaProps. Mainly here for serialization purposes.",
"required": [
"Schema",
"JSONSchemas"
],
"properties": {
"JSONSchemas": {
"type": "array",
"items": {
"$ref": "#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps"
}
},
"Schema": {
"$ref": "#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps"
}
}
}
}
```
OpenAPI isn't able to represent oneOf/anyOf types correctly currently. Until it can, we definitely shouldn't publish a schema containing required fields which aren't even part of the JSON serialization. This PR implements custom openapi type functions, which omit the properties/required/schema attributes for four specific JSONSchema types. This allows kubectl to continue creating these objects without complaining.
/sig api-machinery
/assign @sttts
```release-note
fixed incorrect OpenAPI schema for CustomResourceDefinition objects
```
Kubernetes-commit: ed6c8b7326bd1a1b845719f4bfb302073a18f03f
Automatic merge from submit-queue (batch tested with PRs 63580, 63744, 64541, 64502, 64100). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Improve unit tests for InstallPathHandler()
When adding InstallPathHandler it was suggested to follow-up with an improvement to the unit tests. deads2k suggested that the testing for InstallPathHandler() follow closer to what is already implemented for InstallHandler().
**What this PR does / why we need it**:
Increases the testing around new functionality InstallPathHandler
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 27c327cd33db8c91f1c6ad6c701b0457c780bce2
Automatic merge from submit-queue (batch tested with PRs 64252, 64307, 64163, 64378, 64179). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Modify LoopbackHostPort() so it returns an IPv6 Loopback address when given [::]
Currently when LoopbackHostPort() is called with 0.0.0.0 and [::] it returns the first loopback
address returned from net.InterfaceAddrs() which is typically 127.0.0.1 (golang does not
specify an order that interfaces are returned). It would be more appropriate if when calling
LoopbackHostPort() with [::] that an IPv6 loopback address is returned, this prevents some cert.
generation failures.
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Kubernetes-commit: 6559b98f6410f1419872da2d31e360e2fc2442df
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Make kubernetes json serializer case sensitive
This PR imported the latest jsoniterator library so that case sensitivity during unmarhsaling is optional. The PR also set Kubernetes json serializer to be case sensitive.
Kubernetes json serializer had been case sensitive for 1.1-1.7 as we were using ugorji. This PR restores the behavior.
Fix#64612.
```release-notes
Kubernetes json deserializer is now case-sensitive as it was before 1.8.
If your config files contains fields with wrong case, the config files will be now invalid.
```
Kubernetes-commit: a2de1398f829ef38d645579160bdd6bfec8384d3
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix field removal in mutating admission webhooks
A mutating admission controller webhook doesn't remove object fields
when instructed to.
E.g. when the JSON patch
```
[
{"op": "remove", "path": "/spec/containers/0/resources/limits/fpga-arria10"},
{"op": "add", "path": "/spec/containers/0/resources/limits/fpga-interface-id-524abcf", "value": 1}
]
```
is applied to this pod
```
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
restartPolicy: Never
containers:
-
name: test-pod-container
image: ubuntu:bionic
imagePullPolicy: IfNotPresent
command: [ "ls", "-l", "/" ]
resources:
limits:
fpga-arria10: 1
```
in order to replace the resource name "fpga-arria10" with something understandable
by the device plugin the resulting pod spec still contains the old field plus
a new one. The resulting pod looks like
```
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
restartPolicy: Never
containers:
-
name: test-pod-container
image: ubuntu:bionic
imagePullPolicy: IfNotPresent
command: [ "ls", "-l", "/" ]
resources:
limits:
fpga-arria10: 1
fpga-interface-id-524abcf: 1
```
The patch unmarshals patched JSON into a new empty object instead of
existing one. Otherwise JSON unmarshaling reuses existing maps, keeping
existing entries as specified in the "encoding/json" standard package.
**Release note**:
```release-note
Webhooks for the mutating admission controller now support "remove" operation.
```
Kubernetes-commit: d70e7844ba8e24fa68a6030c400e5a06ee51a5c6
Mikhail Mazurskiy
Kubernetes Publisher
Zhonghu Xu
Dr. Stefan Schimanski
wojtekt
Lubomir I. Ivanov
Clayton Coleman
fisherxu
Cong Ding
David Eads
Jordan Liggitt
Tim Allclair
Chao Xu
Jeff Grafton