kubernetes
/
website
mirror of https://github.com/kubernetes/website.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #41165 from kubernetes/dev-1.28 

Official 1.28 Release Docs
This commit is contained in:
Rey Lejano 2023-08-15 12:11:28 -07:00 committed by GitHub
parent beb7d1c9cd e05d2b72db
commit 0180cc4bf0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
86 changed files with 9551 additions and 24002 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30048
api-ref-assets/api/swagger.json
View File

File diff suppressed because it is too large Load Diff

15
api-ref-assets/config/fields.yaml
View File

@ -88,6 +88,7 @@
- fields:
- nominatedNodeName
- hostIP
- hostIPs
- startTime
- phase
- message
@ -99,6 +100,7 @@
- initContainerStatuses
- containerStatuses
- ephemeralContainerStatuses
- resourceClaimStatuses
- resize
- definition: io.k8s.api.core.v1.Container
@ -137,6 +139,7 @@
- livenessProbe
- readinessProbe
- startupProbe
- restartPolicy
- name: Security Context
fields:
- securityContext
@ -228,6 +231,7 @@
fields:
- terminationMessagePath
- terminationMessagePolicy
- restartPolicy
- name: Debugging
fields:
- stdin
@ -393,9 +397,14 @@
fields:
- selector
- manualSelector
- name: Alpha level
- name: Beta level
fields:
- podFailurePolicy
- name: Alpha level
fields:
- backoffLimitPerIndex
- maxFailedIndexes
- podReplacementPolicy
- definition: io.k8s.api.batch.v1.JobStatus
field_categories:
@ -411,6 +420,10 @@
- name: Beta level
fields:
- ready
- name: Alpha level
fields:
- failedIndexes
- terminating
- definition: io.k8s.api.batch.v1.CronJobSpec
field_categories:

7
api-ref-assets/config/toc.yaml
View File

@ -153,7 +153,7 @@ parts:
version: v1alpha1
- name: SelfSubjectReview
group: authentication.k8s.io
version: v1beta1
version: v1
- name: Authorization Resources
chapters:
- name: LocalSubjectAccessReview
@ -168,9 +168,6 @@ parts:
- name: SubjectAccessReview
group: authorization.k8s.io
version: v1
- name: SelfSubjectReview
group: authentication.k8s.io
version: v1alpha1
- name: ClusterRole
group: rbac.authorization.k8s.io
version: v1
@ -218,7 +215,7 @@ parts:
version: v1
- name: ValidatingAdmissionPolicy
group: admissionregistration.k8s.io
version: v1alpha1
version: v1beta1
otherDefinitions:
- ValidatingAdmissionPolicyList
- ValidatingAdmissionPolicyBinding

7
content/en/blog/_posts/2022-09-14-pod-has-network-condition.md
View File

@ -23,6 +23,13 @@ per container characteristics like image size or payload) can utilize the
the `PodHasNetwork` condition to optimize the set of actions performed when pods
repeatedly fail to come up.
### Updates for Kubernetes 1.28
The `PodHasNetwork` condition has been renamed to `PodReadyToStartContainers`.
Alongside that change, the feature gate `PodHasNetworkCondition` has been replaced by
`PodReadyToStartContainersCondition`. You need to set `PodReadyToStartContainersCondition`
to true in order to use the new feature in v1.28.0 and later.
### How is this different from the existing Initialized condition reported for pods?
The kubelet sets the status of the existing `Initialized` condition reported in

84
content/en/docs/concepts/architecture/mixed-version-proxy.md Normal file
View File

@ -0,0 +1,84 @@
---
reviewers:
- jpbetz
title: Mixed Version Proxy
content_type: concept
weight: 220
---
<!-- overview -->
{{< feature-state state="alpha" for_k8s_version="v1.28" >}}
Kubernetes {{< skew currentVersion >}} includes an alpha feature that lets a
{{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}}
proxy a resource requests to other _peer_ API servers. This is useful when there are multiple
API servers running different versions of Kubernetes in one cluster (for example, during a long-lived
rollout to a new release of Kubernetes).
This enables cluster administrators to configure highly available clusters that can be upgraded
more safely, by directing resource requests (made during the upgrade) to the correct kube-apiserver.
That proxying prevents users from seeing unexpected 404 Not Found errors that stem
from the upgrade process.
This mechanism is called the _Mixed Version Proxy_.
## Enabling the Mixed Version Proxy
Ensure that `UnknownVersionInteroperabilityProxy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
is enabled when you start the {{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}}:
```shell
kube-apiserver \
--feature-gates=UnknownVersionInteroperabilityProxy=true \
# required command line arguments for this feature
--peer-ca-file=<path to kube-apiserver CA cert>
--proxy-client-cert-file=<path to aggregator proxy cert>,
--proxy-client-key-file=<path to aggregator proxy key>,
--requestheader-client-ca-file=<path to aggregator CA cert>,
# requestheader-allowed-names can be set to blank to allow any Common Name
--requestheader-allowed-names=<valid Common Names to verify proxy client cert against>,
# optional flags for this feature
--peer-advertise-ip=`IP of this kube-apiserver that should be used by peers to proxy requests`
--peer-advertise-port=`port of this kube-apiserver that should be used by peers to proxy requests`
# …and other flags as usual
```
### Proxy transport and authentication between API servers {#transport-and-authn}
* The source kube-apiserver reuses the [existing APIserver client authentication flags](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#kubernetes-apiserver-client-authentication) `--proxy-client-cert-file` and `--proxy-client-key-file` to present its identity that will be verified by its peer (the destination kube-apiserver). The destination API server verifies that peer connection based on the configuration you specify using the `--requestheader-client-ca-file` command line argument.
* To authenticate the destination server's serving certs, you must configure a certificate authority bundle by specifying the `--peer-ca-file` command line argument to the **source** API server.
### Configuration for peer API server connectivity
To set the network location of a kube-apiserver that peers will use to proxy requests, use the
`--peer-advertise-ip` and `--peer-advertise-port` command line arguments to kube-apiserver or specify
these fields in the API server configuration file.
If these flags are unspecified, peers will use the value from either `--advertise-address` or
`--bind-address` command line argument to the kube-apiserver. If those too, are unset, the host's default interface is used.
## Mixed version proxying
When you enable mixed version proxying, the [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)
loads a special filter that does the following:
* When a resource request reaches an API server that cannot serve that API (either because it is at a version pre-dating the introduction of the API or the API is turned off on the API server) the API server attempts to send the request to a peer API server that can serve the requested API. It does so by identifying API groups / versions / resources that the local server doesn't recognise, and tries to proxy those requests to a peer API server that is capable of handling the request.
* If the peer API server fails to respond, the _source_ API server responds with 503("Service Unavailable") error.
### How it works under the hood
When an API Server receives a resource request, it first checks which API servers can serve the requested resource. This check happens using the internal [`StorageVersion` API].
* If the resource is known to the API server that received the request (ex: `GET /api/v1/pods/some-pod`), the request is handled locally.
* If there is no internal `StorageVersion` object found for the requested resource (ex: `GET /my-api/v1/my-resource`) and the configured APIService specifies proxying to an extension API server, that proxying happens following the usual
[flow](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) for
extension APIs.
* If a valid internal `StorageVersion` object is found for the requested resource (ex: `GET /batch/v1/jobs`) and the API server trying to handle the request (the _handling API server_) has the `batch` API disabled, then the _handling API server_fetches the peer API servers that do serve the relevant API group / version / resource (`api/v1/batch` in this case) using the information in the fetched `StorageVersion` object. The _handling API server_ then proxies the request to one of the matching peer kube-apiservers that are aware of the requested resource.
* If there is no peer known for that API group / version / resource, the handling API server passes the request to its own handler chain which should eventually return a 404("Not Found") response.
* If the handling API server has identified and selected a peer API server, but that peer fails
to respond (for reasons such as network connectivity issues, or a data race between the request
being received and a controller registering the peer's info into the control plane), then the handling
API server responds with a 503 (“Service Unavailable”) error.

47
content/en/docs/concepts/architecture/nodes.md
View File

@ -571,9 +571,9 @@ the feature is Beta and is enabled by default.
Metrics `graceful_shutdown_start_time_seconds` and `graceful_shutdown_end_time_seconds`
are emitted under the kubelet subsystem to monitor node shutdowns.
## Non Graceful node shutdown {#non-graceful-node-shutdown}
## Non-graceful node shutdown handling {#non-graceful-node-shutdown}
{{< feature-state state="beta" for_k8s_version="v1.26" >}}
{{< feature-state state="stable" for_k8s_version="v1.28" >}}
A node shutdown action may not be detected by kubelet's Node Shutdown Manager,
either because the command does not trigger the inhibitor locks mechanism used by
@ -617,11 +617,7 @@ During a non-graceful shutdown, Pods are terminated in the two phases:
## Swap memory management {#swap-memory}
{{< feature-state state="alpha" for_k8s_version="v1.22" >}}
Prior to Kubernetes 1.22, nodes did not support the use of swap memory, and a
kubelet would by default fail to start if swap was detected on a node. In 1.22
onwards, swap memory support can be enabled on a per-node basis.
{{< feature-state state="beta" for_k8s_version="v1.28" >}}
To enable swap on a node, the `NodeSwap` feature gate must be enabled on
the kubelet, and the `--fail-swap-on` command line flag or `failSwapOn`
@ -638,29 +634,40 @@ specify how a node will use swap memory. For example,
```yaml
memorySwap:
swapBehavior: LimitedSwap
swapBehavior: UnlimitedSwap
```
The available configuration options for `swapBehavior` are:
- `LimitedSwap`: Kubernetes workloads are limited in how much swap they can
use. Workloads on the node not managed by Kubernetes can still swap.
- `UnlimitedSwap`: Kubernetes workloads can use as much swap memory as they
- `UnlimitedSwap` (default): Kubernetes workloads can use as much swap memory as they
request, up to the system limit.
- `LimitedSwap`: The utilization of swap memory by Kubernetes workloads is subject to limitations. Only Pods of Burstable QoS are permitted to employ swap.
If configuration for `memorySwap` is not specified and the feature gate is
enabled, by default the kubelet will apply the same behaviour as the
`LimitedSwap` setting.
`UnlimitedSwap` setting.
The behaviour of the `LimitedSwap` setting depends if the node is running with
v1 or v2 of control groups (also known as "cgroups"):
With `LimitedSwap`, Pods that do not fall under the Burstable QoS classification (i.e.
`BestEffort`/`Guaranteed` Qos Pods) are prohibited from utilizing swap memory.
To maintain the aforementioned security and node
health guarantees, these Pods are not permitted to use swap memory when `LimitedSwap` is
in effect.
- **cgroupsv1:** Kubernetes workloads can use any combination of memory and
swap, up to the pod's memory limit, if set.
- **cgroupsv2:** Kubernetes workloads cannot use swap memory.
Prior to detailing the calculation of the swap limit, it is necessary to define the following terms:
* `nodeTotalMemory`: The total amount of physical memory available on the node.
* `totalPodsSwapAvailable`: The total amount of swap memory on the node that is available for use by Pods (some swap memory may be reserved for system use).
* `containerMemoryRequest`: The container's memory request.
Swap limitation is configured as:
`(containerMemoryRequest / nodeTotalMemory) * totalPodsSwapAvailable`.
It is important to note that, for containers within Burstable QoS Pods, it is possible to
opt-out of swap usage by specifying memory requests that are equal to memory limits.
Containers configured in this manner will not have access to swap memory.
Swap is supported only with **cgroup v2**, cgroup v1 is not supported.
For more information, and to assist with testing and provide feedback, please
see [KEP-2400](https://github.com/kubernetes/enhancements/issues/2400) and its
see the blog-post about [Kubernetes 1.28: NodeSwap graduates to Beta1](/blog/2023/07/18/swap-beta1-1.28-2023/),
[KEP-2400](https://github.com/kubernetes/enhancements/issues/4128) and its
[design proposal](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md).
## {{% heading "whatsnext" %}}

2
content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md
View File

@ -209,7 +209,7 @@ Aggregated APIs offer more advanced API features and customization of other feat
| Feature | Description | CRDs | Aggregated API |
| ------- | ----------- | ---- | -------------- |
| Validation | Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time. | Yes. Most validation can be specified in the CRD using [OpenAPI v3.0 validation](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation). Any other validations supported by addition of a [Validating Webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook-alpha-in-1-8-beta-in-1-9). | Yes, arbitrary validation checks |
| Validation | Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time. | Yes. Most validation can be specified in the CRD using [OpenAPI v3.0 validation](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation). [CRDValidationRatcheting](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-ratcheting) feature gate allows failing validations specified using OpenAPI also can be ignored if the failing part of the resource was unchanged. Any other validations supported by addition of a [Validating Webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook-alpha-in-1-8-beta-in-1-9). | Yes, arbitrary validation checks |
| Defaulting | See above | Yes, either via [OpenAPI v3.0 validation](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#defaulting) `default` keyword (GA in 1.17), or via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) (though this will not be run when reading from etcd for old objects). | Yes |
| Multi-versioning | Allows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions. | [Yes](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning) | Yes |
| Custom Storage | If you need storage with a different performance mode (for example, a time-series database instead of key-value store) or isolation for security (for example, encryption of sensitive information, etc.) | No | Yes |

32
content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md
View File

@ -147,6 +147,22 @@ The general workflow of a device plugin includes the following steps:
runtime configurations for accessing the allocated devices. The kubelet passes this information
to the container runtime.
An `AllocateResponse` contains zero or more `ContainerAllocateResponse` objects. In these, the
device plugin defines modifications that must be made to a container's definition to provide
access to the device. These modifications include:
* annotations
* device nodes
* environment variables
* mounts
* fully-qualified CDI device names
{{< note >}}
The processing of the fully-qualified CDI device names by the Device Manager requires
the `DevicePluginCDIDevices` feature gate to be enabled. This was added as an alpha feature in
v1.28.
{{< /note >}}
### Handling kubelet restarts
A device plugin is expected to detect kubelet restarts and re-register itself with the new
@ -195,7 +211,7 @@ of the device allocations during the upgrade.
## Monitoring device plugin resources
{{< feature-state for_k8s_version="v1.15" state="beta" >}}
{{< feature-state for_k8s_version="v1.28" state="stable" >}}
In order to monitor resources provided by device plugins, monitoring agents need to be able to
discover the set of devices that are in-use on the node and obtain metadata to describe which
@ -312,7 +328,7 @@ below:
### `GetAllocatableResources` gRPC endpoint {#grpc-endpoint-getallocatableresources}
{{< feature-state state="beta" for_k8s_version="v1.23" >}}
{{< feature-state state="stable" for_k8s_version="v1.28" >}}
GetAllocatableResources provides information on resources initially available on the worker node.
It provides more information than kubelet exports to APIServer.
@ -338,16 +354,6 @@ message AllocatableResourcesResponse {
}
```
Starting from Kubernetes v1.23, the `GetAllocatableResources` is enabled by default.
You can disable it by turning off the `KubeletPodResourcesGetAllocatable`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
Preceding Kubernetes v1.23, to enable this feature `kubelet` must be started with the following flag:
```
--feature-gates=KubeletPodResourcesGetAllocatable=true
```
`ContainerDevices` do expose the topology information declaring to which NUMA cells the device is
affine. The NUMA cells are identified using a opaque integer ID, which value is consistent to
what device plugins report
@ -381,8 +387,6 @@ Support for the `PodResourcesLister service` requires `KubeletPodResources`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled.
It is enabled by default starting with Kubernetes 1.15 and is v1 since Kubernetes 1.20.
### `Get` gRPC endpoint {#grpc-endpoint-get}
{{< feature-state state="alpha" for_k8s_version="v1.27" >}}

45
content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md
View File

@ -11,7 +11,7 @@ weight: 65
{{< feature-state for_k8s_version="v1.27" state="alpha" >}}
Dynamic resource allocation is a new API for requesting and sharing resources
Dynamic resource allocation is an API for requesting and sharing resources
between pods and containers inside a pod. It is a generalization of the
persistent volumes API for generic resources. Third-party resource drivers are
responsible for tracking and allocating resources. Different kinds of
@ -32,7 +32,7 @@ check the documentation for that version of Kubernetes.
## API
The `resource.k8s.io/v1alpha2` {{< glossary_tooltip text="API group"
term_id="api-group" >}} provides four new types:
term_id="api-group" >}} provides four types:
ResourceClass
: Defines which resource driver handles a certain kind of
@ -61,7 +61,7 @@ typically using the type defined by a {{< glossary_tooltip
term_id="CustomResourceDefinition" text="CRD" >}} that was created when
installing a resource driver.
The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a new
The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a
`resourceClaims` field. Entries in that list reference either a ResourceClaim
or a ResourceClaimTemplate. When referencing a ResourceClaim, all Pods using
this PodSpec (for example, inside a Deployment or StatefulSet) share the same
@ -168,14 +168,39 @@ The kubelet provides a gRPC service to enable discovery of dynamic resources of
running Pods. For more information on the gRPC endpoints, see the
[resource allocation reporting](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources).
## Limitations
## Pre-scheduled Pods
The scheduler plugin must be involved in scheduling Pods which use
ResourceClaims. Bypassing the scheduler by setting the `nodeName` field leads
to Pods that the kubelet refuses to start because the ResourceClaims are not
reserved or not even allocated. It may be possible to [remove this
limitation](https://github.com/kubernetes/kubernetes/issues/114005) in the
future.
When you - or another API client - create a Pod with `spec.nodeName` already set, the scheduler gets bypassed.
If some ResourceClaim needed by that Pod does not exist yet, is not allocated
or not reserved for the Pod, then the kubelet will fail to run the Pod and
re-check periodically because those requirements might still get fulfilled
later.
Such a situation can also arise when support for dynamic resource allocation
was not enabled in the scheduler at the time when the Pod got scheduled
(version skew, configuration, feature gate, etc.). kube-controller-manager
detects this and tries to make the Pod runnable by triggering allocation and/or
reserving the required ResourceClaims.
However, it is better to avoid this because a Pod that is assigned to a node
blocks normal resources (RAM, CPU) that then cannot be used for other Pods
while the Pod is stuck. To make a Pod run on a specific node while still going
through the normal scheduling flow, create the Pod with a node selector that
exactly matches the desired node:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-with-cats
spec:
nodeSelector:
kubernetes.io/hostname: name-of-the-intended-node
...
```
You may also be able to mutate the incoming Pod, at admission time, to unset
the `.spec.nodeName` field and to use a node selector instead.
## Enabling dynamic resource allocation

4
content/en/docs/concepts/services-networking/dns-pod-service.md
View File

@ -292,7 +292,7 @@ Below are the properties a user can specify in the `dnsConfig` field:
This property is optional. When specified, the provided list will be merged
into the base search domain names generated from the chosen DNS policy.
Duplicate domain names are removed.
Kubernetes allows for at most 6 search domains.
Kubernetes allows up to 32 search domains.
- `options`: an optional list of objects where each object may have a `name`
property (required) and a `value` property (optional). The contents in this
property will be merged to the options generated from the specified DNS policy.
@ -325,7 +325,7 @@ options ndots:5
## DNS search domain list limits
{{< feature-state for_k8s_version="1.26" state="beta" >}}
{{< feature-state for_k8s_version="1.28" state="stable" >}}
Kubernetes itself does not limit the DNS Config until the length of the search
domain list exceeds 32 or the total length of all search domains exceeds 2048.

2
content/en/docs/concepts/services-networking/service.md
View File

@ -517,7 +517,7 @@ spec:
#### Reserve Nodeport Ranges to avoid collisions when port assigning
{{< feature-state for_k8s_version="v1.27" state="alpha" >}}
{{< feature-state for_k8s_version="v1.28" state="beta" >}}
The policy for assigning ports to NodePort services applies to both the auto-assignment and
the manual assignment scenarios. When a user wants to create a NodePort service that

39
content/en/docs/concepts/storage/persistent-volumes.md
View File

@ -371,7 +371,7 @@ the following types of volumes:
* {{< glossary_tooltip text="csi" term_id="csi" >}}
* flexVolume (deprecated)
* gcePersistentDisk (deprecated)
* rbd
* rbd (deprecated)
* portworxVolume (deprecated)
You can only expand a PVC if its storage class's `allowVolumeExpansion` field is set to true.
@ -488,7 +488,7 @@ value you previously tried.
This is useful if expansion to a higher value did not succeed because of capacity constraint.
If that has happened, or you suspect that it might have, you can retry expansion by specifying a
size that is within the capacity limits of underlying storage provider. You can monitor status of
resize operation by watching `.status.resizeStatus` and events on the PVC.
resize operation by watching `.status.allocatedResourceStatuses` and events on the PVC.
Note that,
although you can specify a lower amount of storage than what was requested previously,
@ -501,7 +501,6 @@ Kubernetes does not support shrinking a PVC to less than its current size.
PersistentVolume types are implemented as plugins. Kubernetes currently supports the following plugins:
* [`cephfs`](/docs/concepts/storage/volumes/#cephfs) - CephFS volume
* [`csi`](/docs/concepts/storage/volumes/#csi) - Container Storage Interface (CSI)
* [`fc`](/docs/concepts/storage/volumes/#fc) - Fibre Channel (FC) storage
* [`hostPath`](/docs/concepts/storage/volumes/#hostpath) - HostPath volume
@ -511,7 +510,6 @@ PersistentVolume types are implemented as plugins. Kubernetes currently supports
* [`local`](/docs/concepts/storage/volumes/#local) - local storage devices
mounted on nodes.
* [`nfs`](/docs/concepts/storage/volumes/#nfs) - Network File System (NFS) storage
* [`rbd`](/docs/concepts/storage/volumes/#rbd) - Rados Block Device (RBD) volume
The following types of PersistentVolume are deprecated.
This means that support is still available but will be removed in a future Kubernetes release.
@ -526,6 +524,10 @@ This means that support is still available but will be removed in a future Kuber
(**deprecated** in v1.25)
* [`vsphereVolume`](/docs/concepts/storage/volumes/#vspherevolume) - vSphere VMDK volume
(**deprecated** in v1.19)
* [`cephfs`](/docs/concepts/storage/volumes/#cephfs) - CephFS volume
(**deprecated** in v1.28)
* [`rbd`](/docs/concepts/storage/volumes/#rbd) - Rados Block Device (RBD) volume
(**deprecated** in v1.28)
Older versions of Kubernetes also supported the following in-tree PersistentVolume types:
@ -715,11 +717,12 @@ Not all Persistent Volume types support mount options.
The following volume types support mount options:
* `azureFile`
* `cephfs`
* `gcePersistentDisk`
* `cephfs` (**deprecated** in v1.28)
* `cinder` (**deprecated** in v1.18)
* `gcePersistentDisk` (**deprecated** in v1.28)
* `iscsi`
* `nfs`
* `rbd`
* `rbd` (**deprecated** in v1.28)
* `vsphereVolume`
Mount options are not validated. If a mount option is invalid, the mount fails.
@ -754,6 +757,20 @@ A volume will be in one of the following phases:
The CLI will show the name of the PVC bound to the PV.
#### Phase transition timestamp
{{< feature-state for_k8s_version="v1.28" state="alpha" >}}
Persistent volume status contains `lastPhaseTransitionTime` field which holds
a timestamp of when the volume last transitioned its phase. For newly created
volumes the phase is set to "Pending" and `lastPhaseTransitionTime` is set to
current time.
{{< note >}}
You need to enable the `PersistentVolumeLastPhaseTransitionTime` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
to see `lastPhaseTransitionTime` field.
{{< /note >}}
## PersistentVolumeClaims
Each PVC contains a spec and status, which is the specification and status of the claim.
@ -862,7 +879,7 @@ it won't be supported in a future Kubernetes release.
#### Retroactive default StorageClass assignment
{{< feature-state for_k8s_version="v1.26" state="beta" >}}
{{< feature-state for_k8s_version="v1.28" state="stable" >}}
You can create a PersistentVolumeClaim without specifying a `storageClassName`
for the new PVC, and you can do so even when no default StorageClass exists
@ -932,10 +949,12 @@ applicable:
* CSI
* FC (Fibre Channel)
* GCEPersistentDisk
* GCEPersistentDisk (deprecated)
* iSCSI
* Local volume
* RBD (Ceph Block Device)
* OpenStack Cinder
* RBD (deprecated)
* RBD (Ceph Block Device; deprecated)
* VsphereVolume
### PersistentVolume using a Raw Block Volume {#persistent-volume-using-a-raw-block-volume}

5
content/en/docs/concepts/storage/storage-classes.md
View File

@ -459,6 +459,11 @@ There are few
which you try out for persistent volume management inside Kubernetes for vSphere.
### Ceph RBD
{{< note >}}
{{< feature-state state="deprecated" for_k8s_version="v1.28" >}}
This internal provisioner of Ceph RBD is deprecated. Please use
[CephFS RBD CSI driver](https://github.com/ceph/ceph-csi).
{{< /note >}}
```yaml
apiVersion: storage.k8s.io/v1

17
content/en/docs/concepts/storage/volumes.md
View File

@ -119,6 +119,12 @@ To disable the `azureFile` storage plugin from being loaded by the controller ma
and the kubelet, set the `InTreePluginAzureFileUnregister` flag to `true`.
### cephfs
{{< feature-state for_k8s_version="v1.28" state="deprecated" >}}
{{< note >}}
The Kubernetes project suggests that you use the [CephFS CSI](https://github.com/ceph/ceph-csi) third party
storage driver instead.
{{< /note >}}
A `cephfs` volume allows an existing CephFS volume to be
mounted into your Pod. Unlike `emptyDir`, which is erased when a pod is
@ -762,6 +768,12 @@ A projected volume maps several existing volume sources into the same
directory. For more details, see [projected volumes](/docs/concepts/storage/projected-volumes/).
### rbd
{{< feature-state for_k8s_version="v1.28" state="deprecated" >}}
{{< note >}}
The Kubernetes project suggests that you use the [Ceph CSI](https://github.com/ceph/ceph-csi)
third party storage driver instead, in RBD mode.
{{< /note >}}
An `rbd` volume allows a
[Rados Block Device](https://docs.ceph.com/en/latest/rbd/) (RBD) volume to mount
@ -785,7 +797,7 @@ for more details.
#### RBD CSI migration {#rbd-csi-migration}
{{< feature-state for_k8s_version="v1.23" state="alpha" >}}
{{< feature-state for_k8s_version="v1.28" state="deprecated" >}}
The `CSIMigration` feature for `RBD`, when enabled, redirects all plugin
operations from the existing in-tree plugin to the `rbd.csi.ceph.com` {{<
@ -840,7 +852,8 @@ For more details, see [Configuring Secrets](/docs/concepts/configuration/secret/
### vsphereVolume (deprecated) {#vspherevolume}
{{< note >}}
We recommend to use vSphere CSI out-of-tree driver instead.
The Kubernetes project recommends using the [vSphere CSI](https://github.com/kubernetes-sigs/vsphere-csi-driver)
out-of-tree storage driver instead.
{{< /note >}}
A `vsphereVolume` is used to mount a vSphere VMDK volume into your Pod. The contents

130
content/en/docs/concepts/workloads/controllers/job.md
View File

@ -281,8 +281,11 @@ Jobs with _fixed completion count_ - that is, jobs that have non null
completion is homologous to each other. Note that Jobs that have null
`.spec.completions` are implicitly `NonIndexed`.
- `Indexed`: the Pods of a Job get an associated completion index from 0 to
`.spec.completions-1`. The index is available through three mechanisms:
`.spec.completions-1`. The index is available through four mechanisms:
- The Pod annotation `batch.kubernetes.io/job-completion-index`.
- The Pod label `batch.kubernetes.io/job-completion-index` (for v1.28 and later). Note
the feature gate `PodIndexLabel` must be enabled to use this label, and it is enabled
by default.
- As part of the Pod hostname, following the pattern `$(job-name)-$(index)`.
When you use an Indexed Job in combination with a
{{< glossary_tooltip term_id="Service" >}}, Pods within the Job can use
@ -322,6 +325,10 @@ By default, each pod failure is counted towards the `.spec.backoffLimit` limit,
see [pod backoff failure policy](#pod-backoff-failure-policy). However, you can
customize handling of pod failures by setting the Job's [pod failure policy](#pod-failure-policy).
Additionally, you can choose to count the pod failures independently for each
index of an [Indexed](#completion-mode) Job by setting the `.spec.backoffLimitPerIndex` field
(for more information, see [backoff limit per index](#backoff-limit-per-index)).
Note that even if you specify `.spec.parallelism = 1` and `.spec.completions = 1` and
`.spec.template.spec.restartPolicy = "Never"`, the same program may
sometimes be started twice.
@ -366,6 +373,72 @@ will be terminated once the job backoff limit has been reached. This can make de
from failed Jobs is not lost inadvertently.
{{< /note >}}
### Backoff limit per index {#backoff-limit-per-index}
{{< feature-state for_k8s_version="v1.28" state="alpha" >}}
{{< note >}}
You can only configure the backoff limit per index for an [Indexed](#completion-mode) Job, if you
have the `JobBackoffLimitPerIndex` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
enabled in your cluster.
{{< /note >}}
When you run an [indexed](#completion-mode) Job, you can choose to handle retries
for pod failures independently for each index. To do so, set the
`.spec.backoffLimitPerIndex` to specify the the maximal number of pod failures
per index.
When the per-index backoff limit is exceeded for an index, Kuberentes considers the index as failed and adds it to the
`.status.failedIndexes` field. The succeeded indexes, those with a successfully
executed pods, are recorded in the `.status.completedIndexes` field, regardless of whether you set
the `backoffLimitPerIndex` field.
Note that a failing index does not interrupt execution of other indexes.
Once all indexes finish for a Job where you specified a backoff limit per index,
if at least one of those indexes did fail, the Job controller marks the overall
Job as failed, by setting the Failed condition in the status. The Job gets
marked as failed even if some, potentially nearly all, of the indexes were
processed successfully.
You can additionally limit the maximal number of indexes marked failed by
setting the `.spec.maxFailedIndexes` field.
When the number of failed indexes exceeds the `maxFailedIndexes` field, the
Job controller triggers termination of all remaining running Pods for that Job.
Once all pods are terminated, the entire Job is marked failed by the Job
controller, by setting the Failed condition in the Job status.
Here is an example manifest for a Job that defines a `backoffLimitPerIndex`:
{{< codenew file="/controllers/job-backoff-limit-per-index-example.yaml" >}}
In the example above, the Job controller allows for one restart for each
of the indexes. When the total number of failed indexes exceeds 5, then
the entire Job is terminated.
Once the job is finished, the the Job status looks as follows:
```sh
kubectl get -o yaml job job-backoff-limit-per-index-example
```
```yaml
status:
completedIndexes: 1,3,5,7,9
failedIndexes: 0,2,4,6,8
succeeded: 5 # 1 succeeded pod for each of 5 succeeded indexes
failed: 10 # 2 failed pods (1 retry) for each of 5 failed indexes
conditions:
- message: Job has failed indexes
reason: FailedIndexes
status: "True"
type: Failed
```
Additionally, you may want to use the per-index backoff along with a
[pod failure policy](#pod-failure-policy). When using
per-index backoff, there is a new `FailIndex` action available which allows you to
avoid unnecessary retries within an index.
### Pod failure policy {#pod-failure-policy}
{{< feature-state for_k8s_version="v1.26" state="beta" >}}
@ -448,6 +521,8 @@ These are some requirements and semantics of the API:
should not be incremented and a replacement Pod should be created.
- `Count`: use to indicate that the Pod should be handled in the default way.
The counter towards the `.spec.backoffLimit` should be incremented.
- `FailIndex`: use this action along with [backoff limit per index](#backoff-limit-per-index)
to avoid unnecessary retries within the index of a failed pod.
{{< note >}}
When you use a `podFailurePolicy`, the job controller only matches Pods in the
@ -460,6 +535,12 @@ Since Kubernetes 1.27, Kubelet transitions deleted pods to a terminal phase
ensures that deleted pods have their finalizers removed by the Job controller.
{{< /note >}}
{{< note >}}
Starting with Kubernetes v1.28, when Pod failure policy is used, the Job controller recreates
terminating Pods only once these Pods reach the terminal `Failed` phase. This behavior is similar
to `podReplacementPolicy: Failed`. For more information, see [Pod replacement policy](#pod-replacement-policy).
{{< /note >}}
## Job termination and cleanup
When a Job completes, no more Pods are created, but the Pods are [usually](#pod-backoff-failure-policy) not deleted either.
@ -850,6 +931,53 @@ is disabled, `.spec.completions` is immutable.
Use cases for elastic Indexed Jobs include batch workloads which require
scaling an indexed Job, such as MPI, Horovord, Ray, and PyTorch training jobs.
### Delayed creation of replacement pods {#pod-replacement-policy}
{{< feature-state for_k8s_version="v1.28" state="alpha" >}}
{{< note >}}
You can only set `podReplacementPolicy` on Jobs if you enable the `JobPodReplacementPolicy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
{{< /note >}}
By default, the Job controller recreates Pods as soon they either fail or are terminating (have a deletion timestamp).
This means that, at a given time, when some of the Pods are terminating, the number of running Pods for the Jobs can be greater than `parallelism` or greater than one Pod per index (if using Indexed Jobs).
You may choose to create replacement Pods only when the terminating Pod is fully terminal (has `status.phase: Failed`). To do this, set the `.spec.podReplacementPolicy: Failed`.
This will only recreate Pods once they are terminated.
The default replacement policy depends on whether the Job has a `podFailurePolicy` set.
With no Pod failure policy defined for a Job, omitting the `podReplacementPolicy` field selects the
`TerminatingOrFailed` replacement policy:
the control plane creates replacement Pods immediately upon Pod deletion
(as soon as the control plane sees that a Pod for this Job has `deletionTimestamp` set).
For Jobs with a Pod failure policy set, the default `podReplacementPolicy` is `Failed`, and no other
value is permitted.
See [Pod failure policy](#pod-failure-policy) to learn more about Pod failure policies for Jobs.
```yaml
kind: Job
metadata:
name: new
...
spec:
podReplacementPolicy: Failed
...
```
Provided your cluster has the feature gate enabled, you can inspect the `.status.terminating` field of a Job.
The value of the field is the number of Pods owned by the Job that are currently terminating.
```shell
kubectl get jobs/myjob -o yaml
```
```yaml
apiVersion: batch/v1
kind: Job
# .metadata and .spec omitted
status:
terminating: 3 # three Pods are terminating and have not yet reached the Failed phase
```
## Alternatives
### Bare Pods

13
content/en/docs/concepts/workloads/controllers/statefulset.md
View File

@ -160,7 +160,8 @@ regardless of which node it's (re)scheduled on.
For a StatefulSet with N [replicas](#replicas), each Pod in the StatefulSet
will be assigned an integer ordinal, that is unique over the Set. By default,
pods will be assigned ordinals from 0 up through N-1.
pods will be assigned ordinals from 0 up through N-1. The StatefulSet controller
will also add a pod label with this index: `apps.kubernetes.io/pod-index`.
### Start ordinal
@ -238,6 +239,16 @@ it adds a label, `statefulset.kubernetes.io/pod-name`, that is set to the name o
the Pod. This label allows you to attach a Service to a specific Pod in
the StatefulSet.
### Pod index label
{{< feature-state for_k8s_version="v1.28" state="beta" >}}
When the StatefulSet {{<glossary_tooltip text="controller" term_id="controller">}} creates a Pod,
the new Pod is labelled with `apps.kubernetes.io/pod-index`. The value of this label is the ordinal index of
the Pod. This label allows you to route traffic to a particular pod index, filter logs/metrics
using the pod index label, and more. Note the feature gate `PodIndexLabel` must be enabled for this
feature, and it is enabled by default.
## Deployment and Scaling Guarantees
* For a StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order from {0..N-1}.

15
content/en/docs/concepts/workloads/pods/_index.md
View File

@ -110,7 +110,18 @@ that updates those files from a remote source, as in the following diagram:
{{< figure src="/images/docs/pod.svg" alt="Pod creation diagram" class="diagram-medium" >}}
Some Pods have {{< glossary_tooltip text="init containers" term_id="init-container" >}} as well as {{< glossary_tooltip text="app containers" term_id="app-container" >}}. Init containers run and complete before the app containers are started.
Some Pods have {{< glossary_tooltip text="init containers" term_id="init-container" >}}
as well as {{< glossary_tooltip text="app containers" term_id="app-container" >}}.
By default, init containers run and complete before the app containers are started.
{{< feature-state for_k8s_version="v1.28" state="alpha" >}}
Enabling the `SidecarContainers` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
allows you to specify `restartPolicy: Always` for init containers.
Setting the `Always` restart policy ensures that the init containers where you set it are
kept running during the entire lifetime of the Pod.
See [Sidecar containers and restartPolicy](/docs/concepts/workloads/pods/init-containers/#sidecar-containers-and-restartpolicy)
for more details.
Pods natively provide two kinds of shared resources for their constituent containers:
[networking](#pod-networking) and [storage](#pod-storage).
@ -364,4 +375,4 @@ To understand the context for why Kubernetes wraps a common Pod API in other res
* [Borg](https://research.google.com/pubs/pub43438.html)
* [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html)
* [Omega](https://research.google/pubs/pub41684/)
* [Tupperware](https://engineering.fb.com/data-center-engineering/tupperware/).
* [Tupperware](https://engineering.fb.com/data-center-engineering/tupperware/).

9
content/en/docs/concepts/workloads/pods/downward-api.md
View File

@ -44,7 +44,7 @@ You can pass information from available Container-level fields using
### Information available via `fieldRef` {#downwardapi-fieldRef}
For most Pod-level fields, you can provide them to a container either as
For some Pod-level fields, you can provide them to a container either as
an environment variable or using a `downwardAPI` volume. The fields available
via either mechanism are:
@ -75,9 +75,16 @@ The following information is available through environment variables
`status.hostIP`
: the primary IP address of the node to which the Pod is assigned
`status.hostIPs`
: the IP addresses is a dual-stack version of `status.hostIP`, the first is always the same as `status.hostIP`.
The field is available if you enable the `PodHostIPs` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
`status.podIP`
: the pod's primary IP address (usually, its IPv4 address)
`status.podIPs`
: the IP addresses is a dual-stack version of `status.podIP`, the first is always the same as `status.podIP`
The following information is available through a `downwardAPI` volume
`fieldRef`, **but not as environment variables**:

45
content/en/docs/concepts/workloads/pods/init-containers.md
View File

@ -289,7 +289,49 @@ The Pod which is already running correctly would be killed by `activeDeadlineSec
The name of each app and init container in a Pod must be unique; a
validation error is thrown for any container sharing a name with another.
### Resources
#### API for sidecar containers
{{< feature-state for_k8s_version="v1.28" state="alpha" >}}
Starting with Kubernetes 1.28 in alpha, a feature gate named `SidecarContainers`
allows you to specify a `restartPolicy` for init containers which is independent of
the Pod and other init containers. Container [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe)
can also be added to control their lifecycle.
If an init container is created with its `restartPolicy` set to `Always`, it will
start and remain running during the entire life of the Pod, which is useful for
running supporting services separated from the main application containers.
If a `readinessProbe` is specified for this init container, its result will be used
to determine the `ready` state of the Pod.
Since these containers are defined as init containers, they benefit from the same
ordering and sequential guarantees as other init containers, allowing them to
be mixed with other init containers into complex Pod initialization flows.
Compared to regular init containers, sidecar-style init containers continue to
run and the next init container can begin starting once the kubelet has set
the `started` container status for the sidecar-style init container to true.
That status either becomes true because there is a process running in the
container and no startup probe defined, or
as a result of its `startupProbe` succeeding.
This feature can be used to implement the sidecar container pattern in a more
robust way, as the kubelet always restarts a sidecar container if it fails.
Here's an example of a Deployment with two containers, one of which is a sidecar:
{{% codenew language="yaml" file="application/deployment-sidecar.yaml" %}}
This feature is also useful for running Jobs with sidecars, as the sidecar
container will not prevent the Job from completing after the main container
has finished.
Here's an example of a Job with two containers, one of which is a sidecar:
{{% codenew language="yaml" file="application/job/job-sidecar.yaml" %}}
#### Resource sharing within containers
Given the ordering and execution for init containers, the following rules
for resource usage apply:
@ -335,3 +377,4 @@ Kubernetes, consult the documentation for the version you are using.
* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/)
* Read about an overview of [kubelet](/docs/reference/command-line-tools-reference/kubelet/) and [kubectl](/docs/reference/kubectl/)
* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe.
*

20
content/en/docs/concepts/workloads/pods/pod-lifecycle.md
View File

@ -164,7 +164,7 @@ through which the Pod has or has not passed. Kubelet manages the following
PodConditions:
* `PodScheduled`: the Pod has been scheduled to a node.
* `PodHasNetwork`: (alpha feature; must be [enabled explicitly](#pod-has-network)) the
* `PodReadyToStartContainers`: (alpha feature; must be [enabled explicitly](#pod-has-network)) the
Pod sandbox has been successfully created and networking configured.
* `ContainersReady`: all containers in the Pod are ready.
* `Initialized`: all [init containers](/docs/concepts/workloads/pods/init-containers/)
@ -244,15 +244,19 @@ When a Pod's containers are Ready but at least one custom condition is missing o
{{< feature-state for_k8s_version="v1.25" state="alpha" >}}
{{< note >}}
This condition was renamed from PodHasNetwork to PodReadyToStartContainers.
{{< /note >}}
After a Pod gets scheduled on a node, it needs to be admitted by the Kubelet and
have any volumes mounted. Once these phases are complete, the Kubelet works with
a container runtime (using {{< glossary_tooltip term_id="cri" >}}) to set up a
runtime sandbox and configure networking for the Pod. If the `PodHasNetworkCondition`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled,
Kubelet reports whether a Pod has reached this initialization milestone through
the `PodHasNetwork` condition in the `status.conditions` field of a Pod.
runtime sandbox and configure networking for the Pod. If the
`PodReadyToStartContainersCondition` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled,
Kubelet reports whether a pod has reached this initialization milestone through
the `PodReadyToStartContainers` condition in the `status.conditions` field of a Pod.
The `PodHasNetwork` condition is set to `False` by the Kubelet when it detects a
The `PodReadyToStartContainers` condition is set to `False` by the Kubelet when it detects a
Pod does not have a runtime sandbox with networking configured. This occurs in
the following scenarios:
@ -264,10 +268,10 @@ the following scenarios:
sandbox virtual machine rebooting, which then requires creating a new sandbox and
fresh container network configuration.
The `PodHasNetwork` condition is set to `True` by the kubelet after the
The `PodReadyToStartContainers` condition is set to `True` by the kubelet after the
successful completion of sandbox creation and network configuration for the Pod
by the runtime plugin. The kubelet can start pulling container images and create
containers after `PodHasNetwork` condition has been set to `True`.
containers after `PodReadyToStartContainers` condition has been set to `True`.
For a Pod with init containers, the kubelet sets the `Initialized` condition to
`True` after the init containers have successfully completed (which happens

17
content/en/docs/concepts/workloads/pods/user-namespaces.md
View File

@ -48,11 +48,11 @@ ext4, xfs, fat, tmpfs, overlayfs.
In addition, support is needed in the
{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}
to use this feature with Kubernetes stateless pods:
to use this feature with Kubernetes pods:
* CRI-O: version 1.25 (and later) supports user namespaces for containers.
containerd v1.7 is not compatible with the userns support in Kubernetes v{{< skew currentVersion >}}.
containerd v1.7 is not compatible with the userns support in Kubernetes v1.27 to v{{< skew latestVersion >}}.
Kubernetes v1.25 and v1.26 used an earlier implementation that **is** compatible with containerd v1.7,
in terms of userns support.
If you are using a version of Kubernetes other than {{< skew currentVersion >}},
@ -75,7 +75,7 @@ A pod can opt-in to use user namespaces by setting the `pod.spec.hostUsers` fiel
to `false`.
The kubelet will pick host UIDs/GIDs a pod is mapped to, and will do so in a way
to guarantee that no two stateless pods on the same node use the same mapping.
to guarantee that no two pods on the same node use the same mapping.
The `runAsUser`, `runAsGroup`, `fsGroup`, etc. fields in the `pod.spec` always
refer to the user inside the container.
@ -92,7 +92,7 @@ Most applications that need to run as root but don't access other host
namespaces or resources, should continue to run fine without any changes needed
if user namespaces is activated.
## Understanding user namespaces for stateless pods
## Understanding user namespaces for pods {#pods-and-userns}
Several container runtimes with their default configuration (like Docker Engine,
containerd, CRI-O) use Linux namespaces for isolation. Other technologies exist
@ -162,15 +162,6 @@ allowed to set any of:
* `hostIPC: true`
* `hostPID: true`
The pod is allowed to use no volumes at all or, if using volumes, only these
volume types are allowed:
* configmap
* secret
* projected
* downwardAPI
* emptyDir
## {{% heading "whatsnext" %}}
* Take a look at [Use a User Namespace With a Pod](/docs/tasks/configure-pod-container/user-namespaces/)

18
content/en/docs/reference/access-authn-authz/authentication.md
View File

@ -1220,7 +1220,7 @@ The following `ExecCredential` manifest describes a cluster information sample.
## API access to authentication information for a client {#self-subject-review}
{{< feature-state for_k8s_version="v1.27" state="beta" >}}
{{< feature-state for_k8s_version="v1.28" state="stable" >}}
If your cluster has the API enabled, you can use the `SelfSubjectReview` API to find out how your Kubernetes cluster maps your authentication
information to identify you as a client. This works whether you are authenticating as a user (typically representing
@ -1230,11 +1230,11 @@ a real person) or as a ServiceAccount.
Request example (the body would be a `SelfSubjectReview`):
```
POST /apis/authentication.k8s.io/v1beta1/selfsubjectreviews
POST /apis/authentication.k8s.io/v1/selfsubjectreviews
```
```json
{
"apiVersion": "authentication.k8s.io/v1beta1",
"apiVersion": "authentication.k8s.io/v1",
"kind": "SelfSubjectReview"
}
```
@ -1242,7 +1242,7 @@ Response example:
```json
{
"apiVersion": "authentication.k8s.io/v1beta1",
"apiVersion": "authentication.k8s.io/v1",
"kind": "SelfSubjectReview",
"status": {
"userInfo": {
@ -1285,7 +1285,7 @@ By providing the output flag, it is also possible to print the JSON or YAML repr
{{% tab name="JSON" %}}
```json
{
"apiVersion": "authentication.k8s.io/v1alpha1",
"apiVersion": "authentication.k8s.io/v1",
"kind": "SelfSubjectReview",
"status": {
"userInfo": {
@ -1314,7 +1314,7 @@ By providing the output flag, it is also possible to print the JSON or YAML repr
{{% tab name="YAML" %}}
```yaml
apiVersion: authentication.k8s.io/v1alpha1
apiVersion: authentication.k8s.io/v1
kind: SelfSubjectReview
status:
userInfo:
@ -1351,8 +1351,10 @@ By default, all authenticated users can create `SelfSubjectReview` objects when
You can only make `SelfSubjectReview` requests if:
* the `APISelfSubjectReview`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
is enabled for your cluster (enabled by default after reaching Beta)
* the API server for your cluster has the `authentication.k8s.io/v1alpha1` or `authentication.k8s.io/v1beta1`
is enabled for your cluster (not needed for Kubernetes {{< skew currentVersion >}}, but older
Kubernetes versions might not offer this feature gate, or might default it to be off)
* (if you are running a version of Kubernetes older than v1.28) the API server for your
cluster has the `authentication.k8s.io/v1alpha1` or `authentication.k8s.io/v1beta1`
{{< glossary_tooltip term_id="api-group" text="API group" >}}
enabled.
{{< /note >}}

63
content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md
View File

@ -721,14 +721,9 @@ The `matchPolicy` for an admission webhooks defaults to `Equivalent`.
### Matching requests: `matchConditions`
{{< feature-state state="alpha" for_k8s_version="v1.27" >}}
{{< feature-state state="beta" for_k8s_version="v1.28" >}}
{{< note >}}
Use of `matchConditions` requires the [featuregate](/docs/reference/command-line-tools-reference/feature-gates/)
`AdmissionWebhookMatchConditions` to be explicitly enabled on the kube-apiserver before this feature can be used.
{{< /note >}}
You can define _match conditions_for webhooks if you need fine-grained request filtering. These
You can define _match conditions_ for webhooks if you need fine-grained request filtering. These
conditions are useful if you find that match rules, `objectSelectors` and `namespaceSelectors` still
doesn't provide the filtering you want over when to call out over HTTP. Match conditions are
[CEL expressions](/docs/reference/using-api/cel/). All match conditions must evaluate to true for the
@ -736,55 +731,11 @@ webhook to be called.
Here is an example illustrating a few different uses for match conditions:
```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ['CREATE','UPDATE']
apiGroups: ['*']
apiVersions: ['*']
resources: ['*']
failurePolicy: 'Ignore' # Fail-open (optional)
sideEffects: None
clientConfig:
service:
namespace: my-namespace
name: my-webhook
caBundle: '<omitted>'
matchConditions:
- name: 'exclude-leases' # Each match condition must have a unique name
expression: '!(request.resource.group == "coordination.k8s.io" && request.resource.resource == "leases")' # Match non-lease resources.
- name: 'exclude-kubelet-requests'
expression: '!("system:nodes" in request.userInfo.groups)' # Match requests made by non-node users.
- name: 'rbac' # Skip RBAC requests, which are handled by the second webhook.
expression: 'request.resource.group != "rbac.authorization.k8s.io"'
# This example illustrates the use of the 'authorizer'. The authorization check is more expensive
# than a simple expression, so in this example it is scoped to only RBAC requests by using a second
# webhook. Both webhooks can be served by the same endpoint.
- name: rbac.my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ['CREATE','UPDATE']
apiGroups: ['rbac.authorization.k8s.io']
apiVersions: ['*']
resources: ['*']
failurePolicy: 'Fail' # Fail-closed (the default)
sideEffects: None
clientConfig:
service:
namespace: my-namespace
name: my-webhook
caBundle: '<omitted>'
matchConditions:
- name: 'breakglass'
# Skip requests made by users authorized to 'breakglass' on this webhook.
# The 'breakglass' API verb does not need to exist outside this check.
expression: '!authorizer.group("admissionregistration.k8s.io").resource("validatingwebhookconfigurations").name("my-webhook.example.com").check("breakglass").allowed()'
```
{{< codenew file="access/validating-webhook-configuration-match-conditions.yaml" >}}
{{< note >}}
You can define up to 64 elements in the `matchConditions` field per webhook.
{{< /note >}}
Match conditions have access to the following CEL variables:

290
content/en/docs/reference/access-authn-authz/validating-admission-policy.md
View File

@ -9,7 +9,7 @@ content_type: concept
<!-- overview -->
{{< feature-state state="alpha" for_k8s_version="v1.26" >}}
{{< feature-state state="beta" for_k8s_version="v1.28" >}}
This page provides an overview of Validating Admission Policy.
@ -45,12 +45,12 @@ At least a `ValidatingAdmissionPolicy` and a corresponding `ValidatingAdmission
must be defined for a policy to have an effect.
If a `ValidatingAdmissionPolicy` does not need to be configured via parameters, simply leave
`spec.paramKind` in `ValidatingAdmissionPolicy` unset.
`spec.paramKind` in `ValidatingAdmissionPolicy` not specified.
## {{% heading "prerequisites" %}}
- Ensure the `ValidatingAdmissionPolicy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled.
- Ensure that the `admissionregistration.k8s.io/v1alpha1` API is enabled.
- Ensure that the `admissionregistration.k8s.io/v1beta1` API is enabled.
## Getting Started with Validating Admission Policy
@ -61,22 +61,7 @@ with great caution. The following describes how to quickly experiment with Valid
The following is an example of a ValidatingAdmissionPolicy.
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
name: "demo-policy.example.com"
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.spec.replicas <= 5"
```
{{% codenew language="yaml" file="validatingadmissionpolicy/basic-example-policy.yaml" %}}
`spec.validations` contains CEL expressions which use the [Common Expression Language (CEL)](https://github.com/google/cel-spec)
to validate the request. If an expression evaluates to false, the validation check is enforced
@ -85,19 +70,7 @@ according to the `spec.failurePolicy` field.
To configure a validating admission policy for use in a cluster, a binding is required.
The following is an example of a ValidatingAdmissionPolicyBinding.:
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "demo-binding-test.example.com"
spec:
policyName: "demo-policy.example.com"
validationActions: [Deny]
matchResources:
namespaceSelector:
matchLabels:
environment: test
```
{{% codenew language="yaml" file="validatingadmissionpolicy/basic-example-binding.yaml" %}}
When trying to create a deployment with replicas set not satisfying the validation expression, an
error will return containing message:
@ -133,13 +106,13 @@ API response body and the HTTP warning headers.
A `validation` that evaluates to false is always enforced according to these
actions. Failures defined by the `failurePolicy` are enforced
according to these actions only if the `failurePolicy` is set to `Fail` (or unset),
according to these actions only if the `failurePolicy` is set to `Fail` (or not specified),
otherwise the failures are ignored.
See [Audit Annotations: validation falures](/docs/reference/labels-annotations-taints/audit-annotations/#validation-policy-admission-k8s-io-validation_failure)
for more details about the validation failure audit annotation.
#### Parameter resources
### Parameter resources
Parameter resources allow a policy configuration to be separate from its definition.
A policy can define paramKind, which outlines GVK of the parameter resource,
@ -148,26 +121,7 @@ and then a policy binding ties a policy by name (via policyName) to a particular
If parameter configuration is needed, the following is an example of a ValidatingAdmissionPolicy
with parameter configuration.
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
name: "replicalimit-policy.example.com"
spec:
failurePolicy: Fail
paramKind:
apiVersion: rules.example.com/v1
kind: ReplicaLimit
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.spec.replicas <= params.maxReplicas"
reason: Invalid
```
{{% codenew language="yaml" file="validatingadmissionpolicy/policy-with-param.yaml" %}}
The `spec.paramKind` field of the ValidatingAdmissionPolicy specifies the kind of resources used
to parameterize this policy. For this example, it is configured by ReplicaLimit custom resources.
@ -182,89 +136,51 @@ validation check is enforced according to the `spec.failurePolicy` field.
The validating admission policy author is responsible for providing the ReplicaLimit parameter CRD.
To configure an validating admission policy for use in a cluster, a binding and parameter resource
are created. The following is an example of a ValidatingAdmissionPolicyBinding.
are created. The following is an example of a ValidatingAdmissionPolicyBinding
that uses a **cluster-wide** param - the same param will be used to validate
every resource request that matches the binding:
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "replicalimit-binding-test.example.com"
spec:
policyName: "replicalimit-policy.example.com"
validationActions: [Deny]
paramRef:
name: "replica-limit-test.example.com"
matchResources:
namespaceSelector:
matchLabels:
environment: test
```
{{% codenew language="yaml" file="validatingadmissionpolicy/binding-with-param.yaml" %}}
Notice this binding applies a parameter to the policy for all resources which
are in the `test` environment.
The parameter resource could be as following:
```yaml
apiVersion: rules.example.com/v1
kind: ReplicaLimit
metadata:
name: "replica-limit-test.example.com"
maxReplicas: 3
```
{{% codenew language="yaml" file="validatingadmissionpolicy/replicalimit-param.yaml" %}}
This policy parameter resource limits deployments to a max of 3 replicas in all namespaces in the
test environment. An admission policy may have multiple bindings. To bind all other environments
environment to have a maxReplicas limit of 100, create another ValidatingAdmissionPolicyBinding:
This policy parameter resource limits deployments to a max of 3 replicas.
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "replicalimit-binding-nontest"
spec:
policyName: "replicalimit-policy.example.com"
validationActions: [Deny]
paramRef:
name: "replica-limit-clusterwide.example.com"
matchResources:
namespaceSelector:
matchExpressions:
- key: environment
operator: NotIn
values:
- test
```
An admission policy may have multiple bindings. To bind all other environments
to have a maxReplicas limit of 100, create another ValidatingAdmissionPolicyBinding:
And have a parameter resource like:
{{% codenew language="yaml" file="validatingadmissionpolicy/binding-with-param-prod.yaml" %}}
```yaml
apiVersion: rules.example.com/v1
kind: ReplicaLimit
metadata:
name: "replica-limit-clusterwide.example.com"
maxReplicas: 100
```
Notice this binding applies a different parameter to resources which
are not in the `test` environment.
Bindings can have overlapping match criteria. The policy is evaluated for each matching binding.
In the above example, the "nontest" policy binding could instead have been defined as a global policy:
And have a parameter resource:
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "replicalimit-binding-global"
spec:
policyName: "replicalimit-policy.example.com"
validationActions: [Deny]
params: "replica-limit-clusterwide.example.com"
matchResources:
namespaceSelector:
matchExpressions:
- key: environment
operator: Exists
```
{{% codenew language="yaml" file="validatingadmissionpolicy/replicalimit-param-prod.yaml" %}}
For each admission request, the API server evaluates CEL expressions of each
(policy, binding, param) combination that match the request. For a request
to be admitted it must pass **all** evaluations.
If multiple bindings match the request, the policy will be evaluated for each,
and they must all pass evaluation for the policy to be considered passed.
If multiple parameters match a single binding, the policy rules will be evaluated
for each param, and they too must all pass for the binding to be considered passed.
Bindings can have overlapping match criteria. The policy is evaluated for each
matching binding-parameter combination. A policy may even be evaluated multiple
times if multiple bindings match it, or a single binding that matches multiple
parameters.
The params object representing a parameter resource will not be set if a parameter resource has
not been bound, so for policies requiring a parameter resource, it can be useful to add a check to
ensure one has been bound.
ensure one has been bound. A parameter resource will not be bound and `params` will be null
if `paramKind` of the policy, or `paramRef` of the binding are not specified.
For the use cases require parameter configuration, we recommend to add a param check in
`spec.validations[0].expression`:
@ -274,6 +190,8 @@ For the use cases require parameter configuration, we recommend to add a param c
message: "params missing but required to bind to this policy"
```
#### Optional parameters
It can be convenient to be able to have optional parameters as part of a parameter resource, and
only validate them if present. CEL provides `has()`, which checks if the key passed to it exists.
CEL also implements Boolean short-circuiting. If the first half of a logical OR evaluates to true,
@ -291,7 +209,38 @@ Here, we first check that the optional parameter is present with `!has(params.op
evaluated, and optionalNumber will be checked to ensure that it contains a value between 5 and
10 inclusive.
#### Authorization Check
#### Per-namespace Parameters
As the author of a ValidatingAdmissionPolicy and its ValidatingAdmissionPolicyBinding,
you can choose to specify cluster-wide, or per-namespace parameters.
If you specify a `namespace` for the binding's `paramRef`, the control plane only
searches for parameters in that namespace.
However, if `namespace` is not specified in the ValidatingAdmissionPolicyBinding, the
API server can search for relevant parameters in the namespace that a request is against.
For example, if you make a request to modify a ConfigMap in the `default` namespace and
there is a relevant ValidatingAdmissionPolicyBinding with no `namespace` set, then the
API server looks for a parameter object in `default`.
This design enables policy configuration that depends on the namespace
of the resource being manipulated, for more fine-tuned control.
#### Parameter selector
In addition to specify a parameter in a binding by `name`, you may
choose instead to specify label selector, such that all resources of the
policy's `paramKind`, and the param's `namespace` (if applicable) that match the
label selector are selected for evaluation. See {{< glossary_tooltip text="selector" term_id="selector">}} for more information on how label selectors match resources.
If multiple parameters are found to meet the condition, the policy's rules are
evaluated for each parameter found and the results will be ANDed together.
If `namespace` is provided, only objects of the `paramKind` in the provided
namespace are eligible for selection. Otherwise, when `namespace` is empty and
`paramKind` is namespace-scoped, the `namespace` used in the request being
admitted will be used.
#### Authorization checks {#authorization-check}
We introduced the authorization check for parameter resources.
User is expected to have `read` access to the resources referenced by `paramKind` in
@ -312,15 +261,7 @@ admission policy are handled. Allowed values are `Ignore` or `Fail`.
Note that the `failurePolicy` is defined inside `ValidatingAdmissionPolicy`:
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
spec:
...
failurePolicy: Ignore # The default is "Fail"
validations:
- expression: "object.spec.xyz == params.x"
```
{{% codenew language="yaml" file="validatingadmissionpolicy/failure-policy-ignore.yaml" %}}
### Validation Expression
@ -333,7 +274,9 @@ variables as well as some other useful variables:
- 'oldObject' - The existing object. The value is null for CREATE requests.
- 'request' - Attributes of the [admission request](/docs/reference/config-api/apiserver-admission.v1/#admission-k8s-io-v1-AdmissionRequest).
- 'params' - Parameter resource referred to by the policy binding being evaluated. The value is
null if `ParamKind` is unset.
null if `ParamKind` is not specified.
- `namespaceObject` - The namespace, as a Kubernetes resource, that the incoming object belongs to.
The value is null if the incoming object is cluster-scoped.
- `authorizer` - A CEL Authorizer. May be used to perform authorization checks for the principal
(authenticated user) of the request. See
[Authz](https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#Authz) in the Kubernetes CEL library
@ -466,8 +409,8 @@ event and all other values will be ignored.
To return a more friendly message when the policy rejects a request, we can use a CEL expression
to composite a message with `spec.validations[i].messageExpression`. Similar to the validation expression,
a message expression has access to `object`, `oldObject`, `request`, and `params`. Unlike validations,
message expression must evaluate to a string.
a message expression has access to `object`, `oldObject`, `request`, `params`, and `namespaceObject`.
Unlike validations, message expression must evaluate to a string.
For example, to better inform the user of the reason of denial when the policy refers to a parameter,
we can have the following validation:
@ -502,23 +445,7 @@ and an empty `status.typeChecking` means that no errors were detected.
For example, given the following policy definition:
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
name: "deploy-replica-policy.example.com"
spec:
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.replicas > 1" # should be "object.spec.replicas > 1"
message: "must be replicated"
reason: Invalid
```
{{< codenew language="yaml" file="validatingadmissionpolicy/typechecking.yaml" >}}
The status will yield the following information:
@ -536,23 +463,8 @@ status:
If multiple resources are matched in `spec.matchConstraints`, all of matched resources will be checked against.
For example, the following policy definition
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
name: "replica-policy.example.com"
spec:
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments","replicasets"]
validations:
- expression: "object.replicas > 1" # should be "object.spec.replicas > 1"
message: "must be replicated"
reason: Invalid
```
{{% codenew language="yaml" file="validatingadmissionpolicy/typechecking-multiple-match.yaml" %}}
will have multiple types and type checking result of each type in the warning message.
@ -579,3 +491,39 @@ Type Checking has the following limitation:
- Type Checking does not affect the policy behavior in any way. Even if the type checking detects errors, the policy will continue
to evaluate. If errors do occur during evaluate, the failure policy will decide its outcome.
- Type Checking does not apply to CRDs, including matched CRD types and reference of paramKind. The support for CRDs will come in future release.
### Variable composition
If an expression grows too complicated, or part of the expression is reusable and computationally expensive to evaluate,
you can extract some part of the expressions into variables. A variable is a named expression that can be referred later
in `variables` in other expressions.
```yaml
spec:
variables:
- name: foo
expression: "'foo' in object.spec.metadata.labels ? object.spec.metadata.labels['foo'] : 'default'"
validations:
- expression: variables.foo == 'bar'
```
A variable is lazily evaluated when it is first referred. Any error that occurs during the evaluation will be
reported during the evaluation of the referring expression. Both the result and potential error are memorized and
count only once towards the runtime cost.
The order of variables are important because a variable can refer to other variables that are defined before it.
This ordering prevents circular references.
The following is a more complex example of enforcing that image repo names match the environment defined in its namespace.
{{< codenew file="access/image-matches-namespace-environment.policy.yaml" >}}
With the policy bound to the namespace `default`, which is labeled `environment: prod`,
the following attempt to create a deployment would be rejected.
```shell
kubectl create deploy --image=dev.example.com/nginx invalid
```
The error message is similar to this.
```console
error: failed to create deployment: deployments.apps "invalid" is forbidden: ValidatingAdmissionPolicy 'image-matches-namespace-environment.policy.example.com' with binding 'demo-binding-test.example.com' denied request: only prod images are allowed in namespace default
```

21
content/en/docs/reference/command-line-tools-reference/feature-gates-removed.md
View File

@ -32,6 +32,9 @@ In the following table:
|---------|---------|-------|-------|-------|
| `Accelerators` | `false` | Alpha | 1.6 | 1.10 |
| `Accelerators` | - | Deprecated | 1.11 | 1.11 |
| `AdvancedAuditing` | `false` | Alpha | 1.7 | 1.7 |
| `AdvancedAuditing` | `true` | Beta | 1.8 | 1.11 |
| `AdvancedAuditing` | `true` | GA | 1.12 | 1.27 |
| `AffinityInAnnotations` | `false` | Alpha | 1.6 | 1.7 |
| `AffinityInAnnotations` | - | Deprecated | 1.8 | 1.8 |
| `AllowExtTrafficLocalEndpoints` | `false` | Beta | 1.4 | 1.6 |
@ -134,6 +137,12 @@ In the following table:
| `DefaultPodTopologySpread` | `false` | Alpha | 1.19 | 1.19 |
| `DefaultPodTopologySpread` | `true` | Beta | 1.20 | 1.23 |
| `DefaultPodTopologySpread` | `true` | GA | 1.24 | 1.25 |
| `DelegateFSGroupToCSIDriver` | `false` | Alpha | 1.22 | 1.22 |
| `DelegateFSGroupToCSIDriver` | `true` | Beta | 1.23 | 1.25 |
| `DelegateFSGroupToCSIDriver` | `true` | GA | 1.26 | 1.27 |
| `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 |
| `DevicePlugins` | `true` | Beta | 1.10 | 1.25 |
| `DevicePlugins` | `true` | GA | 1.26 | 1.27 |
| `DynamicAuditing` | `false` | Alpha | 1.13 | 1.18 |
| `DynamicAuditing` | - | Deprecated | 1.19 | 1.19 |
| `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 |
@ -244,6 +253,7 @@ In the following table:
| `PodDisruptionBudget` | `false` | Alpha | 1.3 | 1.4 |
| `PodDisruptionBudget` | `true` | Beta | 1.5 | 1.20 |
| `PodDisruptionBudget` | `true` | GA | 1.21 | 1.25 |
| `PodHasNetworkCondition` | `false` | Alpha | 1.25 | 1.27 |
| `PodOverhead` | `false` | Alpha | 1.16 | 1.17 |
| `PodOverhead` | `true` | Beta | 1.18 | 1.23 |
| `PodOverhead` | `true` | GA | 1.24 | 1.25 |
@ -389,6 +399,8 @@ In the following table:
- `AffinityInAnnotations`: Enable setting
[Pod affinity or anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit)
- `AllowExtTrafficLocalEndpoints`: Enable a service to route external requests to node local endpoints.
- `AllowInsecureBackendProxy`: Enable the users to skip TLS verification of
@ -564,6 +576,13 @@ In the following table:
- `DefaultPodTopologySpread`: Enables the use of `PodTopologySpread` scheduling plugin to do
[default spreading](/docs/concepts/scheduling-eviction/topology-spread-constraints/#internal-default-constraints).
- `DelegateFSGroupToCSIDriver`: If supported by the CSI driver, delegates the
role of applying `fsGroup` from a Pod's `securityContext` to the driver by
passing `fsGroup` through the NodeStageVolume and NodePublishVolume CSI calls.
- `DevicePlugins`: Enable the [device-plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
based resource provisioning on nodes.
- `DynamicAuditing`: Used to enable dynamic auditing before v1.19.
- `DynamicKubeletConfig`: Enable the dynamic configuration of kubelet. The
@ -699,6 +718,8 @@ In the following table:
- `PodDisruptionBudget`: Enable the [PodDisruptionBudget](/docs/tasks/run-application/configure-pdb/) feature.
- `PodHasNetwork`: Enable the kubelet to mark the [PodHasNetwork](/docs/concepts/workloads/pods/pod-lifecycle/#pod-has-network)
condition on pods. This was renamed to `PodReadyToStartContainersCondition` in 1.28.
- `PodOverhead`: Enable the [PodOverhead](/docs/concepts/scheduling-eviction/pod-overhead/)
feature to account for pod overheads.

153
content/en/docs/reference/command-line-tools-reference/feature-gates.md
View File

@ -62,13 +62,12 @@ For a reference to old feature gates that are removed, please refer to
| `APIPriorityAndFairness` | `true` | Beta | 1.20 | |
| `APIResponseCompression` | `false` | Alpha | 1.7 | 1.15 |
| `APIResponseCompression` | `true` | Beta | 1.16 | |
| `APISelfSubjectReview` | `false` | Alpha | 1.26 | 1.26 |
| `APISelfSubjectReview` | `true` | Beta | 1.27 | |
| `APIServerIdentity` | `false` | Alpha | 1.20 | 1.25 |
| `APIServerIdentity` | `true` | Beta | 1.26 | |
| `APIServerTracing` | `false` | Alpha | 1.22 | 1.26 |
| `APIServerTracing` | `true` | Beta | 1.27 | |
| `AdmissionWebhookMatchConditions` | `false` | Alpha | 1.27 | |
| `AdmissionWebhookMatchConditions` | `false` | Alpha | 1.27 | 1.27 |
| `AdmissionWebhookMatchConditions` | `true` | Beta | 1.28 | |
| `AggregatedDiscoveryEndpoint` | `false` | Alpha | 1.26 | 1.26 |
| `AggregatedDiscoveryEndpoint` | `true` | Beta | 1.27 | |
| `AnyVolumeDataSource` | `false` | Alpha | 1.18 | 1.23 |
@ -80,7 +79,6 @@ For a reference to old feature gates that are removed, please refer to
| `CPUManagerPolicyOptions` | `true` | Beta | 1.23 | |
| `CSIMigrationPortworx` | `false` | Alpha | 1.23 | 1.24 |
| `CSIMigrationPortworx` | `false` | Beta | 1.25 | |
| `CSIMigrationRBD` | `false` | Alpha | 1.23 | |
| `CSINodeExpandSecret` | `false` | Alpha | 1.25 | 1.26 |
| `CSINodeExpandSecret` | `true` | Beta | 1.27 | |
| `CSIVolumeHealth` | `false` | Alpha | 1.21 | |
@ -89,8 +87,10 @@ For a reference to old feature gates that are removed, please refer to
| `ClusterTrustBundle` | false | Alpha | 1.27 | |
| `ComponentSLIs` | `false` | Alpha | 1.26 | 1.26 |
| `ComponentSLIs` | `true` | Beta | 1.27 | |
| `ConsistentListFromCache` | `false` | Alpha | 1.28 |
| `ContainerCheckpoint` | `false` | Alpha | 1.25 | |
| `ContextualLogging` | `false` | Alpha | 1.24 | |
| CRDValidationRatcheting | false | Alpha | 1.28 |
| `CrossNamespaceVolumeDataSource` | `false` | Alpha| 1.26 | |
| `CustomCPUCFSQuotaPeriod` | `false` | Alpha | 1.12 | |
| `CustomResourceValidationExpressions` | `false` | Alpha | 1.23 | 1.24 |
@ -101,8 +101,6 @@ For a reference to old feature gates that are removed, please refer to
| `ElasticIndexedJob` | `true` | Beta` | 1.27 | |
| `EventedPLEG` | `false` | Alpha | 1.26 | 1.26 |
| `EventedPLEG` | `false` | Beta | 1.27 | - |
| `ExpandedDNSConfig` | `false` | Alpha | 1.22 | 1.25 |
| `ExpandedDNSConfig` | `true` | Beta | 1.26 | |
| `ExperimentalHostUserNamespaceDefaulting` | `false` | Beta | 1.5 | |
| `GracefulNodeShutdown` | `false` | Alpha | 1.20 | 1.20 |
| `GracefulNodeShutdown` | `true` | Beta | 1.21 | |
@ -112,8 +110,6 @@ For a reference to old feature gates that are removed, please refer to
| `HPAContainerMetrics` | `true` | Beta | 1.27 | |
| `HPAScaleToZero` | `false` | Alpha | 1.16 | |
| `HonorPVReclaimPolicy` | `false` | Alpha | 1.23 | |
| `IPTablesOwnershipCleanup` | `false` | Alpha | 1.25 | 1.26 |
| `IPTablesOwnershipCleanup` | `true` | Beta | 1.27 | |
| `InPlacePodVerticalScaling` | `false` | Alpha | 1.27 | |
| `InTreePluginAWSUnregister` | `false` | Alpha | 1.21 | |
| `InTreePluginAzureDiskUnregister` | `false` | Alpha | 1.21 | |
@ -123,23 +119,22 @@ For a reference to old feature gates that are removed, please refer to
| `InTreePluginPortworxUnregister` | `false` | Alpha | 1.23 | |
| `InTreePluginRBDUnregister` | `false` | Alpha | 1.23 | |
| `InTreePluginvSphereUnregister` | `false` | Alpha | 1.21 | |
| `JobBackoffLimitPerIndex` | `false` | Alpha | 1.28 | |
| `JobPodFailurePolicy` | `false` | Alpha | 1.25 | 1.25 |
| `JobPodFailurePolicy` | `true` | Beta | 1.26 | |
| `JobPodReplacementPolicy` | `false` | Alpha | 1.28 | |
| `JobReadyPods` | `false` | Alpha | 1.23 | 1.23 |
| `JobReadyPods` | `true` | Beta | 1.24 | |
| `KMSv2` | `false` | Alpha | 1.25 | 1.26 |
| `KMSv2` | `true` | Beta | 1.27 | |
| `KMSv2KDF` | `false` | Beta | 1.28 | |
| `KubeletCgroupDriverFromCRI` | `false` | Alpha | 1.28 | |
| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | |
| `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 |
| `KubeletPodResources` | `true` | Beta | 1.15 | |
| `KubeletPodResourcesDynamicResources` | `false` | Alpha | 1.27 | |
| `KubeletPodResourcesGet` | `false` | Alpha | 1.27 | |
| `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | 1.22 |
| `KubeletPodResourcesGetAllocatable` | `true` | Beta | 1.23 | |
| `KubeletTracing` | `false` | Alpha | 1.25 | 1.26 |
| `KubeletTracing` | `true` | Beta | 1.27 | |
| `LegacyServiceAccountTokenTracking` | `false` | Alpha | 1.26 | 1.26 |
| `LegacyServiceAccountTokenTracking` | `true` | Beta | 1.27 | |
| `LegacyServiceAccountTokenCleanUp` | `false` | Alpha | 1.28 | |
| `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | - |
| `LogarithmicScaleDown` | `false` | Alpha | 1.21 | 1.21 |
| `LogarithmicScaleDown` | `true` | Beta | 1.22 | |
@ -154,8 +149,6 @@ For a reference to old feature gates that are removed, please refer to
| `MinDomainsInPodTopologySpread` | `false` | Alpha | 1.24 | 1.24 |
| `MinDomainsInPodTopologySpread` | `false` | Beta | 1.25 | 1.26 |
| `MinDomainsInPodTopologySpread` | `true` | Beta | 1.27 | |
| `MinimizeIPTablesRestore` | `false` | Alpha | 1.26 | 1.26 |
| `MinimizeIPTablesRestore` | `true` | Beta | 1.27 | |
| `MultiCIDRRangeAllocator` | `false` | Alpha | 1.25 | |
| `MultiCIDRServiceAllocator` | `false` | Alpha | 1.27 | |
| `NetworkPolicyStatus` | `false` | Alpha | 1.24 | |
@ -163,9 +156,8 @@ For a reference to old feature gates that are removed, please refer to
| `NodeInclusionPolicyInPodTopologySpread` | `false` | Alpha | 1.25 | 1.25 |
| `NodeInclusionPolicyInPodTopologySpread` | `true` | Beta | 1.26 | |
| `NodeLogQuery` | `false` | Alpha | 1.27 | |
| `NodeOutOfServiceVolumeDetach` | `false` | Alpha | 1.24 | 1.25 |
| `NodeOutOfServiceVolumeDetach` | `true` | Beta | 1.26 | |
| `NodeSwap` | `false` | Alpha | 1.22 | |
| `NodeSwap` | `false` | Alpha | 1.22 | 1.27 |
| `NodeSwap` | `false` | Beta1 | 1.28 | |
| `OpenAPIEnums` | `false` | Alpha | 1.23 | 1.23 |
| `OpenAPIEnums` | `true` | Beta | 1.24 | |
| `PDBUnhealthyPodEvictionPolicy` | `false` | Alpha | 1.26 | 1.26 |
@ -175,29 +167,28 @@ For a reference to old feature gates that are removed, please refer to
| `PodDeletionCost` | `true` | Beta | 1.22 | |
| `PodDisruptionConditions` | `false` | Alpha | 1.25 | 1.25 |
| `PodDisruptionConditions` | `true` | Beta | 1.26 | |
| `PodHasNetworkCondition` | `false` | Alpha | 1.25 | |
| `PodHostIPs` | `false` | Alpha | 1.28 | |
| `PodReadyToStartContainersCondition` | `false` | Alpha | 1.28 | |
| `PodSchedulingReadiness` | `false` | Alpha | 1.26 | 1.26 |
| `PodSchedulingReadiness` | `true` | Beta | 1.27 | |
| `ProbeTerminationGracePeriod` | `false` | Alpha | 1.21 | 1.21 |
| `ProbeTerminationGracePeriod` | `false` | Beta | 1.22 | 1.24 |
| `ProbeTerminationGracePeriod` | `true` | Beta | 1.25 | |
| `ProcMountType` | `false` | Alpha | 1.12 | |
| `ProxyTerminatingEndpoints` | `false` | Alpha | 1.22 | 1.25 |
| `ProxyTerminatingEndpoints` | `true` | Beta | 1.26 | |
| `QOSReserved` | `false` | Alpha | 1.11 | |
| `ReadWriteOncePod` | `false` | Alpha | 1.22 | 1.26 |
| `ReadWriteOncePod` | `true` | Beta | 1.27 | |
| `RecoverVolumeExpansionFailure` | `false` | Alpha | 1.23 | |
| `RemainingItemCount` | `false` | Alpha | 1.15 | 1.15 |
| `RemainingItemCount` | `true` | Beta | 1.16 | |
| `RetroactiveDefaultStorageClass` | `false` | Alpha | 1.25 | 1.25 |
| `RetroactiveDefaultStorageClass` | `true` | Beta | 1.26 | |
| `RotateKubeletServerCertificate` | `false` | Alpha | 1.7 | 1.11 |
| `RotateKubeletServerCertificate` | `true` | Beta | 1.12 | |
| `SELinuxMountReadWriteOncePod` | `false` | Alpha | 1.25 | 1.26 |
| `SELinuxMountReadWriteOncePod` | `true` | Beta | 1.27 | |
| `SecurityContextDeny` | `false` | Alpha | 1.27 | |
| `ServiceNodePortStaticSubrange` | `false` | Alpha | 1.27 | |
| `ServiceNodePortStaticSubrange` | `false` | Alpha | 1.27 | 1.27 |
| `ServiceNodePortStaticSubrange` | `true` | Beta | 1.28 | |
| `SidecarContainers` | `false` | Alpha | 1.28 | |
| `SizeMemoryBackedVolumes` | `false` | Alpha | 1.20 | 1.21 |
| `SizeMemoryBackedVolumes` | `true` | Beta | 1.22 | |
| `StableLoadBalancerNodeSet` | `true` | Beta | 1.27 | |
@ -212,9 +203,13 @@ For a reference to old feature gates that are removed, please refer to
| `TopologyAwareHints` | `false` | Beta | 1.23 | 1.23 |
| `TopologyAwareHints` | `true` | Beta | 1.24 | |
| `TopologyManagerPolicyAlphaOptions` | `false` | Alpha | 1.26 | |
| `TopologyManagerPolicyBetaOptions` | `false` | Beta | 1.26 | |
| `TopologyManagerPolicyOptions` | `false` | Alpha | 1.26 | |
| `UserNamespacesStatelessPodsSupport` | `false` | Alpha | 1.25 | |
| `TopologyManagerPolicyBetaOptions` | `false` | Beta | 1.26 | 1.27 |
| `TopologyManagerPolicyBetaOptions` | `true` | Beta | 1.28 | |
| `TopologyManagerPolicyOptions` | `false` | Alpha | 1.26 | 1.27 |
| `TopologyManagerPolicyOptions` | `true` | Beta | 1.28 | |
| `UnknownVersionInteroperabilityProxy` | `false` | Alpha | 1.28 | |
| `UserNamespacesStatelessPodsSupport` | `false` | Alpha | 1.25 | 1.27 |
| `UserNamespacesSupport` | `false` | Alpha | 1.28 | |
| `ValidatingAdmissionPolicy` | `false` | Alpha | 1.26 | |
| `VolumeCapacityPriority` | `false` | Alpha | 1.21 | - |
| `WatchList` | false | Alpha | 1.27 | |
@ -230,9 +225,9 @@ For a reference to old feature gates that are removed, please refer to
| Feature | Default | Stage | Since | Until |
|---------|---------|-------|-------|-------|
| `AdvancedAuditing` | `false` | Alpha | 1.7 | 1.7 |
| `AdvancedAuditing` | `true` | Beta | 1.8 | 1.11 |
| `AdvancedAuditing` | `true` | GA | 1.12 | - |
| `APISelfSubjectReview` | `false` | Alpha | 1.26 | 1.26 |
| `APISelfSubjectReview` | `true` | Beta | 1.27 | 1.27 |
| `APISelfSubjectReview` | `true` | GA | 1.28 | - |
| `CPUManager` | `false` | Alpha | 1.8 | 1.9 |
| `CPUManager` | `true` | Beta | 1.10 | 1.25 |
| `CPUManager` | `true` | GA | 1.26 | - |
@ -244,6 +239,8 @@ For a reference to old feature gates that are removed, please refer to
| `CSIMigrationGCE` | `false` | Beta | 1.17 | 1.22 |
| `CSIMigrationGCE` | `true` | Beta | 1.23 | 1.24 |
| `CSIMigrationGCE` | `true` | GA | 1.25 | - |
| `CSIMigrationRBD` | `false` | Alpha | 1.23 | 1.27 |
| `CSIMigrationRBD` | `false` | Deprecated | 1.28 | - |
| `CSIMigrationvSphere` | `false` | Alpha | 1.18 | 1.18 |
| `CSIMigrationvSphere` | `false` | Beta | 1.19 | 1.24 |
| `CSIMigrationvSphere` | `true` | Beta | 1.25 | 1.25 |
@ -255,12 +252,9 @@ For a reference to old feature gates that are removed, please refer to
| `CronJobTimeZone` | `false` | Alpha | 1.24 | 1.24 |
| `CronJobTimeZone` | `true` | Beta | 1.25 | 1.26 |
| `CronJobTimeZone` | `true` | GA | 1.27 | - |
| `DelegateFSGroupToCSIDriver` | `false` | Alpha | 1.22 | 1.22 |
| `DelegateFSGroupToCSIDriver` | `true` | Beta | 1.23 | 1.25 |
| `DelegateFSGroupToCSIDriver` | `true` | GA | 1.26 |-|
| `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 |
| `DevicePlugins` | `true` | Beta | 1.10 | 1.25 |
| `DevicePlugins` | `true` | GA | 1.26 | - |
| `DaemonSetUpdateSurge` | `false` | Alpha | 1.21 | 1.21 |
| `DaemonSetUpdateSurge` | `true` | Beta | 1.22 | 1.24 |
| `DaemonSetUpdateSurge` | `true` | GA | 1.25 | - |
| `DisableAcceleratorUsageMetrics` | `false` | Alpha | 1.19 | 1.19 |
| `DisableAcceleratorUsageMetrics` | `true` | Beta | 1.20 | 1.24 |
| `DisableAcceleratorUsageMetrics` | `true` | GA | 1.25 |- |
@ -278,32 +272,60 @@ For a reference to old feature gates that are removed, please refer to
| `EndpointSliceTerminatingCondition` | `true` | Beta | 1.22 | 1.25 |
| `EndpointSliceTerminatingCondition` | `true` | GA | 1.26 | |
| `ExecProbeTimeout` | `true` | GA | 1.20 | - |
| `ExpandedDNSConfig` | `false` | Alpha | 1.22 | 1.25 |
| `ExpandedDNSConfig` | `true` | Beta | 1.26 | 1.27 |
| `ExpandedDNSConfig` | `true` | GA | 1.28 | - |
| `GRPCContainerProbe` | `false` | Alpha | 1.23 | 1.23 |
| `GRPCContainerProbe` | `true` | Beta | 1.24 | 1.26 |
| `GRPCContainerProbe` | `true` | GA | 1.27 | |
| `IPTablesOwnershipCleanup` | `false` | Alpha | 1.25 | 1.26 |
| `IPTablesOwnershipCleanup` | `true` | Beta | 1.27 | 1.27 |
| `IPTablesOwnershipCleanup` | `true` | GA | 1.28 | - |
| `JobMutableNodeSchedulingDirectives` | `true` | Beta | 1.23 | 1.26 |
| `JobMutableNodeSchedulingDirectives` | `true` | GA | 1.27 | |
| `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | 1.22 |
| `JobTrackingWithFinalizers` | `false` | Beta | 1.23 | 1.24 |
| `JobTrackingWithFinalizers` | `true` | Beta | 1.25 | 1.25 |
| `JobTrackingWithFinalizers` | `true` | GA | 1.26 | - |
| `KMSv1` | `true` | Deprecated | 1.28 | - |
| `KubeletCredentialProviders` | `false` | Alpha | 1.20 | 1.23 |
| `KubeletCredentialProviders` | `true` | Beta | 1.24 | 1.25 |
| `KubeletCredentialProviders` | `true` | GA | 1.26 | - |
| `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 |
| `KubeletPodResources` | `true` | Beta | 1.15 | 1.27 |
| `KubeletPodResources` | `true` | GA | 1.28 | - |
| `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | 1.22 |
| `KubeletPodResourcesGetAllocatable` | `true` | Beta | 1.23 | 1.27 |
| `KubeletPodResourcesGetAllocatable` | `true` | GA | 1.28 | - |
| `LegacyServiceAccountTokenNoAutoGeneration` | `true` | Beta | 1.24 | 1.25 |
| `LegacyServiceAccountTokenNoAutoGeneration` | `true` | GA | 1.26 | - |
| `LegacyServiceAccountTokenTracking` | `false` | Alpha | 1.26 | 1.26 |
| `LegacyServiceAccountTokenTracking` | `true` | Beta | 1.27 | 1.27 |
| `LegacyServiceAccountTokenTracking` | `true` | GA | 1.28 | - |
| `MinimizeIPTablesRestore` | `false` | Alpha | 1.26 | 1.26 |
| `MinimizeIPTablesRestore` | `true` | Beta | 1.27 | 1.27 |
| `MinimizeIPTablesRestore` | `true` | GA | 1.28 | - |
| `MixedProtocolLBService` | `false` | Alpha | 1.20 | 1.23 |
| `MixedProtocolLBService` | `true` | Beta | 1.24 | 1.25 |
| `MixedProtocolLBService` | `true` | GA | 1.26 | - |
| `NodeOutOfServiceVolumeDetach` | `false` | Alpha | 1.24 | 1.25 |
| `NodeOutOfServiceVolumeDetach` | `true` | Beta | 1.26 | 1.27 |
| `NodeOutOfServiceVolumeDetach` | `true` | GA | 1.28 | - |
| `OpenAPIV3` | `false` | Alpha | 1.23 | 1.23 |
| `OpenAPIV3` | `true` | Beta | 1.24 | 1.26 |
| `OpenAPIV3` | `true` | GA | 1.27 | - |
| `PodSecurity` | `false` | Alpha | 1.22 | 1.22 |
| `PodSecurity` | `true` | Beta | 1.23 | 1.24 |
| `PodSecurity` | `true` | GA | 1.25 | |
| `ProxyTerminatingEndpoints` | `false` | Alpha | 1.22 | 1.25 |
| `ProxyTerminatingEndpoints` | `true` | Beta | 1.26 | 1.27 |
| `ProxyTerminatingEndpoints` | `true` | GA | 1.28 | - |
| `RemoveSelfLink` | `false` | Alpha | 1.16 | 1.19 |
| `RemoveSelfLink` | `true` | Beta | 1.20 | 1.23 |
| `RemoveSelfLink` | `true` | GA | 1.24 | - |
| `RetroactiveDefaultStorageClass` | `false` | Alpha | 1.25 | 1.25 |
| `RetroactiveDefaultStorageClass` | `true` | Beta | 1.26 | 1.27 |
| `RetroactiveDefaultStorageClass` | `true` | GA | 1.28 | - |
| `SeccompDefault` | `false` | Alpha | 1.22 | 1.24 |
| `SeccompDefault` | `true` | Beta | 1.25 | 1.26 |
| `SeccompDefault` | `true` | GA | 1.27 | - |
@ -387,7 +409,6 @@ Each feature gate is designed for enabling/disabling a specific feature:
- `APIServerIdentity`: Assign each API server an ID in a cluster, using a [Lease](/docs/concepts/architecture/leases).
- `APIServerTracing`: Add support for distributed tracing in the API server.
See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/system-traces) for more details.
- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit)
- `AggregatedDiscoveryEndpoint`: Enable a single HTTP endpoint `/discovery/<version>` which
supports native HTTP caching with ETags containing all APIResources known to the API server.
- `AnyVolumeDataSource`: Enable use of any custom resource as the `DataSource` of a
@ -452,11 +473,15 @@ Each feature gate is designed for enabling/disabling a specific feature:
allowing you to scrape health check metrics.
- `ConsistentHTTPGetHandlers`: Normalize HTTP get URL and Header passing for lifecycle
handlers with probers.
- `ConsistentListFromCache`: Allow the API server to serve consistent lists from cache.
- `ContainerCheckpoint`: Enables the kubelet `checkpoint` API.
See [Kubelet Checkpoint API](/docs/reference/node/kubelet-checkpoint-api/) for more details.
- `ContextualLogging`: When you enable this feature gate, Kubernetes components that support
contextual logging add extra detail to log output.
- `CronJobTimeZone`: Allow the use of the `timeZone` optional field in [CronJobs](/docs/concepts/workloads/controllers/cron-jobs/)
- `CRDValidationRatcheting`: Enable updates to custom resources to contain
violations of their OpenAPI schema if the offending portions of the resource
update did not change. See [Validation Ratcheting](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-ratcheting) for more details.
- `CrossNamespaceVolumeDataSource`: Enable the usage of cross namespace volume data source
to allow you to specify a source namespace in the `dataSourceRef` field of a
PersistentVolumeClaim.
@ -465,11 +490,9 @@ Each feature gate is designed for enabling/disabling a specific feature:
- `CustomResourceValidationExpressions`: Enable expression language validation in CRD
which will validate customer resource based on validation rules written in
the `x-kubernetes-validations` extension.
- `DelegateFSGroupToCSIDriver`: If supported by the CSI driver, delegates the
role of applying `fsGroup` from a Pod's `securityContext` to the driver by
passing `fsGroup` through the NodeStageVolume and NodePublishVolume CSI calls.
- `DevicePlugins`: Enable the [device-plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
based resource provisioning on nodes.
- `DaemonSetUpdateSurge`: Enables the DaemonSet workloads to maintain
availability during update per node.
See [Perform a Rolling Update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/).
- `DisableAcceleratorUsageMetrics`:
[Disable accelerator metrics collected by the kubelet](/docs/concepts/cluster-administration/system-metrics/#disable-accelerator-metrics).
- `DisableCloudProviders`: Disables any functionality in `kube-apiserver`,
@ -549,8 +572,11 @@ Each feature gate is designed for enabling/disabling a specific feature:
and volume controllers.
- `JobMutableNodeSchedulingDirectives`: Allows updating node scheduling directives in
the pod template of [Job](/docs/concepts/workloads/controllers/job).
- `JobBackoffLimitPerIndex`: Allows specifying the maximal number of pod
retries per index in Indexed jobs.
- `JobPodFailurePolicy`: Allow users to specify handling of pod failures based on container
exit codes and pod conditions.
- `JobPodReplacementPolicy`: Allows you to specify pod replacement for terminating pods in a [Job](/docs/concepts/workloads/controllers/job)
- `JobReadyPods`: Enables tracking the number of Pods that have a `Ready`
[condition](/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions).
The count of `Ready` pods is recorded in the
@ -560,7 +586,22 @@ Each feature gate is designed for enabling/disabling a specific feature:
completions without relying on Pods remaining in the cluster indefinitely.
The Job controller uses Pod finalizers and a field in the Job status to keep
track of the finished Pods to count towards completion.
- `KMSv1`: Enables KMS v1 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details.
- `KMSv2`: Enables KMS v2 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details.
- `KMSv2KDF`: Enables KMS v2 to generate single use data encryption keys.
See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details.
If the `KMSv2` feature gate is not enabled in your cluster, the value of the `KMSv2KDF` feature gate has no effect.
- `KubeletCgroupDriverFromCRI`: Enable detection of the kubelet cgroup driver
configuration option from the {{<glossary_tooltip term_id="cri" text="CRI">}}.
You can use this feature gate on nodes with a kubelet that supports the feature gate
and where there is a CRI container runtime that supports the `RuntimeConfig`
CRI call. If both CRI and kubelet support this feature, the kubelet ignores the
`cgroupDriver` configuration setting (or deprecated `--cgroup-driver` command
line argument). If you enable this feature gate and the container runtime
doesn't support it, the kubelet falls back to using the driver configured using
the `cgroupDriver` configuration setting.
See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver)
for more details.
- `KubeletCredentialProviders`: Enable kubelet exec credential providers for
image pull credentials.
- `KubeletInUserNamespace`: Enables support for running kubelet in a
@ -584,9 +625,12 @@ Each feature gate is designed for enabling/disabling a specific feature:
OpenTelemetry trace spans.
See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/system-traces) for more details.
- `LegacyServiceAccountTokenNoAutoGeneration`: Stop auto-generation of Secret-based
[service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens).
[service account tokens](/docs/concepts/security/service-accounts/#get-a-token).
- `LegacyServiceAccountTokenCleanUp`: Enable cleaning up Secret-based
[service account tokens](/docs/concepts/security/service-accounts/#get-a-token)
when they are not used in a specified time (default to be one year).
- `LegacyServiceAccountTokenTracking`: Track usage of Secret-based
[service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens).
[service account tokens](/docs/concepts/security/service-accounts/#get-a-token).
- `LocalStorageCapacityIsolationFSQuotaMonitoring`: When `LocalStorageCapacityIsolation`
is enabled for
[local ephemeral storage](/docs/concepts/configuration/manage-resources-containers/)
@ -650,7 +694,11 @@ Each feature gate is designed for enabling/disabling a specific feature:
- `PodAndContainerStatsFromCRI`: Configure the kubelet to gather container and pod stats from the CRI container runtime rather than gathering them from cAdvisor.
As of 1.26, this also includes gathering metrics from CRI and emitting them over `/metrics/cadvisor` (rather than having cAdvisor emit them directly).
- `PodDisruptionConditions`: Enables support for appending a dedicated pod condition indicating that the pod is being deleted due to a disruption.
- `PodHasNetworkCondition`: Enable the kubelet to mark the [PodHasNetwork](/docs/concepts/workloads/pods/pod-lifecycle/#pod-has-network) condition on pods.
- `PodHostIPs`: Enable the `status.hostIPs` field for pods and the {{< glossary_tooltip term_id="downward-api" text="downward API" >}}.
The field lets you expose host IP addresses to workloads.
- `PodIndexLabel`: Enables the Job controller and StatefulSet controller to add the pod index as a label when creating new pods. See [Job completion mode docs](/docs/concepts/workloads/controllers/job#completion-mode) and [StatefulSet pod index label docs](/docs/concepts/workloads/controllers/statefulset/#pod-index-label) for more details.
- `PodReadyToStartContainersCondition`: Enable the kubelet to mark the [PodReadyToStartContainers](/docs/concepts/workloads/pods/pod-lifecycle/#pod-has-network)
condition on pods. This was previously (1.25-1.27) known as `PodHasNetworkCondition`.
- `PodSchedulingReadiness`: Enable setting `schedulingGates` field to control a Pod's [scheduling readiness](/docs/concepts/scheduling-eviction/pod-scheduling-readiness).
- `PodSecurity`: Enables the `PodSecurity` admission plugin.
- `ProbeTerminationGracePeriod`: Enable [setting probe-level
@ -700,6 +748,11 @@ Each feature gate is designed for enabling/disabling a specific feature:
risk of collision. See
[Avoiding collisions](/docs/reference/networking/virtual-ips/#avoiding-collisions)
for more details.
- `SidecarContainers`: Allow setting the `restartPolicy` of an init container to
`Always` so that the container becomes a sidecar container (restartable init containers).
See
[Sidecar containers and restartPolicy](/docs/concepts/workloads/pods/init-containers/#sidecar-containers-and-restartpolicy)
for more details.
- `SizeMemoryBackedVolumes`: Enable kubelets to determine the size limit for
memory-backed volumes (mainly `emptyDir` volumes).
- `StableLoadBalancerNodeSet`: Enables less load balancer re-configurations by
@ -728,7 +781,11 @@ Each feature gate is designed for enabling/disabling a specific feature:
This feature gate guards *a group* of topology manager options whose quality level is beta.
This feature gate will never graduate to stable.
- `TopologyManagerPolicyOptions`: Allow fine-tuning of topology manager policies,
- `UserNamespacesStatelessPodsSupport`: Enable user namespace support for stateless Pods.
- `UnknownVersionInteroperabilityProxy`: Proxy resource requests to the correct peer kube-apiserver when
multiple kube-apiservers exist at varied versions.
See [Mixed version proxy](/docs/concepts/architecture/mixed-version-proxy/) for more information.
- `UserNamespacesStatelessPodsSupport`: Enable user namespace support for stateless Pods. This flag was renamed on newer releases to `UserNamespacesSupport`.
- `UserNamespacesSupport`: Enable user namespace support for Pods.
- `ValidatingAdmissionPolicy`: Enable [ValidatingAdmissionPolicy](/docs/reference/access-authn-authz/validating-admission-policy/) support for CEL validations be used in Admission Control.
- `VolumeCapacityPriority`: Enable support for prioritizing nodes in different
topologies based on available PV capacity.
@ -748,4 +805,4 @@ Each feature gate is designed for enabling/disabling a specific feature:
feature, you will also need to enable any associated API resources.
For example, to enable a particular resource like
`storage.k8s.io/v1beta1/csistoragecapacities`, set `--runtime-config=storage.k8s.io/v1beta1/csistoragecapacities`.
See [API Versioning](/docs/reference/using-api/#api-versioning) for more details on the command line flags.
See [API Versioning](/docs/reference/using-api/#api-versioning) for more details on the command line flags.

21
content/en/docs/reference/glossary/mixed-version-proxy.md Normal file
View File

@ -0,0 +1,21 @@
---
title: Mixed Version Proxy (MVP)
id: mvp
date: 2023-07-24
full_link: /docs/concepts/architecture/mixed-version-proxy/
short_description: >
Feature that lets a kube-apiserver proxy a resource request to a different peer API server.
aka: ["MVP"]
tags:
- architecture
---
Feature to let a kube-apiserver proxy a resource request to a different peer API server.
<!--more-->
When a cluster has multiple API servers running different versions of Kubernetes, this
feature enables resource requests to be served by the correct API server.
MVP is disabled by default and can be activated by enabling
the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) named `UnknownVersionInteroperabilityProxy` when
the {{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}} is started.

402
content/en/docs/reference/instrumentation/metrics.md
View File

@ -6,16 +6,16 @@ description: >-
Details of the metric data that Kubernetes components export.
---
## Metrics (v1.27)
## Metrics (v1.28)
<!-- (auto-generated 2023 Mar 16) -->
<!-- (auto-generated v1.27) -->
<!-- (auto-generated 2023 Jul 20) -->
<!-- (auto-generated v1.28) -->
This page details the metrics that different Kubernetes components export. You can query the metrics endpoint for these
components using an HTTP scrape, and fetch the current metrics data in Prometheus format.
### List of Stable Kubernetes Metrics
Stable metrics observe strict API contracts and no labels can be added or removed from stable metrics during their lifetime.
Stable metrics observe strict API contracts and no labels can be added or removed from stable metrics during their lifetime.
<table class="table metrics" caption="This is the list of STABLE metrics emitted from core Kubernetes components">
<thead>
@ -225,8 +225,7 @@ Stable metrics observe strict API contracts and no labels can be added or remove
### List of Beta Kubernetes Metrics
Beta metrics observe a looser API contract than its stable counterparts. No labels can be removed from beta metrics during their lifetime, however, labels can be added while the metric is in the beta stage. This offers the assurance that beta metrics will honor existing dashboards and alerts, while allowing for amendments in the future.
Beta metrics observe a looser API contract than its stable counterparts. No labels can be removed from beta metrics during their lifetime, however, labels can be added while the metric is in the beta stage. This offers the assurance that beta metrics will honor existing dashboards and alerts, while allowing for amendments in the future.
<table class="table metrics" caption="This is the list of BETA metrics emitted from core Kubernetes components">
<thead>
@ -242,12 +241,103 @@ Beta metrics observe a looser API contract than its stable counterparts. No labe
</thead>
<tbody>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_current_executing_requests</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of requests in initial (for a WATCH) or any (for a non-WATCH) execution stage in the API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_current_executing_seats</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Concurrency (number of seats) occupied by the currently executing (initial stage for a WATCH, any stage otherwise) requests in the API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_current_inqueue_requests</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of requests currently pending in queues of the API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_dispatched_requests_total</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of requests executed by API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_nominal_limit_seats</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Nominal number of execution seats configured for each priority level</td>
<td class="metric_labels_varying"><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_rejected_requests_total</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of requests rejected by API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div><div class="metric_label">reason</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_request_wait_duration_seconds</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Length of time a request spent waiting in its queue</td>
<td class="metric_labels_varying"><div class="metric_label">execute</div><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">disabled_metrics_total</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">The count of disabled metrics.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">hidden_metrics_total</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">The count of hidden metrics.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubernetes_feature_enabled</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">This metric records the data about the stage and enablement of a k8s feature.</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">stage</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubernetes_healthcheck</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">This metric records the result of a single healthcheck.</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubernetes_healthchecks_total</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">This metric records the results of all healthcheck.</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">status</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">registered_metrics_total</td>
<td class="metric_stability_level" data-stability="beta">BETA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">The count of registered metrics broken by stability level and deprecation version.</td>
<td class="metric_labels_varying"><div class="metric_label">deprecated_version</div><div class="metric_label">stability_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
</tbody>
</table>
### List of Alpha Kubernetes Metrics
Alpha metrics do not have any API guarantees. These metrics must be used at your own risk, subsequent versions of Kubernetes may remove these metrics altogether, or mutate the API in such a way that breaks existing dashboards and alerts.
Alpha metrics do not have any API guarantees. These metrics must be used at your own risk, subsequent versions of Kubernetes may remove these metrics altogether, or mutate the API in such a way that breaks existing dashboards and alerts.
<table class="table metrics" caption="This is the list of ALPHA metrics emitted from core Kubernetes components">
<thead>
@ -312,11 +402,25 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">crd</div><div class="metric_label">group</div><div class="metric_label">reason</div><div class="metric_label">version</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_admission_admission_match_condition_evaluation_errors_total</td>
<tr class="metric"><td class="metric_name">apiserver_admission_match_condition_evaluation_errors_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Admission match condition evaluation errors count, identified by name of resource containing the match condition and broken out for each admission type (validating or mutating).</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">type</div></td>
<td class="metric_description">Admission match condition evaluation errors count, identified by name of resource containing the match condition and broken out for each kind containing matchConditions (webhook or policy), operation and admission type (validate or admit).</td>
<td class="metric_labels_varying"><div class="metric_label">kind</div><div class="metric_label">name</div><div class="metric_label">operation</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_admission_match_condition_evaluation_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Admission match condition evaluation time in seconds, identified by name and broken out for each kind containing matchConditions (webhook or policy), operation and type (validate or admit).</td>
<td class="metric_labels_varying"><div class="metric_label">kind</div><div class="metric_label">name</div><div class="metric_label">operation</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_admission_match_condition_exclusions_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Admission match condition evaluation exclusions count, identified by name of resource containing the match condition and broken out for each kind containing matchConditions (webhook or policy), operation and admission type (validate or admit).</td>
<td class="metric_labels_varying"><div class="metric_label">kind</div><div class="metric_label">name</div><div class="metric_label">operation</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_admission_step_admission_duration_seconds_summary</td>
@ -431,7 +535,21 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_crd_webhook_conversion_duration_seconds</td>
<tr class="metric"><td class="metric_name">apiserver_conversion_webhook_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Conversion webhook request latency</td>
<td class="metric_labels_varying"><div class="metric_label">failure_type</div><div class="metric_label">result</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_conversion_webhook_request_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Counter for conversion webhook requests with success/failure and failure error type</td>
<td class="metric_labels_varying"><div class="metric_label">failure_type</div><div class="metric_label">result</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_crd_conversion_webhook_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">CRD webhook conversion duration in seconds</td>
@ -494,6 +612,27 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">protocol</div><div class="metric_label">transport</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_encryption_config_controller_automatic_reload_failures_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Total number of failed automatic reloads of encryption configuration.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Timestamp of the last successful or failed automatic reload of encryption configuration.</td>
<td class="metric_labels_varying"><div class="metric_label">status</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_encryption_config_controller_automatic_reload_success_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Total number of successful automatic reloads of encryption configuration.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_envelope_encryption_dek_cache_fill_percent</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -543,20 +682,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">grpc_status_code</div><div class="metric_label">method_name</div><div class="metric_label">provider_name</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_current_executing_requests</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of requests in initial (for a WATCH) or any (for a non-WATCH) execution stage in the API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_current_inqueue_requests</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of requests currently pending in queues of the API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_current_limit_seats</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -613,13 +738,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_dispatched_requests_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of requests executed by API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_epoch_advance_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -655,13 +773,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">bound</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_nominal_limit_seats</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Nominal number of execution seats configured for each priority level</td>
<td class="metric_labels_varying"><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_priority_level_request_utilization</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="timingratiohistogram">TimingRatioHistogram</td>
@ -683,27 +794,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">phase</div><div class="metric_label">request_kind</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_rejected_requests_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of requests rejected by API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div><div class="metric_label">reason</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_request_concurrency_in_use</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Concurrency (number of seats) occupied by the currently executing (initial stage for a WATCH, any stage otherwise) requests in the API Priority and Fairness subsystem</td>
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<td class="metric_deprecated_version">1.31.0</td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_request_concurrency_limit</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Shared concurrency limit in the API Priority and Fairness subsystem</td>
<td class="metric_description">Nominal number of execution seats configured for each priority level</td>
<td class="metric_labels_varying"><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<td class="metric_deprecated_version">1.30.0</td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_request_dispatch_no_accommodation_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -725,13 +829,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_request_wait_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Length of time a request spent waiting in its queue</td>
<td class="metric_labels_varying"><div class="metric_label">execute</div><div class="metric_label">flow_schema</div><div class="metric_label">priority_level</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_flowcontrol_seat_fair_frac</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -844,6 +941,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">code_path</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_rerouted_request_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Total number of requests that were proxied to a peer kube apiserver because the local apiserver was not capable of serving it</td>
<td class="metric_labels_varying"><div class="metric_label">code</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_selfrequest_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -871,7 +975,7 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_description">Total size of the storage database file physically allocated in bytes.</td>
<td class="metric_labels_varying"><div class="metric_label">endpoint</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<td class="metric_deprecated_version">1.28.0</td></tr>
<tr class="metric"><td class="metric_name">apiserver_storage_decode_errors_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -921,6 +1025,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">resource</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_storage_size_bytes</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="custom">Custom</td>
<td class="metric_description">Size of the storage database file physically allocated in bytes.</td>
<td class="metric_labels_varying"><div class="metric_label">cluster</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">apiserver_storage_transformation_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
@ -931,7 +1042,7 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<tr class="metric"><td class="metric_name">apiserver_storage_transformation_operations_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Total number of transformations.</td>
<td class="metric_description">Total number of transformations. Successful transformation will have a status 'OK' and a varied status string when the transformation fails. This status and transformation_type fields may be used for alerting on encryption/decryption failure using transformation_type from_storage for decryption and to_storage for encryption</td>
<td class="metric_labels_varying"><div class="metric_label">status</div><div class="metric_label">transformation_type</div><div class="metric_label">transformer_prefix</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
@ -1019,11 +1130,11 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">attachdetach_controller_forced_detaches</td>
<tr class="metric"><td class="metric_name">attach_detach_controller_attachdetach_controller_forced_detaches</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of times the A/D Controller performed a forced detach</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_varying"><div class="metric_label">reason</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">attachdetach_controller_total_volumes</td>
@ -1082,6 +1193,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">status</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">authorization_attempts_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Counter of authorization attempts broken down by result. It can be either 'allowed', 'denied', 'no-opinion' or 'error'.</td>
<td class="metric_labels_varying"><div class="metric_label">result</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">authorization_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Authorization duration in seconds broken out by result.</td>
<td class="metric_labels_varying"><div class="metric_label">result</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">cloud_provider_webhook_request_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
@ -1208,6 +1333,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">container</div><div class="metric_label">pod</div><div class="metric_label">namespace</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">container_swap_usage_bytes</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="custom">Custom</td>
<td class="metric_description">Current amount of the container swap usage in bytes. Reported only on non-windows systems</td>
<td class="metric_labels_varying"><div class="metric_label">container</div><div class="metric_label">pod</div><div class="metric_label">namespace</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">csi_operations_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
@ -1369,6 +1501,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">operation</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">etcd_request_errors_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Etcd failed request counts for each operation and object type.</td>
<td class="metric_labels_varying"><div class="metric_label">operation</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">etcd_requests_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Etcd request counts for each operation and object type.</td>
<td class="metric_labels_varying"><div class="metric_label">operation</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">etcd_version_info</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -1495,20 +1641,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kube_apiserver_nodeport_allocator_allocation_errors_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of errors trying to allocate NodePort</td>
<td class="metric_labels_varying"><div class="metric_label">scope</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kube_apiserver_nodeport_allocator_allocation_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of NodePort allocations</td>
<td class="metric_labels_varying"><div class="metric_label">scope</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kube_apiserver_nodeport_allocator_available_ports</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -1967,14 +2099,14 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<tr class="metric"><td class="metric_name">kubelet_started_host_process_containers_errors_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Cumulative number of errors when starting hostprocess containers. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.</td>
<td class="metric_description">Cumulative number of errors when starting hostprocess containers. This metric will only be collected on Windows.</td>
<td class="metric_labels_varying"><div class="metric_label">code</div><div class="metric_label">container_type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubelet_started_host_process_containers_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Cumulative number of hostprocess containers started. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.</td>
<td class="metric_description">Cumulative number of hostprocess containers started. This metric will only be collected on Windows.</td>
<td class="metric_labels_varying"><div class="metric_label">container_type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
@ -2083,6 +2215,34 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_proxy_healthz_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Cumulative proxy healthz HTTP status</td>
<td class="metric_labels_varying"><div class="metric_label">code</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_proxy_livez_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Cumulative proxy livez HTTP status</td>
<td class="metric_labels_varying"><div class="metric_label">code</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_sync_full_proxy_rules_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">SyncProxyRules latency in seconds for full resyncs</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_sync_partial_proxy_rules_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">SyncProxyRules latency in seconds for partial resyncs</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_sync_proxy_rules_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
@ -2104,6 +2264,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_sync_proxy_rules_iptables_last</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of iptables rules written by kube-proxy in last sync</td>
<td class="metric_labels_varying"><div class="metric_label">table</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubeproxy_sync_proxy_rules_iptables_partial_restore_failures_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -2121,7 +2288,7 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<tr class="metric"><td class="metric_name">kubeproxy_sync_proxy_rules_iptables_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of proxy iptables rules programmed</td>
<td class="metric_description">Total number of iptables rules owned by kube-proxy</td>
<td class="metric_labels_varying"><div class="metric_label">table</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
@ -2167,27 +2334,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">build_date</div><div class="metric_label">compiler</div><div class="metric_label">git_commit</div><div class="metric_label">git_tree_state</div><div class="metric_label">git_version</div><div class="metric_label">go_version</div><div class="metric_label">major</div><div class="metric_label">minor</div><div class="metric_label">platform</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubernetes_feature_enabled</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">This metric records the data about the stage and enablement of a k8s feature.</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">stage</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubernetes_healthcheck</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">This metric records the result of a single healthcheck.</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">kubernetes_healthchecks_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">This metric records the results of all healthcheck.</td>
<td class="metric_labels_varying"><div class="metric_label">name</div><div class="metric_label">status</div><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">leader_election_master_status</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -2237,6 +2383,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">zone</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">node_controller_cloud_provider_taint_removal_delay_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Number of seconds after node creation when NodeController removed the cloud-provider taint of a single node.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">node_controller_initial_node_sync_delay_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">Number of seconds after node creation when NodeController finished the initial synchronization of a single node.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">node_cpu_usage_seconds_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="custom">Custom</td>
@ -2321,6 +2481,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">node_swap_usage_bytes</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="custom">Custom</td>
<td class="metric_description">Current swap usage of the node in bytes. Reported only on non-windows systems</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">number_of_l4_ilbs</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
@ -2346,14 +2513,14 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of errors encountered when forcefully deleting the pods since the Pod GC Controller started.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_varying"><div class="metric_label">namespace</div><div class="metric_label">reason</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">pod_gc_collector_force_delete_pods_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of pods that are being forcefully deleted since the Pod GC Controller started.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_varying"><div class="metric_label">namespace</div><div class="metric_label">reason</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">pod_memory_working_set_bytes</td>
@ -2384,6 +2551,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">request_operation</div><div class="metric_label">resource</div><div class="metric_label">subresource</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">pod_swap_usage_bytes</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="custom">Custom</td>
<td class="metric_description">Current amount of the pod swap usage in bytes. Reported only on non-windows systems</td>
<td class="metric_labels_varying"><div class="metric_label">pod</div><div class="metric_label">namespace</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">prober_probe_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
@ -2468,6 +2642,13 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">rest_client_dns_resolution_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>
<td class="metric_description">DNS resolver latency in seconds. Broken down by host.</td>
<td class="metric_labels_varying"><div class="metric_label">host</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">rest_client_exec_plugin_call_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -2531,6 +2712,20 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">host</div><div class="metric_label">verb</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">rest_client_transport_cache_entries</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of transport entries in the internal cache.</td>
<td class="metric_labels_varying"></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">rest_client_transport_create_calls_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
<td class="metric_description">Number of calls to get a new transport, partitioned by the result of the operation hit: obtained from the cache, miss: created and added to the cache, uncacheable: created and not cached</td>
<td class="metric_labels_varying"><div class="metric_label">result</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">retroactive_storageclass_errors_total</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="counter">Counter</td>
@ -2601,13 +2796,6 @@ Alpha metrics do not have any API guarantees. These metrics must be used at your
<td class="metric_labels_varying"><div class="metric_label">type</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version"></td></tr>
<tr class="metric"><td class="metric_name">scheduler_scheduler_goroutines</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="gauge">Gauge</td>
<td class="metric_description">Number of running goroutines split by the work they do such as binding. This metric is replaced by the \"goroutines\" metric.</td>
<td class="metric_labels_varying"><div class="metric_label">work</div></td>
<td class="metric_labels_constant"></td>
<td class="metric_deprecated_version">1.26.0</td></tr>
<tr class="metric"><td class="metric_name">scheduler_scheduling_algorithm_duration_seconds</td>
<td class="metric_stability_level" data-stability="alpha">ALPHA</td>
<td class="metric_type" data-type="histogram">Histogram</td>

8
content/en/docs/reference/kubectl/kubectl.md
View File

@ -369,6 +369,14 @@ kubectl [flags]
</td>
</tr>
<tr>
<td colspan="2">KUBECTL_INTERACTIVE_DELETE</td>
</tr>
<tr>
<td></td><td style="line-height: 130%; word-wrap: break-word;">When set to true, the --interactive flag in the kubectl delete command will be activated, allowing users to preview and confirm resources before proceeding to delete by passing this flag.
</td>
</tr>
</tbody>
</table>

24
content/en/docs/reference/kubernetes-api/authentication-resources/self-subject-review-v1beta1.md → content/en/docs/reference/kubernetes-api/authentication-resources/self-subject-review-v1.md
View File

@ -1,11 +1,11 @@
---
api_metadata:
apiVersion: "authentication.k8s.io/v1beta1"
import: "k8s.io/api/authentication/v1beta1"
apiVersion: "authentication.k8s.io/v1"
import: "k8s.io/api/authentication/v1"
kind: "SelfSubjectReview"
content_type: "api_reference"
description: "SelfSubjectReview contains the user information that the kube-apiserver has about the user making this request."
title: "SelfSubjectReview v1beta1"
title: "SelfSubjectReview"
weight: 6
auto_generated: true
---
@ -21,9 +21,9 @@ guide. You can file document formatting bugs against the
[reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project.
-->
`apiVersion: authentication.k8s.io/v1beta1`
`apiVersion: authentication.k8s.io/v1`
`import "k8s.io/api/authentication/v1beta1"`
`import "k8s.io/api/authentication/v1"`
## SelfSubjectReview {#SelfSubjectReview}
@ -32,7 +32,7 @@ SelfSubjectReview contains the user information that the kube-apiserver has abou
<hr>
- **apiVersion**: authentication.k8s.io/v1beta1
- **apiVersion**: authentication.k8s.io/v1
- **kind**: SelfSubjectReview
@ -42,7 +42,7 @@ SelfSubjectReview contains the user information that the kube-apiserver has abou
Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
- **status** (<a href="{{< ref "../authentication-resources/self-subject-review-v1beta1#SelfSubjectReviewStatus" >}}">SelfSubjectReviewStatus</a>)
- **status** (<a href="{{< ref "../authentication-resources/self-subject-review-v1#SelfSubjectReviewStatus" >}}">SelfSubjectReviewStatus</a>)
Status is filled in by the server with the user attributes.
@ -98,12 +98,12 @@ SelfSubjectReviewStatus is filled by the kube-apiserver and sent back to a user.
#### HTTP Request
POST /apis/authentication.k8s.io/v1beta1/selfsubjectreviews
POST /apis/authentication.k8s.io/v1/selfsubjectreviews
#### Parameters
- **body**: <a href="{{< ref "../authentication-resources/self-subject-review-v1beta1#SelfSubjectReview" >}}">SelfSubjectReview</a>, required
- **body**: <a href="{{< ref "../authentication-resources/self-subject-review-v1#SelfSubjectReview" >}}">SelfSubjectReview</a>, required
@ -132,11 +132,11 @@ POST /apis/authentication.k8s.io/v1beta1/selfsubjectreviews
#### Response
200 (<a href="{{< ref "../authentication-resources/self-subject-review-v1beta1#SelfSubjectReview" >}}">SelfSubjectReview</a>): OK
200 (<a href="{{< ref "../authentication-resources/self-subject-review-v1#SelfSubjectReview" >}}">SelfSubjectReview</a>): OK
201 (<a href="{{< ref "../authentication-resources/self-subject-review-v1beta1#SelfSubjectReview" >}}">SelfSubjectReview</a>): Created
201 (<a href="{{< ref "../authentication-resources/self-subject-review-v1#SelfSubjectReview" >}}">SelfSubjectReview</a>): Created
202 (<a href="{{< ref "../authentication-resources/self-subject-review-v1beta1#SelfSubjectReview" >}}">SelfSubjectReview</a>): Accepted
202 (<a href="{{< ref "../authentication-resources/self-subject-review-v1#SelfSubjectReview" >}}">SelfSubjectReview</a>): Accepted
401: Unauthorized

4
content/en/docs/reference/kubernetes-api/authorization-resources/cluster-role-binding-v1.md
View File

@ -6,7 +6,7 @@ api_metadata:
content_type: "api_reference"
description: "ClusterRoleBinding references a ClusterRole, but not contain it."
title: "ClusterRoleBinding"
weight: 7
weight: 6
auto_generated: true
---
@ -44,7 +44,7 @@ ClusterRoleBinding references a ClusterRole, but not contain it. It can referen
- **roleRef** (RoleRef), required
RoleRef can only reference a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error.
RoleRef can only reference a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error. This field is immutable.
<a name="RoleRef"></a>
*RoleRef contains information that points to the role being used*

2
content/en/docs/reference/kubernetes-api/authorization-resources/cluster-role-v1.md
View File

@ -6,7 +6,7 @@ api_metadata:
content_type: "api_reference"
description: "ClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding."
title: "ClusterRole"
weight: 6
weight: 5
auto_generated: true
---

4
content/en/docs/reference/kubernetes-api/authorization-resources/role-binding-v1.md
View File

@ -6,7 +6,7 @@ api_metadata:
content_type: "api_reference"
description: "RoleBinding references a role, but does not contain it."
title: "RoleBinding"
weight: 9
weight: 8
auto_generated: true
---
@ -44,7 +44,7 @@ RoleBinding references a role, but does not contain it. It can reference a Role
- **roleRef** (RoleRef), required
RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error.
RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error. This field is immutable.
<a name="RoleRef"></a>
*RoleRef contains information that points to the role being used*

2
content/en/docs/reference/kubernetes-api/authorization-resources/role-v1.md
View File

@ -6,7 +6,7 @@ api_metadata:
content_type: "api_reference"
description: "Role is a namespaced, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding."
title: "Role"
weight: 8
weight: 7
auto_generated: true
---

142
content/en/docs/reference/kubernetes-api/authorization-resources/self-subject-review-v1alpha1.md
View File

@ -1,142 +0,0 @@
---
api_metadata:
apiVersion: "authentication.k8s.io/v1alpha1"
import: "k8s.io/api/authentication/v1alpha1"
kind: "SelfSubjectReview"
content_type: "api_reference"
description: "SelfSubjectReview contains the user information that the kube-apiserver has about the user making this request."
title: "SelfSubjectReview v1alpha1"
weight: 5
auto_generated: true
---
<!--
The file is auto-generated from the Go source code of the component using a generic
[generator](https://github.com/kubernetes-sigs/reference-docs/). To learn how
to generate the reference documentation, please read
[Contributing to the reference documentation](/docs/contribute/generate-ref-docs/).
To update the reference content, please follow the
[Contributing upstream](/docs/contribute/generate-ref-docs/contribute-upstream/)
guide. You can file document formatting bugs against the
[reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project.
-->
`apiVersion: authentication.k8s.io/v1alpha1`
`import "k8s.io/api/authentication/v1alpha1"`
## SelfSubjectReview {#SelfSubjectReview}
SelfSubjectReview contains the user information that the kube-apiserver has about the user making this request. When using impersonation, users will receive the user info of the user being impersonated. If impersonation or request header authentication is used, any extra keys will have their case ignored and returned as lowercase.
<hr>
- **apiVersion**: authentication.k8s.io/v1alpha1
- **kind**: SelfSubjectReview
- **metadata** (<a href="{{< ref "../common-definitions/object-meta#ObjectMeta" >}}">ObjectMeta</a>)
Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
- **status** (<a href="{{< ref "../authorization-resources/self-subject-review-v1alpha1#SelfSubjectReviewStatus" >}}">SelfSubjectReviewStatus</a>)
Status is filled in by the server with the user attributes.
## SelfSubjectReviewStatus {#SelfSubjectReviewStatus}
SelfSubjectReviewStatus is filled by the kube-apiserver and sent back to a user.
<hr>
- **userInfo** (UserInfo)
User attributes of the user making this request.
<a name="UserInfo"></a>
*UserInfo holds the information about the user needed to implement the user.Info interface.*
- **userInfo.extra** (map[string][]string)
Any additional information provided by the authenticator.
- **userInfo.groups** ([]string)
The names of groups this user is a part of.
- **userInfo.uid** (string)
A unique value that identifies this user across time. If this user is deleted and another user by the same name is added, they will have different UIDs.
- **userInfo.username** (string)
The name that uniquely identifies this user among all active users.
## Operations {#Operations}
<hr>
### `create` create a SelfSubjectReview
#### HTTP Request
POST /apis/authentication.k8s.io/v1alpha1/selfsubjectreviews
#### Parameters
- **body**: <a href="{{< ref "../authorization-resources/self-subject-review-v1alpha1#SelfSubjectReview" >}}">SelfSubjectReview</a>, required
- **dryRun** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#dryRun" >}}">dryRun</a>
- **fieldManager** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#fieldManager" >}}">fieldManager</a>
- **fieldValidation** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#fieldValidation" >}}">fieldValidation</a>
- **pretty** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#pretty" >}}">pretty</a>
#### Response
200 (<a href="{{< ref "../authorization-resources/self-subject-review-v1alpha1#SelfSubjectReview" >}}">SelfSubjectReview</a>): OK
201 (<a href="{{< ref "../authorization-resources/self-subject-review-v1alpha1#SelfSubjectReview" >}}">SelfSubjectReview</a>): Created
202 (<a href="{{< ref "../authorization-resources/self-subject-review-v1alpha1#SelfSubjectReview" >}}">SelfSubjectReview</a>): Accepted
401: Unauthorized

25
content/en/docs/reference/kubernetes-api/cluster-resources/priority-level-configuration-v1beta3.md
View File

@ -64,6 +64,27 @@ PriorityLevelConfigurationSpec specifies the configuration of a priority level.
`type` indicates whether this priority level is subject to limitation on request execution. A value of `"Exempt"` means that requests of this priority level are not subject to a limit (and thus are never queued) and do not detract from the capacity made available to other priority levels. A value of `"Limited"` means that (a) requests of this priority level _are_ subject to limits and (b) some of the server's limited capacity is made available exclusively to this priority level. Required.
- **exempt** (ExemptPriorityLevelConfiguration)
`exempt` specifies how requests are handled for an exempt priority level. This field MUST be empty if `type` is `"Limited"`. This field MAY be non-empty if `type` is `"Exempt"`. If empty and `type` is `"Exempt"` then the default values for `ExemptPriorityLevelConfiguration` apply.
<a name="ExemptPriorityLevelConfiguration"></a>
*ExemptPriorityLevelConfiguration describes the configurable aspects of the handling of exempt requests. In the mandatory exempt configuration object the values in the fields here can be modified by authorized users, unlike the rest of the `spec`.*
- **exempt.lendablePercent** (int32)
`lendablePercent` prescribes the fraction of the level's NominalCL that can be borrowed by other priority levels. This value of this field must be between 0 and 100, inclusive, and it defaults to 0. The number of seats that other levels can borrow from this level, known as this level's LendableConcurrencyLimit (LendableCL), is defined as follows.
LendableCL(i) = round( NominalCL(i) * lendablePercent(i)/100.0 )
- **exempt.nominalConcurrencyShares** (int32)
`nominalConcurrencyShares` (NCS) contributes to the computation of the NominalConcurrencyLimit (NominalCL) of this level. This is the number of execution seats nominally reserved for this priority level. This DOES NOT limit the dispatching from this priority level but affects the other priority levels through the borrowing mechanism. The server's concurrency limit (ServerCL) is divided among all the priority levels in proportion to their NCS values:
NominalCL(i) = ceil( ServerCL * NCS(i) / sum_ncs ) sum_ncs = sum[priority level k] NCS(k)
Bigger numbers mean a larger nominal concurrency limit, at the expense of every other priority level. This field has a default value of zero.
- **limited** (LimitedPriorityLevelConfiguration)
`limited` specifies how requests are handled for a Limited priority level. This field must be non-empty if and only if `type` is `"Limited"`.
@ -121,9 +142,9 @@ PriorityLevelConfigurationSpec specifies the configuration of a priority level.
`nominalConcurrencyShares` (NCS) contributes to the computation of the NominalConcurrencyLimit (NominalCL) of this level. This is the number of execution seats available at this priority level. This is used both for requests dispatched from this priority level as well as requests dispatched from other priority levels borrowing seats from this level. The server's concurrency limit (ServerCL) is divided among the Limited priority levels in proportion to their NCS values:
NominalCL(i) = ceil( ServerCL * NCS(i) / sum_ncs ) sum_ncs = sum[limited priority level k] NCS(k)
NominalCL(i) = ceil( ServerCL * NCS(i) / sum_ncs ) sum_ncs = sum[priority level k] NCS(k)
Bigger numbers mean a larger nominal concurrency limit, at the expense of every other Limited priority level. This field has a default value of 30.
Bigger numbers mean a larger nominal concurrency limit, at the expense of every other priority level. This field has a default value of 30.

2
content/en/docs/reference/kubernetes-api/common-definitions/label-selector.md
View File

@ -39,8 +39,6 @@ A label selector is a label query over a set of resources. The result of matchLa
- **matchExpressions.key** (string), required
*Patch strategy: merge on key `key`*
key is the label key that the selector applies to.
- **matchExpressions.operator** (string), required

49
content/en/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-claim-v1.md
View File

@ -163,9 +163,52 @@ PersistentVolumeClaimStatus is the current status of a persistent volume claim.
accessModes contains the actual access modes the volume backing the PVC has. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
- **allocatedResourceStatuses** (map[string]string)
allocatedResourceStatuses stores status of resource being resized for the given PVC. Key names follow standard Kubernetes label syntax. Valid values are either:
* Un-prefixed keys:
- storage - the capacity of the volume.
* Custom resources must use implementation-defined prefixed names such as "example.com/my-custom-resource"
Apart from above values - keys that are unprefixed or have kubernetes.io prefix are considered reserved and hence may not be used.
ClaimResourceStatus can be in any of following states:
- ControllerResizeInProgress:
State set when resize controller starts resizing the volume in control-plane.
- ControllerResizeFailed:
State set when resize has failed in resize controller with a terminal error.
- NodeResizePending:
State set when resize controller has finished resizing the volume but further resizing of
volume is needed on the node.
- NodeResizeInProgress:
State set when kubelet starts resizing the volume.
- NodeResizeFailed:
State set when resizing has failed in kubelet with a terminal error. Transient errors don't set
NodeResizeFailed.
For example: if expanding a PVC for more capacity - this field can be one of the following states:
- pvc.status.allocatedResourceStatus['storage'] = "ControllerResizeInProgress"
- pvc.status.allocatedResourceStatus['storage'] = "ControllerResizeFailed"
- pvc.status.allocatedResourceStatus['storage'] = "NodeResizePending"
- pvc.status.allocatedResourceStatus['storage'] = "NodeResizeInProgress"
- pvc.status.allocatedResourceStatus['storage'] = "NodeResizeFailed"
When this field is not set, it means that no resize operation is in progress for the given PVC.
A controller that receives PVC update with previously unknown resourceName or ClaimResourceStatus should ignore the update for the purpose it was designed. For example - a controller that only is responsible for resizing capacity of the volume, should ignore PVC updates that change other valid resources associated with PVC.
This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.
- **allocatedResources** (map[string]<a href="{{< ref "../common-definitions/quantity#Quantity" >}}">Quantity</a>)
allocatedResources is the storage resource within AllocatedResources tracks the capacity allocated to a PVC. It may be larger than the actual capacity when a volume expansion operation is requested. For storage quota, the larger value from allocatedResources and PVC.spec.resources is used. If allocatedResources is not set, PVC.spec.resources alone is used for quota calculation. If a volume expansion capacity request is lowered, allocatedResources is only lowered if there are no expansion operations in progress and if the actual volume capacity is equal or lower than the requested capacity. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.
allocatedResources tracks the resources allocated to a PVC including its capacity. Key names follow standard Kubernetes label syntax. Valid values are either:
* Un-prefixed keys:
- storage - the capacity of the volume.
* Custom resources must use implementation-defined prefixed names such as "example.com/my-custom-resource"
Apart from above values - keys that are unprefixed or have kubernetes.io prefix are considered reserved and hence may not be used.
Capacity reported here may be larger than the actual capacity when a volume expansion operation is requested. For storage quota, the larger value from allocatedResources and PVC.spec.resources is used. If allocatedResources is not set, PVC.spec.resources alone is used for quota calculation. If a volume expansion capacity request is lowered, allocatedResources is only lowered if there are no expansion operations in progress and if the actual volume capacity is equal or lower than the requested capacity.
A controller that receives PVC update with previously unknown resourceName should ignore the update for the purpose it was designed. For example - a controller that only is responsible for resizing capacity of the volume, should ignore PVC updates that change other valid resources associated with PVC.
This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.
- **capacity** (map[string]<a href="{{< ref "../common-definitions/quantity#Quantity" >}}">Quantity</a>)
@ -212,10 +255,6 @@ PersistentVolumeClaimStatus is the current status of a persistent volume claim.
phase represents the current phase of PersistentVolumeClaim.
- **resizeStatus** (string)
resizeStatus stores status of resize operation. ResizeStatus is not set by default but when expansion is complete resizeStatus is set to empty string by resize controller or kubelet. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.

7
content/en/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-v1.md
View File

@ -852,6 +852,13 @@ PersistentVolumeStatus is the current status of a persistent volume.
<hr>
- **lastPhaseTransitionTime** (Time)
lastPhaseTransitionTime is the time the phase transitioned from one to another and automatically resets to current time everytime a volume phase transitions. This is an alpha field and requires enabling PersistentVolumeLastPhaseTransitionTime feature.
<a name="Time"></a>
*Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.*
- **message** (string)
message is a human-readable message indicating details about why the volume is in this state.

8
content/en/docs/reference/kubernetes-api/extend-resources/custom-resource-definition-v1.md
View File

@ -529,6 +529,10 @@ JSONSchemaProps is a JSON-Schema following Specification Draft 4 (http://json-sc
are overwritten by values in `Y` when the key sets of `X` and `Y` intersect. Elements in `Y` with
non-intersecting keys are appended, retaining their partial order.
- **x-kubernetes-validations.fieldPath** (string)
fieldPath represents the field path returned when the validation fails. It must be a relative JSON path (i.e. with array notation) scoped to the location of this x-kubernetes-validations extension in the schema and refer to an existing field. e.g. when validation checks if a specific attribute `foo` under a map `testMap`, the fieldPath could be set to `.testMap.foo` If the validation checks two lists must have unique attributes, the fieldPath could be set to either of the list: e.g. `.testList` It does not support list numeric index. It supports child operation to refer to an existing field currently. Refer to [JSONPath support in Kubernetes](https://kubernetes.io/docs/reference/kubectl/jsonpath/) for more info. Numeric index of array is not supported. For field name which contains special characters, use `['specialName']` to refer the field name. e.g. for attribute `foo.34$` appears in a list `testList`, the fieldPath could be set to `.testList['foo.34$']`
- **x-kubernetes-validations.message** (string)
Message represents the message displayed when validation fails. The message is required if the Rule contains line breaks. The message must not contain line breaks. If unset, the message is "failed rule: {Rule}". e.g. "must be a URL with the host matching spec.host"
@ -537,6 +541,10 @@ JSONSchemaProps is a JSON-Schema following Specification Draft 4 (http://json-sc
MessageExpression declares a CEL expression that evaluates to the validation failure message that is returned when this rule fails. Since messageExpression is used as a failure message, it must evaluate to a string. If both message and messageExpression are present on a rule, then messageExpression will be used if validation fails. If messageExpression results in a runtime error, the runtime error is logged, and the validation failure message is produced as if the messageExpression field were unset. If messageExpression evaluates to an empty string, a string with only spaces, or a string that contains line breaks, then the validation failure message will also be produced as if the messageExpression field were unset, and the fact that messageExpression produced an empty string/string with only spaces/string with line breaks will be logged. messageExpression has access to all the same variables as the rule; the only difference is the return type. Example: "x must be less than max ("+string(self.max)+")"
- **x-kubernetes-validations.reason** (string)
reason provides a machine-readable validation failure reason that is returned to the caller when a request fails this validation rule. The HTTP status code returned to the caller will match the reason of the reason of the first failed validation rule. The currently supported reasons are: "FieldValueInvalid", "FieldValueForbidden", "FieldValueRequired", "FieldValueDuplicate". If not set, default to use "FieldValueInvalid". All future added reasons must be accepted by clients when reading this value and unknown reasons should be treated as FieldValueInvalid.

2
content/en/docs/reference/kubernetes-api/extend-resources/mutating-webhook-configuration-v1.md
View File

@ -132,7 +132,7 @@ MutatingWebhookConfiguration describes the configuration of and admission webhoo
- If failurePolicy=Fail, reject the request
- If failurePolicy=Ignore, the error is ignored and the webhook is skipped
This is an alpha feature and managed by the AdmissionWebhookMatchConditions feature gate.
This is a beta feature and managed by the AdmissionWebhookMatchConditions feature gate.
<a name="MatchCondition"></a>
*MatchCondition represents a condition which must by fulfilled for a request to be sent to a webhook.*

135
content/en/docs/reference/kubernetes-api/extend-resources/validating-admission-policy-v1alpha1.md → content/en/docs/reference/kubernetes-api/extend-resources/validating-admission-policy-v1beta1.md
View File

@ -1,11 +1,11 @@
---
api_metadata:
apiVersion: "admissionregistration.k8s.io/v1alpha1"
import: "k8s.io/api/admissionregistration/v1alpha1"
apiVersion: "admissionregistration.k8s.io/v1beta1"
import: "k8s.io/api/admissionregistration/v1beta1"
kind: "ValidatingAdmissionPolicy"
content_type: "api_reference"
description: "ValidatingAdmissionPolicy describes the definition of an admission validation policy that accepts or rejects an object without changing it."
title: "ValidatingAdmissionPolicy v1alpha1"
title: "ValidatingAdmissionPolicy v1beta1"
weight: 4
auto_generated: true
---
@ -21,9 +21,9 @@ guide. You can file document formatting bugs against the
[reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project.
-->
`apiVersion: admissionregistration.k8s.io/v1alpha1`
`apiVersion: admissionregistration.k8s.io/v1beta1`
`import "k8s.io/api/admissionregistration/v1alpha1"`
`import "k8s.io/api/admissionregistration/v1beta1"`
## ValidatingAdmissionPolicy {#ValidatingAdmissionPolicy}
@ -32,7 +32,7 @@ ValidatingAdmissionPolicy describes the definition of an admission validation po
<hr>
- **apiVersion**: admissionregistration.k8s.io/v1alpha1
- **apiVersion**: admissionregistration.k8s.io/v1beta1
- **kind**: ValidatingAdmissionPolicy
@ -106,7 +106,7 @@ ValidatingAdmissionPolicy describes the definition of an admission validation po
- If failurePolicy=Ignore, the policy is skipped
<a name="MatchCondition"></a>
**
*MatchCondition represents a condition which must be fulfilled for a request to be sent to a webhook.*
- **spec.matchConditions.expression** (string), required
@ -307,7 +307,9 @@ ValidatingAdmissionPolicy describes the definition of an admission validation po
Expression represents the expression which will be evaluated by CEL. ref: https://github.com/google/cel-spec CEL expressions have access to the contents of the API request/response, organized into CEL variables as well as some other useful variables:
- 'object' - The object from the incoming request. The value is null for DELETE requests. - 'oldObject' - The existing object. The value is null for CREATE requests. - 'request' - Attributes of the API request([ref](/pkg/apis/admission/types.go#AdmissionRequest)). - 'params' - Parameter resource referred to by the policy binding being evaluated. Only populated if the policy has a ParamKind. - 'authorizer' - A CEL Authorizer. May be used to perform authorization checks for the principal (user or service account) of the request.
- 'object' - The object from the incoming request. The value is null for DELETE requests. - 'oldObject' - The existing object. The value is null for CREATE requests. - 'request' - Attributes of the API request([ref](/pkg/apis/admission/types.go#AdmissionRequest)). - 'params' - Parameter resource referred to by the policy binding being evaluated. Only populated if the policy has a ParamKind. - 'namespaceObject' - The namespace object that the incoming object belongs to. The value is null for cluster-scoped resources. - 'variables' - Map of composited variables, from its name to its lazily evaluated value.
For example, a variable named 'foo' can be accessed as 'variables.foo'.
- 'authorizer' - A CEL Authorizer. May be used to perform authorization checks for the principal (user or service account) of the request.
See https://pkg.go.dev/k8s.io/apiserver/pkg/cel/library#Authz
- 'authorizer.requestResource' - A CEL ResourceCheck constructed from the 'authorizer' and configured with the
request resource.
@ -342,12 +344,33 @@ ValidatingAdmissionPolicy describes the definition of an admission validation po
Reason represents a machine-readable description of why this validation failed. If this is the first validation in the list to fail, this reason, as well as the corresponding HTTP response code, are used in the HTTP response to the client. The currently supported reasons are: "Unauthorized", "Forbidden", "Invalid", "RequestEntityTooLarge". If not set, StatusReasonInvalid is used in the response to the client.
- **spec.variables** ([]Variable)
*Patch strategy: merge on key `name`*
*Map: unique values on key name will be kept during a merge*
Variables contain definitions of variables that can be used in composition of other expressions. Each variable is defined as a named CEL expression. The variables defined here will be available under `variables` in other expressions of the policy except MatchConditions because MatchConditions are evaluated before the rest of the policy.
The expression of a variable can refer to other variables defined earlier in the list but not those after. Thus, Variables must be sorted by the order of first appearance and acyclic.
<a name="Variable"></a>
*Variable is the definition of a variable that is used for composition. A variable is defined as a named expression.*
- **spec.variables.expression** (string), required
Expression is the expression that will be evaluated as the value of the variable. The CEL expression has access to the same identifiers as the CEL expressions in Validation.
- **spec.variables.name** (string), required
Name is the name of the variable. The name must be a valid CEL identifier and unique among all variables. The variable can be accessed in other expressions through `variables` For example, if name is "foo", the variable will be available as `variables.foo`
- **status** (ValidatingAdmissionPolicyStatus)
The status of the ValidatingAdmissionPolicy, including warnings that are useful to determine if the policy behaves in the expected way. Populated by the system. Read-only.
<a name="ValidatingAdmissionPolicyStatus"></a>
*ValidatingAdmissionPolicyStatus represents the status of a ValidatingAdmissionPolicy.*
*ValidatingAdmissionPolicyStatus represents the status of an admission validation policy.*
- **status.conditions** ([]Condition)
@ -427,7 +450,7 @@ ValidatingAdmissionPolicyList is a list of ValidatingAdmissionPolicy.
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
- **items** ([]<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>)
- **items** ([]<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>)
List of ValidatingAdmissionPolicy.
@ -447,6 +470,10 @@ ValidatingAdmissionPolicyList is a list of ValidatingAdmissionPolicy.
ValidatingAdmissionPolicyBinding binds the ValidatingAdmissionPolicy with paramerized resources. ValidatingAdmissionPolicyBinding and parameter CRDs together define how cluster administrators configure policies for clusters.
For a given admission request, each binding will cause its policy to be evaluated N times, where N is 1 for policies/bindings that don't use params, otherwise N is the number of parameters selected by the binding.
The CEL expressions of a policy must have a computed CEL cost below the maximum CEL budget. Each evaluation of the policy is given an independent CEL cost budget. Adding/removing policies, bindings, or params can not affect whether a given (policy, binding, param) combination is within its own CEL budget.
<hr>
- **apiVersion** (string)
@ -623,18 +650,44 @@ ValidatingAdmissionPolicyBinding binds the ValidatingAdmissionPolicy with parame
- **spec.paramRef** (ParamRef)
ParamRef specifies the parameter resource used to configure the admission control policy. It should point to a resource of the type specified in ParamKind of the bound ValidatingAdmissionPolicy. If the policy specifies a ParamKind and the resource referred to by ParamRef does not exist, this binding is considered mis-configured and the FailurePolicy of the ValidatingAdmissionPolicy applied.
paramRef specifies the parameter resource used to configure the admission control policy. It should point to a resource of the type specified in ParamKind of the bound ValidatingAdmissionPolicy. If the policy specifies a ParamKind and the resource referred to by ParamRef does not exist, this binding is considered mis-configured and the FailurePolicy of the ValidatingAdmissionPolicy applied. If the policy does not specify a ParamKind then this field is ignored, and the rules are evaluated without a param.
<a name="ParamRef"></a>
*ParamRef references a parameter resource*
*ParamRef describes how to locate the params to be used as input to expressions of rules applied by a policy binding.*
- **spec.paramRef.name** (string)
Name of the resource being referenced.
name is the name of the resource being referenced.
One of `name` or `selector` must be set, but `name` and `selector` are mutually exclusive properties. If one is set, the other must be unset.
A single parameter used for all admission requests can be configured by setting the `name` field, leaving `selector` blank, and setting namespace if `paramKind` is namespace-scoped.
- **spec.paramRef.namespace** (string)
Namespace of the referenced resource. Should be empty for the cluster-scoped resources
namespace is the namespace of the referenced resource. Allows limiting the search for params to a specific namespace. Applies to both `name` and `selector` fields.
A per-namespace parameter may be used by specifying a namespace-scoped `paramKind` in the policy and leaving this field empty.
- If `paramKind` is cluster-scoped, this field MUST be unset. Setting this field results in a configuration error.
- If `paramKind` is namespace-scoped, the namespace of the object being evaluated for admission will be used when this field is left unset. Take care that if this is left empty the binding must not match any cluster-scoped resources, which will result in an error.
- **spec.paramRef.parameterNotFoundAction** (string)
`parameterNotFoundAction` controls the behavior of the binding when the resource exists, and name or selector is valid, but there are no parameters matched by the binding. If the value is set to `Allow`, then no matched parameters will be treated as successful validation by the binding. If set to `Deny`, then no matched parameters will be subject to the `failurePolicy` of the policy.
Allowed values are `Allow` or `Deny`
Required
- **spec.paramRef.selector** (<a href="{{< ref "../common-definitions/label-selector#LabelSelector" >}}">LabelSelector</a>)
selector can be used to match multiple param objects based on their labels. Supply selector: {} to match all resources of the ParamKind.
If multiple params are found, they are all evaluated with the policy expressions and the results are ANDed together.
One of `name` or `selector` must be set, but `name` and `selector` are mutually exclusive properties. If one is set, the other must be unset.
- **spec.policyName** (string)
@ -683,7 +736,7 @@ ValidatingAdmissionPolicyBinding binds the ValidatingAdmissionPolicy with parame
#### HTTP Request
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
GET /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}
#### Parameters
@ -702,7 +755,7 @@ GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
401: Unauthorized
@ -711,7 +764,7 @@ GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### HTTP Request
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}/status
GET /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}/status
#### Parameters
@ -730,7 +783,7 @@ GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
401: Unauthorized
@ -739,7 +792,7 @@ GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### HTTP Request
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
GET /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies
#### Parameters
@ -803,7 +856,7 @@ GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicyList" >}}">ValidatingAdmissionPolicyList</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicyList" >}}">ValidatingAdmissionPolicyList</a>): OK
401: Unauthorized
@ -812,12 +865,12 @@ GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
#### HTTP Request
POST /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
POST /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies
#### Parameters
- **body**: <a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>, required
- **body**: <a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>, required
@ -846,11 +899,11 @@ POST /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
202 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Accepted
202 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Accepted
401: Unauthorized
@ -859,7 +912,7 @@ POST /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
#### HTTP Request
PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
PUT /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}
#### Parameters
@ -869,7 +922,7 @@ PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
name of the ValidatingAdmissionPolicy
- **body**: <a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>, required
- **body**: <a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>, required
@ -898,9 +951,9 @@ PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
401: Unauthorized
@ -909,7 +962,7 @@ PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### HTTP Request
PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}/status
PUT /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}/status
#### Parameters
@ -919,7 +972,7 @@ PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
name of the ValidatingAdmissionPolicy
- **body**: <a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>, required
- **body**: <a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>, required
@ -948,9 +1001,9 @@ PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
401: Unauthorized
@ -959,7 +1012,7 @@ PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{nam
#### HTTP Request
PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
PATCH /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}
#### Parameters
@ -1003,9 +1056,9 @@ PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{n
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
401: Unauthorized
@ -1014,7 +1067,7 @@ PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{n
#### HTTP Request
PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}/status
PATCH /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}/status
#### Parameters
@ -1058,9 +1111,9 @@ PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{n
#### Response
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
200 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): OK
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1alpha1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
201 (<a href="{{< ref "../extend-resources/validating-admission-policy-v1beta1#ValidatingAdmissionPolicy" >}}">ValidatingAdmissionPolicy</a>): Created
401: Unauthorized
@ -1069,7 +1122,7 @@ PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{n
#### HTTP Request
DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
DELETE /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies/{name}
#### Parameters
@ -1119,7 +1172,7 @@ DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{
#### HTTP Request
DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
DELETE /apis/admissionregistration.k8s.io/v1beta1/validatingadmissionpolicies
#### Parameters

2
content/en/docs/reference/kubernetes-api/extend-resources/validating-webhook-configuration-v1.md
View File

@ -132,7 +132,7 @@ ValidatingWebhookConfiguration describes the configuration of and admission webh
- If failurePolicy=Fail, reject the request
- If failurePolicy=Ignore, the error is ignored and the webhook is skipped
This is an alpha feature and managed by the AdmissionWebhookMatchConditions feature gate.
This is a beta feature and managed by the AdmissionWebhookMatchConditions feature gate.
<a name="MatchCondition"></a>
*MatchCondition represents a condition which must by fulfilled for a request to be sent to a webhook.*

10
content/en/docs/reference/kubernetes-api/other-resources/validating-admission-policy-binding-list-v1alpha1.md → content/en/docs/reference/kubernetes-api/other-resources/validating-admission-policy-binding-list-v1beta1.md
View File

@ -1,11 +1,11 @@
---
api_metadata:
apiVersion: "admissionregistration.k8s.io/v1alpha1"
import: "k8s.io/api/admissionregistration/v1alpha1"
apiVersion: "admissionregistration.k8s.io/v1beta1"
import: "k8s.io/api/admissionregistration/v1beta1"
kind: "ValidatingAdmissionPolicyBindingList"
content_type: "api_reference"
description: ""
title: "ValidatingAdmissionPolicyBindingList v1alpha1"
title: "ValidatingAdmissionPolicyBindingList v1beta1"
weight: 1
auto_generated: true
---
@ -21,8 +21,8 @@ guide. You can file document formatting bugs against the
[reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project.
-->
`apiVersion: admissionregistration.k8s.io/v1alpha1`
`apiVersion: admissionregistration.k8s.io/v1beta1`
`import "k8s.io/api/admissionregistration/v1alpha1"`
`import "k8s.io/api/admissionregistration/v1beta1"`

200
content/en/docs/reference/kubernetes-api/policy-resources/network-policy-v1.md
View File

@ -46,10 +46,6 @@ NetworkPolicy describes what network traffic is allowed for a set of Pods
spec represents the specification of the desired behavior for this NetworkPolicy.
- **status** (<a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicyStatus" >}}">NetworkPolicyStatus</a>)
status represents the current state of the NetworkPolicy. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
@ -198,54 +194,6 @@ NetworkPolicySpec provides the specification of a NetworkPolicy
## NetworkPolicyStatus {#NetworkPolicyStatus}
NetworkPolicyStatus describes the current state of the NetworkPolicy.
<hr>
- **conditions** ([]Condition)
*Patch strategy: merge on key `type`*
*Map: unique values on key type will be kept during a merge*
conditions holds an array of metav1.Condition that describe the state of the NetworkPolicy. Current service state
<a name="Condition"></a>
*Condition contains details for one aspect of the current state of this API Resource.*
- **conditions.lastTransitionTime** (Time), required
lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
<a name="Time"></a>
*Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.*
- **conditions.message** (string), required
message is a human readable message indicating details about the transition. This may be an empty string.
- **conditions.reason** (string), required
reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
- **conditions.status** (string), required
status of the condition, one of True, False, Unknown.
- **conditions.type** (string), required
type of condition in CamelCase or in foo.example.com/CamelCase.
- **conditions.observedGeneration** (int64)
observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
## NetworkPolicyList {#NetworkPolicyList}
NetworkPolicyList is a list of NetworkPolicy objects.
@ -306,39 +254,6 @@ GET /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
#### Response
200 (<a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicy" >}}">NetworkPolicy</a>): OK
401: Unauthorized
### `get` read status of the specified NetworkPolicy
#### HTTP Request
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status
#### Parameters
- **name** (*in path*): string, required
name of the NetworkPolicy
- **namespace** (*in path*): string, required
<a href="{{< ref "../common-parameters/common-parameters#namespace" >}}">namespace</a>
- **pretty** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#pretty" >}}">pretty</a>
#### Response
@ -574,61 +489,6 @@ PUT /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
- **dryRun** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#dryRun" >}}">dryRun</a>
- **fieldManager** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#fieldManager" >}}">fieldManager</a>
- **fieldValidation** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#fieldValidation" >}}">fieldValidation</a>
- **pretty** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#pretty" >}}">pretty</a>
#### Response
200 (<a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicy" >}}">NetworkPolicy</a>): OK
201 (<a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicy" >}}">NetworkPolicy</a>): Created
401: Unauthorized
### `update` replace status of the specified NetworkPolicy
#### HTTP Request
PUT /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status
#### Parameters
- **name** (*in path*): string, required
name of the NetworkPolicy
- **namespace** (*in path*): string, required
<a href="{{< ref "../common-parameters/common-parameters#namespace" >}}">namespace</a>
- **body**: <a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicy" >}}">NetworkPolicy</a>, required
- **dryRun** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#dryRun" >}}">dryRun</a>
@ -684,66 +544,6 @@ PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
- **dryRun** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#dryRun" >}}">dryRun</a>
- **fieldManager** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#fieldManager" >}}">fieldManager</a>
- **fieldValidation** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#fieldValidation" >}}">fieldValidation</a>
- **force** (*in query*): boolean
<a href="{{< ref "../common-parameters/common-parameters#force" >}}">force</a>
- **pretty** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#pretty" >}}">pretty</a>
#### Response
200 (<a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicy" >}}">NetworkPolicy</a>): OK
201 (<a href="{{< ref "../policy-resources/network-policy-v1#NetworkPolicy" >}}">NetworkPolicy</a>): Created
401: Unauthorized
### `patch` partially update status of the specified NetworkPolicy
#### HTTP Request
PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status
#### Parameters
- **name** (*in path*): string, required
name of the NetworkPolicy
- **namespace** (*in path*): string, required
<a href="{{< ref "../common-parameters/common-parameters#namespace" >}}">namespace</a>
- **body**: <a href="{{< ref "../common-definitions/patch#Patch" >}}">Patch</a>, required
- **dryRun** (*in query*): string
<a href="{{< ref "../common-parameters/common-parameters#dryRun" >}}">dryRun</a>

2
content/en/docs/reference/kubernetes-api/service-resources/endpoint-slice-v1.md
View File

@ -149,6 +149,8 @@ EndpointSlice represents a subset of the endpoints that implement a service. For
* Kubernetes-defined prefixed names:
* 'kubernetes.io/h2c' - HTTP/2 over cleartext as described in https://www.rfc-editor.org/rfc/rfc7540
* 'kubernetes.io/ws' - WebSocket over cleartext as described in https://www.rfc-editor.org/rfc/rfc6455
* 'kubernetes.io/wss' - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455
* Other protocols should use implementation-defined prefixed names such as mycompany.com/my-custom-protocol.

2
content/en/docs/reference/kubernetes-api/service-resources/endpoints-v1.md
View File

@ -144,6 +144,8 @@ Endpoints is a collection of endpoints that implement the actual service. Exampl
* Kubernetes-defined prefixed names:
* 'kubernetes.io/h2c' - HTTP/2 over cleartext as described in https://www.rfc-editor.org/rfc/rfc7540
* 'kubernetes.io/ws' - WebSocket over cleartext as described in https://www.rfc-editor.org/rfc/rfc6455
* 'kubernetes.io/wss' - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455
* Other protocols should use implementation-defined prefixed names such as mycompany.com/my-custom-protocol.

17
content/en/docs/reference/kubernetes-api/service-resources/service-v1.md
View File

@ -100,7 +100,16 @@ ServiceSpec describes the attributes that a user creates on a service.
- **ports.appProtocol** (string)
The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.
The application protocol for this port. This is used as a hint for implementations to offer richer behavior for protocols that they understand. This field follows standard Kubernetes label syntax. Valid values are either:
* Un-prefixed protocol names - reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names).
* Kubernetes-defined prefixed names:
* 'kubernetes.io/h2c' - HTTP/2 over cleartext as described in https://www.rfc-editor.org/rfc/rfc7540
* 'kubernetes.io/ws' - WebSocket over cleartext as described in https://www.rfc-editor.org/rfc/rfc6455
* 'kubernetes.io/wss' - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455
* Other protocols should use implementation-defined prefixed names such as mycompany.com/my-custom-protocol.
- **type** (string)
@ -140,7 +149,7 @@ ServiceSpec describes the attributes that a user creates on a service.
- **loadBalancerIP** (string)
Only applies to Service Type: LoadBalancer. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature. Deprecated: This field was under-specified and its meaning varies across implementations, and it cannot support dual-stack. As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available. This field may be removed in a future API version.
Only applies to Service Type: LoadBalancer. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature. Deprecated: This field was under-specified and its meaning varies across implementations. Using it is non-portable and it may not support dual-stack. Users are encouraged to use implementation-specific annotations when available.
- **loadBalancerSourceRanges** ([]string)
@ -262,6 +271,10 @@ ServiceStatus represents the current status of a service.
IP is set for load-balancer ingress points that are IP based (typically GCE or OpenStack load-balancers)
- **loadBalancer.ingress.ipMode** (string)
IPMode specifies how the load-balancer IP behaves, and may only be specified when the ip field is specified. Setting this to "VIP" indicates that traffic is delivered to the node with the destination set to the load-balancer's IP and port. Setting this to "Proxy" indicates that traffic is delivered to the node or pod with the destination set to the node's IP and node port or the pod's IP and port. Service implementations may use this information to adjust traffic routing.
- **loadBalancer.ingress.ports** ([]PortStatus)
*Atomic: will be replaced during a merge*

37
content/en/docs/reference/kubernetes-api/workload-resources/job-v1.md
View File

@ -144,6 +144,10 @@ JobSpec describes how the job execution will look like.
- FailJob: indicates that the pod's job is marked as Failed and all
running pods are terminated.
- FailIndex: indicates that the pod's index is marked as Failed and will
not be restarted.
This value is alpha-level. It can be used when the
`JobBackoffLimitPerIndex` feature gate is enabled (disabled by default).
- Ignore: indicates that the counter towards the .backoffLimit is not
incremented and a replacement pod is created.
- Count: indicates that the pod is handled in the default way - the
@ -196,6 +200,26 @@ JobSpec describes how the job execution will look like.
Restricts the check for exit codes to the container with the specified name. When null, the rule applies to all containers. When specified, it should match one the container or initContainer names in the pod template.
### Alpha level
- **backoffLimitPerIndex** (int32)
Specifies the limit for the number of retries within an index before marking this index as failed. When enabled the number of failures per index is kept in the pod's batch.kubernetes.io/job-index-failure-count annotation. It can only be set when Job's completionMode=Indexed, and the Pod's restart policy is Never. The field is immutable. This field is alpha-level. It can be used when the `JobBackoffLimitPerIndex` feature gate is enabled (disabled by default).
- **maxFailedIndexes** (int32)
Specifies the maximal number of failed indexes before marking the Job as failed, when backoffLimitPerIndex is set. Once the number of failed indexes exceeds this number the entire Job is marked as Failed and its execution is terminated. When left as null the job continues execution of all of its indexes and is marked with the `Complete` Job condition. It can only be specified when backoffLimitPerIndex is set. It can be null or up to completions. It is required and must be less than or equal to 10^4 when is completions greater than 10^5. This field is alpha-level. It can be used when the `JobBackoffLimitPerIndex` feature gate is enabled (disabled by default).
- **podReplacementPolicy** (string)
podReplacementPolicy specifies when to create replacement Pods. Possible values are: - TerminatingOrFailed means that we recreate pods
when they are terminating (has a metadata.deletionTimestamp) or failed.
- Failed means to wait until a previously created Pod is fully terminated (has phase
Failed or Succeeded) before creating a replacement Pod.
When using podFailurePolicy, Failed is the the only allowed value. TerminatingOrFailed and Failed are allowed values when podFailurePolicy is not in use. This is an alpha field. Enable JobPodReplacementPolicy to be able to use this field.
## JobStatus {#JobStatus}
@ -312,6 +336,19 @@ JobStatus represents the current state of a Job.
This field is beta-level. The job controller populates the field when the feature gate JobReadyPods is enabled (enabled by default).
### Alpha level
- **failedIndexes** (string)
FailedIndexes holds the failed indexes when backoffLimitPerIndex=true. The indexes are represented in the text format analogous as for the `completedIndexes` field, ie. they are kept as decimal integers separated by commas. The numbers are listed in increasing order. Three or more consecutive numbers are compressed and represented by the first and last element of the series, separated by a hyphen. For example, if the failed indexes are 1, 3, 4, 5 and 7, they are represented as "1,3-5,7". This field is alpha-level. It can be used when the `JobBackoffLimitPerIndex` feature gate is enabled (disabled by default).
- **terminating** (int32)
The number of pods which are terminating (in phase Pending or Running and have a deletionTimestamp).
This field is alpha-level. The job controller populates the field when the feature gate JobPodReplacementPolicy is enabled (disabled by default).
## JobList {#JobList}

72
content/en/docs/reference/kubernetes-api/workload-resources/pod-v1.md
View File

@ -430,7 +430,7 @@ PodSpec is a description of a pod.
- **securityContext.seccompProfile.localhostProfile** (string)
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type is "Localhost".
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type.
- **securityContext.seLinuxOptions** (SELinuxOptions)
@ -487,7 +487,7 @@ PodSpec is a description of a pod.
- **securityContext.windowsOptions.hostProcess** (boolean)
HostProcess determines if a container should be run as a 'Host Process' container. This field is alpha-level and will only be honored by components that enable the WindowsHostProcessContainers feature flag. Setting this field without the feature flag will result in errors when validating the Pod. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
- **securityContext.windowsOptions.runAsUserName** (string)
@ -536,9 +536,7 @@ PodSpec is a description of a pod.
ResourceClaimTemplateName is the name of a ResourceClaimTemplate object in the same namespace as this pod.
The template will be used to create a new ResourceClaim, which will be bound to this pod. When this pod is deleted, the ResourceClaim will also be deleted. The name of the ResourceClaim will be \<pod name>-\<resource name>, where \<resource name> is the PodResourceClaim.Name. Pod validation will reject the pod if the concatenated name is not valid for a ResourceClaim (e.g. too long).
An existing ResourceClaim with that name that is not owned by the pod will not be used for the pod to avoid using an unrelated resource by mistake. Scheduling and pod startup are then blocked until the unrelated ResourceClaim is removed.
The template will be used to create a new ResourceClaim, which will be bound to this pod. When this pod is deleted, the ResourceClaim will also be deleted. The pod name and resource name, along with a generated component, will be used to form a unique name for the ResourceClaim, which will be recorded in pod.status.resourceClaimStatuses.
This field is immutable and no changes will be made to the corresponding ResourceClaim by the control plane after creating the ResourceClaim.
@ -903,6 +901,10 @@ A single application container that you want to run within a pod.
StartupProbe indicates that the Pod has successfully initialized. If specified, no other probes are executed until this completes successfully. If this probe fails, the Pod will be restarted, just as if the livenessProbe failed. This can be used to provide different probe parameters at the beginning of a Pod's lifecycle, when it might take a long time to load data or warm a cache, than during steady-state operation. This cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
- **restartPolicy** (string)
RestartPolicy defines the restart behavior of individual containers in a pod. This field may only be set for init containers, and the only allowed value is "Always". For non-init containers or when this field is not specified, the restart behavior is defined by the Pod's restart policy and the container type. Setting the RestartPolicy as "Always" for the init container will have the following effect: this init container will be continually restarted on exit until all regular containers have terminated. Once all regular containers have completed, all init containers with restartPolicy "Always" will be shut down. This lifecycle differs from normal init containers and is often referred to as a "sidecar" container. Although this init container still starts in the init container sequence, it does not wait for the container to complete before proceeding to the next init container. Instead, the next init container starts immediately after this init container is started, or after any startupProbe has successfully completed.
### Security Context
@ -971,7 +973,7 @@ A single application container that you want to run within a pod.
- **securityContext.seccompProfile.localhostProfile** (string)
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type is "Localhost".
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type.
- **securityContext.seLinuxOptions** (SELinuxOptions)
@ -1013,7 +1015,7 @@ A single application container that you want to run within a pod.
- **securityContext.windowsOptions.hostProcess** (boolean)
HostProcess determines if a container should be run as a 'Host Process' container. This field is alpha-level and will only be honored by components that enable the WindowsHostProcessContainers feature flag. Setting this field without the feature flag will result in errors when validating the Pod. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
- **securityContext.windowsOptions.runAsUserName** (string)
@ -1284,6 +1286,10 @@ To add an ephemeral container, use the ephemeralcontainers subresource of an exi
Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.
- **restartPolicy** (string)
Restart policy for the container to manage the restart behavior of each container within a pod. This may only be set for init containers. You cannot set this field on ephemeral containers.
### Debugging
@ -1367,7 +1373,7 @@ To add an ephemeral container, use the ephemeralcontainers subresource of an exi
- **securityContext.seccompProfile.localhostProfile** (string)
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type is "Localhost".
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type.
- **securityContext.seLinuxOptions** (SELinuxOptions)
@ -1409,7 +1415,7 @@ To add an ephemeral container, use the ephemeralcontainers subresource of an exi
- **securityContext.windowsOptions.hostProcess** (boolean)
HostProcess determines if a container should be run as a 'Host Process' container. This field is alpha-level and will only be honored by components that enable the WindowsHostProcessContainers feature flag. Setting this field without the feature flag will result in errors when validating the Pod. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
- **securityContext.windowsOptions.runAsUserName** (string)
@ -1554,7 +1560,7 @@ LifecycleHandler defines a specific action that should be taken in a lifecycle h
- **httpGet.httpHeaders.name** (string), required
The header field name
The header field name. This will be canonicalized upon output, so case-variant names will be understood as the same header.
- **httpGet.httpHeaders.value** (string), required
@ -1826,7 +1832,7 @@ Probe describes a health check to be performed against a container to determine
- **httpGet.httpHeaders.name** (string), required
The header field name
The header field name. This will be canonicalized upon output, so case-variant names will be understood as the same header.
- **httpGet.httpHeaders.value** (string), required
@ -1915,7 +1921,22 @@ PodStatus represents information about the status of a pod. Status may trail the
- **hostIP** (string)
IP address of the host to which the pod is assigned. Empty if not yet scheduled.
hostIP holds the IP address of the host to which the pod is assigned. Empty if the pod has not started yet. A pod can be assigned to a node that has a problem in kubelet which in turns mean that HostIP will not be updated even if there is a node is assigned to pod
- **hostIPs** ([]HostIP)
*Patch strategy: merge on key `ip`*
*Atomic: will be replaced during a merge*
hostIPs holds the IP addresses allocated to the host. If this field is specified, the first entry must match the hostIP field. This list is empty if the pod has not started yet. A pod can be assigned to a node that has a problem in kubelet which in turns means that HostIPs will not be updated even if there is a node is assigned to this pod.
<a name="HostIP"></a>
*HostIP represents a single IP address allocated to the host.*
- **hostIPs.ip** (string)
IP is the IP address assigned to the host
- **startTime** (Time)
@ -1942,7 +1963,7 @@ PodStatus represents information about the status of a pod. Status may trail the
- **podIP** (string)
IP address allocated to the pod. Routable at least within the cluster. Empty if not yet allocated.
podIP address allocated to the pod. Routable at least within the cluster. Empty if not yet allocated.
- **podIPs** ([]PodIP)
@ -1951,13 +1972,11 @@ PodStatus represents information about the status of a pod. Status may trail the
podIPs holds the IP addresses allocated to the pod. If this field is specified, the 0th entry must match the podIP field. Pods may be allocated at most 1 value for each of IPv4 and IPv6. This list is empty if no IPs have been allocated yet.
<a name="PodIP"></a>
*IP address information for entries in the (plural) PodIPs field. Each entry includes:
IP: An IP address allocated to the pod. Routable at least within the cluster.*
*PodIP represents a single IP address allocated to the pod.*
- **podIPs.ip** (string)
ip is an IP address (IPv4 or IPv6) assigned to the pod
IP is the IP address assigned to the pod
- **conditions** ([]PodCondition)
@ -2023,6 +2042,25 @@ PodStatus represents information about the status of a pod. Status may trail the
<a name="ContainerStatus"></a>
*ContainerStatus contains details for the current status of this container.*
- **resourceClaimStatuses** ([]PodResourceClaimStatus)
*Patch strategies: retainKeys, merge on key `name`*
*Map: unique values on key name will be kept during a merge*
Status of resource claims.
<a name="PodResourceClaimStatus"></a>
*PodResourceClaimStatus is stored in the PodStatus for each PodResourceClaim which references a ResourceClaimTemplate. It stores the generated name for the corresponding ResourceClaim.*
- **resourceClaimStatuses.name** (string), required
Name uniquely identifies this resource claim inside the pod. This must match the name of an entry in pod.spec.resourceClaims, which implies that the string must be a DNS_LABEL.
- **resourceClaimStatuses.resourceClaimName** (string)
ResourceClaimName is the name of the ResourceClaim that was generated for the Pod in the namespace of the Pod. It this is unset, then generating a ResourceClaim was not necessary. The pod.spec.resourceClaims entry can be ignored in this case.
- **resize** (string)
Status of resources resize desired for pod's containers. It is empty if no resources resize is pending. Any changes to container resources will automatically set this to "Proposed"

35
content/en/docs/reference/labels-annotations-taints/_index.md
View File

@ -227,6 +227,21 @@ This annotation is applied to the parent object used to track an ApplySet to ind
tooling manages that ApplySet. Tooling should refuse to mutate ApplySets belonging to other tools.
The value must be in the format `<toolname>/<semver>`.
### apps.kubernetes.io/pod-index (beta) {#apps-kubernetes.io-pod-index}
Type: Label
Example: `apps.kubernetes.io/pod-index: "0"`
Used on: Pod
When a StatefulSet controller creates a Pod for the StatefulSet, it sets this label on that Pod.
The value of the label is the ordinal index of the pod being created.
See [Pod Index Label](/docs/concepts/workloads/controllers/statefulset/#pod-index-label)
in the StatefulSet topic for more details. Note the [PodIndexLabel](content/en/docs/reference/command-line-tools-reference/feature-gates.md) feature gate must be enabled
for this label to be added to pods.
### cluster-autoscaler.kubernetes.io/safe-to-evict
Type: Annotation
@ -1055,15 +1070,31 @@ by the cloud-controller-manager.
### batch.kubernetes.io/job-completion-index
Type: Annotation
Type: Annotation, Label
Example: `batch.kubernetes.io/job-completion-index: "3"`
Used on: Pod
The Job controller in the kube-controller-manager sets this annotation for Pods
The Job controller in the kube-controller-manager sets this as a label and annotation for Pods
created with Indexed [completion mode](/docs/concepts/workloads/controllers/job/#completion-mode).
Note the [PodIndexLabel](content/en/docs/reference/command-line-tools-reference/feature-gates.md) feature gate must be enabled
for this to be added as a pod **label**, otherwise it will just be an annotation.
### batch.kubernetes.io/cronjob-scheduled-timestamp
Type: Annotation
Example: `batch.kubernetes.io/cronjob-scheduled-timestamp: "2016-05-19T03:00:00-07:00"`
Used on: Jobs and Pods controlled by CronJobs
This annotation is used to record the original (expected) creation timestamp for a Job,
when that Job is part of a CronJob.
The control plane sets the value to that timestamp in RFC3339 format. If the Job belongs to a CronJob
with a timezone specified, then the timestamp is in that timezone. Otherwise, the timestamp is in controller-manager's local time.
### kubectl.kubernetes.io/default-container
Type: Annotation

38
content/en/docs/reference/networking/virtual-ips.md
View File

@ -138,26 +138,6 @@ iptables:
...
```
##### Performance optimization for `iptables` mode {#minimize-iptables-restore}
{{< feature-state for_k8s_version="v1.27" state="beta" >}}
In Kubernetes {{< skew currentVersion >}} the kube-proxy defaults to a minimal approach
to `iptables-restore` operations, only making updates where Services or EndpointSlices have
actually changed. This is a performance optimization.
The original implementation updated all the rules for all Services on every sync; this
sometimes led to performance issues (update lag) in large clusters.
If you are not running kube-proxy from Kubernetes {{< skew currentVersion >}}, check
the behavior and associated advice for the version that you are actually running.
If you were previously overriding `minSyncPeriod`, you should try
removing that override and letting kube-proxy use the default value
(`1s`) or at least a smaller value than you were using before upgrading.
You can select the legacy behavior by disabling the `MinimizeIPTablesRestore`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
(you should not need to).
##### `minSyncPeriod`
The `minSyncPeriod` parameter sets the minimum duration between
@ -189,6 +169,22 @@ Especially, if kube-proxy's `sync_proxy_rules_duration_seconds` metric
indicates an average time much larger than 1 second, then bumping up
`minSyncPeriod` may make updates more efficient.
##### Updating legacy `minSyncPeriod` configuration {#minimize-iptables-restore}
Older versions of kube-proxy updated all the rules for all Services on
every sync; this led to performance issues (update lag) in large
clusters, and the recommended solution was to set a larger
`minSyncPeriod`. Since Kubernetes v1.28, the iptables mode of
kube-proxy uses a more minimal approach, only making updates where
Services or EndpointSlices have actually changed.
If you were previously overriding `minSyncPeriod`, you should try
removing that override and letting kube-proxy use the default value
(`1s`) or at least a smaller value than you were using before upgrading.
If you are not running kube-proxy from Kubernetes {{< skew currentVersion >}}, check
the behavior and associated advice for the version that you are actually running.
##### `syncPeriod`
The `syncPeriod` parameter controls a handful of synchronization
@ -415,7 +411,7 @@ relevant Service.
### Traffic to terminating endpoints
{{< feature-state for_k8s_version="v1.26" state="beta" >}}
{{< feature-state for_k8s_version="v1.28" state="stable" >}}
If the `ProxyTerminatingEndpoints`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)

77
content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_validate.md Normal file
View File

@ -0,0 +1,77 @@
<!--
The file is auto-generated from the Go source code of the component using a generic
[generator](https://github.com/kubernetes-sigs/reference-docs/). To learn how
to generate the reference documentation, please read
[Contributing to the reference documentation](/docs/contribute/generate-ref-docs/).
To update the reference content, please follow the
[Contributing upstream](/docs/contribute/generate-ref-docs/contribute-upstream/)
guide. You can file document formatting bugs against the
[reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project.
-->
PLACEHOLDER
### Synopsis
PLACEHOLDER
```
kubeadm config validate [flags]
```
### Options
<table style="width: 100%; table-layout: fixed;">
<colgroup>
<col span="1" style="width: 10px;" />
<col span="1" />
</colgroup>
<tbody>
<tr>
<td colspan="2">-h, --help</td>
</tr>
<tr>
<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>help for validate</p></td>
</tr>
<tr>
<td colspan="2">--config string</td>
</tr>
<tr>
<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>PLACEHOLDER</p></td>
</tr>
</tbody>
</table>
### Options inherited from parent commands
<table style="width: 100%; table-layout: fixed;">
<colgroup>
<col span="1" style="width: 10px;" />
<col span="1" />
</colgroup>
<tbody>
<tr>
<td colspan="2">--kubeconfig string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "/etc/kubernetes/admin.conf"</td>
</tr>
<tr>
<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.</p></td>
</tr>
<tr>
<td colspan="2">--rootfs string</td>
</tr>
<tr>
<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>[EXPERIMENTAL] The path to the 'real' host root filesystem.</p></td>
</tr>
</tbody>
</table>

6
content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md
View File

@ -31,6 +31,8 @@ For more information on using the kubeadm configuration API navigate to
You can use `kubeadm config migrate` to convert your old configuration files that contain a deprecated
API version to a newer, supported API version.
`kubeadm config validate` can be used for validating a configuration file.
`kubeadm config images list` and `kubeadm config images pull` can be used to list and pull the images
that kubeadm requires.
@ -51,6 +53,10 @@ that kubeadm requires.
{{< include "generated/kubeadm_config_migrate.md" >}}
## kubeadm config validate {#cmd-config-validate}
{{< include "generated/kubeadm_config_validate.md" >}}
## kubeadm config images list {#cmd-config-images-list}
{{< include "generated/kubeadm_config_images_list.md" >}}

24
content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md
View File

@ -189,6 +189,30 @@ or `kubeadm upgrade apply`), kubeadm respects the value of `UnversionedKubeletCo
(during `kubeadm join`, `kubeadm reset`, `kubeadm upgrade ...`), kubeadm attempts to use unversioned ConfigMap name first;
if that does not succeed, kubeadm falls back to using the legacy (versioned) name for that ConfigMap.
List of deprecated feature gates:
{{< table caption="kubeadm deprecated feature gates" >}}
Feature | Default
:-------|:--------
`UpgradeAddonsBeforeControlPlane` | `false`
{{< /table >}}
Feature gate descriptions:
`UpgradeAddonsBeforeControlPlane`
: This is as a **disabled** feature gate that was introduced for Kubernetes v1.28, in order to allow reactivating a legacy
and deprecated behavior during cluster upgrade. For kubeadm versions prior to v1.28, kubeadm upgrades cluster addons (including
CoreDNS and kube-proxy) immediately during `kubeadm upgrade apply`, regardless of whether there are other control plane
instances that have not been upgraded. This may cause compatibility problems. Since v1.28, kubeadm defaults to a mode that
always checks whether all the control plane instances have been upgraded before starting to upgrade the addons. This behavior
is applied to both `kubeadm upgrade apply` and `kubeadm upgrade node`. kubeadm determines whether a control plane instance
has been upgraded by checking whether the image of the kube-apiserver Pod has been upgraded. You must perform control plane
instances upgrade sequentially or at least ensure that the last control plane instance upgrade is not started until all the
other control plane instances have been upgraded completely, and the addons upgrade will be performed after the last control plane
instance is upgraded. The deprecated `UpgradeAddonsBeforeControlPlane` feature gate gives you a chance to keep the old upgrade
behavior. You should not need this old behavior; if you do, you should consider changing your cluster or upgrade processes, as this
feature gate will be removed in a future release.
### Adding kube-proxy parameters {#kube-proxy}
For information about kube-proxy parameters in the kubeadm configuration see:

14
content/en/docs/setup/production-environment/container-runtimes.md
View File

@ -157,6 +157,12 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user
the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`.
{{< /note >}}
In Kubernetes v1.28, with the `KubeletCgroupDriverFromCRI`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
enabled and a container runtime that supports the `RuntimeConfig` CRI RPC,
the kubelet automatically detects the appropriate cgroup driver from the runtime,
and ignores the `cgroupDriver` setting within the kubelet configuration.
If you configure `systemd` as the cgroup driver for the kubelet, you must also
configure `systemd` as the cgroup driver for the container runtime. Refer to
the documentation for your container runtime for instructions. For example:
@ -251,6 +257,10 @@ sudo systemctl restart containerd
When using kubeadm, manually configure the
[cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver).
In Kubernetes v1.28, you can enable automatic detection of the
cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.
#### Overriding the sandbox (pause) image {#override-pause-image-containerd}
In your [containerd config](https://github.com/containerd/containerd/blob/main/docs/cri/config.md) you can overwrite the
@ -292,6 +302,10 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val
cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O
in sync.
In Kubernetes v1.28, you can enable automatic detection of the
cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.
For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default.
#### Overriding the sandbox (pause) image {#override-pause-image-cri-o}

5
content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md
View File

@ -76,9 +76,8 @@ Install a {{< glossary_tooltip term_id="container-runtime" text="container runti
For detailed instructions and other prerequisites, see [Installing kubeadm](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/).
{{< note >}}
If you have already installed kubeadm, run
`apt-get update && apt-get upgrade` or
`yum update` to get the latest version of kubeadm.
If you have already installed kubeadm, see the first two steps of the
[Upgrading Linux nodes](/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes) document for instructions on how to upgrade kubeadm.
When you upgrade, the kubelet restarts every few seconds as it waits in a crashloop for
kubeadm to tell it what to do. This crashloop is expected and normal.

154
content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md
View File

@ -144,13 +144,75 @@ For more information on version skews, see:
* Kubernetes [version and version-skew policy](/docs/setup/release/version-skew-policy/)
* Kubeadm-specific [version skew policy](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#version-skew-policy)
{{< note >}}
Kubernetes has two different package repositories starting from August 2023.
The Google-hosted repository is deprecated and it's being replaced with the
Kubernetes (community-owned) package repositories. The Kubernetes project strongly
recommends using the Kubernetes community-owned package repositories, because the
project plans to stop publishing packages to the Google-hosted repository in the future.
There are some important considerations for the Kubernetes package repositories:
- The Kubernetes package repositories contain packages beginning with those
Kubernetes versions that were still under support when the community took
over the package builds. This means that anything before v1.24.0 will only be
available in the Google-hosted repository.
- There's a dedicated package repository for each Kubernetes minor version.
When upgrading to to a different minor release, you must bear in mind that
the package repository details also change.
{{< /note >}}
{{< tabs name="k8s_install" >}}
{{% tab name="Debian-based distributions" %}}
### Kubernetes package repositories {#dpkg-k8s-package-repo}
These instructions are for Kubernetes {{< skew currentVersion >}}.
1. Update the `apt` package index and install packages needed to use the Kubernetes `apt` repository:
```shell
sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl
```
2. Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories so you can disregard the version in the URL:
```shell
curl -fsSL https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
```
3. Add the appropriate Kubernetes `apt` repository:
```shell
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
4. Update the `apt` package index, install kubelet, kubeadm and kubectl, and pin their version:
```shell
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```
{{< note >}}
In releases older than Debian 12 and Ubuntu 22.04, `/etc/apt/keyrings` does not exist by default;
you can create it by running `sudo mkdir -m 755 /etc/apt/keyrings`
{{< /note >}}
### Google-hosted package repository (deprecated) {#dpkg-google-package-repo}
These instructions are for Kubernetes {{< skew currentVersion >}}.
1. Update the `apt` package index and install packages needed to use the Kubernetes `apt` repository:
```shell
sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl
```
@ -160,27 +222,86 @@ For more information on version skews, see:
curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
```
3. Add the Kubernetes `apt` repository:
3. Add the Google-hosted `apt` repository:
```shell
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
4. Update `apt` package index, install kubelet, kubeadm and kubectl, and pin their version:
4. Update the `apt` package index, install kubelet, kubeadm and kubectl, and pin their version:
```shell
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```
{{< note >}}
In releases older than Debian 12 and Ubuntu 22.04, `/etc/apt/keyrings` does not exist by default.
You can create this directory if you need to, making it world-readable but writeable only by admins.
In releases older than Debian 12 and Ubuntu 22.04, `/etc/apt/keyrings` does not exist by default;
you can create it by running `sudo mkdir -m 755 /etc/apt/keyrings`
{{< /note >}}
{{% /tab %}}
{{% tab name="Red Hat-based distributions" %}}
```bash
1. Set SELinux to `permissive` mode:
```shell
# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
```
{{< caution >}}
- Setting SELinux in permissive mode by running `setenforce 0` and `sed ...`
effectively disables it. This is required to allow containers to access the host
filesystem; for example, some cluster network plugins require that. You have to
do this until SELinux support is improved in the kubelet.
- You can leave SELinux enabled if you know how to configure it but it may require
settings that are not supported by kubeadm.
{{< /caution >}}
### Kubernetes package repositories {#rpm-k8s-package-repo}
These instructions are for Kubernetes {{< skew currentVersion >}}.
2. Add the Kubernetes `yum` repository. The `exclude` parameter in the
repository definition ensures that the packages related to Kubernetes are
not upgraded upon running `yum update` as there's a special procedure that
must be followed for upgrading Kubernetes.
```shell
# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
```
3. Install kubelet, kubeadm and kubectl, and enable kubelet to ensure it's automatically started on startup:
```shell
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
```
### Google-hosted package repository (deprecated) {#rpm-google-package-repo}
These instructions are for Kubernetes {{< skew currentVersion >}}.
2. Add the Google-hosted `yum` repository. The `exclude` parameter in the
repository definition ensures that the packages related to Kubernetes are
not upgraded upon running `yum update` as there's a special procedure that
must be followed for upgrading Kubernetes.
```shell
# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
@ -190,27 +311,20 @@ gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
```
# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
3. Install kubelet, kubeadm and kubectl, and enable kubelet to ensure it's automatically started on startup:
```shell
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
```
**Notes:**
- Setting SELinux in permissive mode by running `setenforce 0` and `sed ...` effectively disables it.
This is required to allow containers to access the host filesystem, which is needed by pod networks for example.
You have to do this until SELinux support is improved in the kubelet.
- You can leave SELinux enabled if you know how to configure it but it may require settings that are not supported by kubeadm.
- If the `baseurl` fails because your Red Hat-based distribution cannot interpret `basearch`, replace `\$basearch` with your computer's architecture.
Type `uname -m` to see that value.
For example, the `baseurl` URL for `x86_64` could be: `https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64`.
{{< note >}}
If the `baseurl` fails because your RPM-based distribution cannot interpret `$basearch`, replace `\$basearch` with your computer's architecture.
Type `uname -m` to see that value.
For example, the `baseurl` URL for `x86_64` could be: `https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64`.
{{< /note >}}
{{% /tab %}}
{{% tab name="Without a package manager" %}}

10
content/en/docs/tasks/administer-cluster/encrypt-data.md
View File

@ -225,7 +225,7 @@ The following table describes each available provider:
<td colspan="4">Not recommended for use except when an automated key rotation scheme is implemented. Key material accessible from control plane host.</td>
</tr>
<tr>
<th rowspan="2" scope="row"><tt>kms</tt> v1</th>
<th rowspan="2" scope="row"><tt>kms</tt> v1 <em>(deprecated since Kubernetes v1.28)</em></th>
<td>Uses envelope encryption scheme with DEK per resource.</td>
<td>Strongest</td>
<td>Slow (<em>compared to <tt>kms</tt> version 2</em>)</td>
@ -254,8 +254,12 @@ The following table describes each available provider:
Data is encrypted by data encryption keys (DEKs) using AES-GCM; DEKs
are encrypted by key encryption keys (KEKs) according to configuration
in Key Management Service (KMS).
A new DEK is generated at API server startup, and is then reused for
encryption. The DEK is rotated whenever the KEK is rotated.
Kubernetes defaults to generating a new DEK at API server startup, which is then
reused for object encryption.
If you enable the <tt>KMSv2KDF</tt>
<a href="/docs/reference/command-line-tools-reference/feature-gates/">feature gate</a>,
Kubernetes instead generates a new DEK per encryption from a secret seed.
Whichever approach you configure, the DEK or seed is also rotated whenever the KEK is rotated.<br/>
A good choice if using a third party tool for key management.
Available in beta from Kubernetes v1.27.
<br />

64
content/en/docs/tasks/administer-cluster/kms-provider.md
View File

@ -8,26 +8,30 @@ weight: 370
---
<!-- overview -->
This page shows how to configure a Key Management Service (KMS) provider and plugin to enable secret data encryption.
Currently there are two KMS API versions. New integrations that only need to support Kubernetes v1.27+
should use KMS v2 as it offers significantly better performance characteristics than v1
(note the `Caution` sections below for specific cases when KMS v2 must not be used.)
In Kubernetes {{< skew currentVersion >}} there are two versions of KMS at-rest encryption.
You should use KMS v2 if feasible because KMS v1 is deprecated (since Kubernetes v1.28).
However, you should also read and observe the **Caution** notices in this page that highlight specific
cases when you must not use KMS v2. KMS v2 offers significantly better performance characteristics than KMS v1.
## {{% heading "prerequisites" %}}
{{< include "task-tutorial-prereqs.md" >}}
The version of Kubernetes that you need depends on which KMS API version
you have selected.
you have selected. Kubernetes recommends using KMS v2.
- If you selected KMS API v1, any supported Kubernetes version will work fine.
- If you selected KMS API v2, you should use Kubernetes v{{< skew currentVersion >}}
(if you are running a different version of Kubernetes that also supports the v2 KMS
API, switch to the documentation for that version of Kubernetes).
- If you selected KMS API v1 to support clusters prior to version v1.27
or if you have a legacy KMS plugin that only supports KMS v1,
any supported Kubernetes version will work. This API is deprecated as of Kubernetes v1.28.
Kubernetes does not recommend the use of this API.
{{< version-check >}}
### KMS v1
{{< feature-state for_k8s_version="v1.12" state="beta" >}}
{{< feature-state for_k8s_version="v1.28" state="deprecated" >}}
* Kubernetes version 1.10.0 or later is required
@ -38,13 +42,22 @@ you have selected.
* For version 1.25 and 1.26, enabling the feature via kube-apiserver feature gate is required.
Set `--feature-gates=KMSv2=true` to configure a KMS v2 provider.
For environments where all API servers are running version 1.28 or later, and you do not require the ability
to downgrade to Kubernetes v1.27, you can enable the `KMSv2KDF` feature gate (a beta feature) for more
robust data encryption key generation. The Kubernetes project recommends enabling KMS v2 KDF if those
preconditions are met.
* Your cluster must use etcd v3 or later
{{< caution >}}
The KMS v2 API and implementation changed in incompatible ways in-between the alpha release in v1.25
and the beta release in v1.27. Attempting to upgrade from old versions with the alpha feature
enabled will result in data loss.
---
Running mixed API server versions with some servers at v1.27, and others at v1.28 _with the
`KMSv2KDF` feature gate enabled_ is **not supported** - and is likely to result in data loss.
{{< /caution >}}
<!-- steps -->
@ -52,18 +65,33 @@ enabled will result in data loss.
The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd.
The data is encrypted using a data encryption key (DEK).
The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS.
With KMS v1, a new DEK is generated for each encryption.
With KMS v2, a new DEK is generated on server startup and when the KMS plugin informs the API server
that a KEK rotation has occurred (see `Understanding key_id and Key Rotation` section below).
With KMS v2, there are two ways for the API server to generate a DEK.
Kubernetes defaults to generating a new DEK at API server startup, which is then reused
for resource encryption. However, if you use KMS v2 _and_ enable the `KMSv2KDF`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/), then
Kubernetes instead generates a new DEK **per encryption**: the API server uses a
_key derivation function_ to generate single use data encryption keys from a secret seed
combined with some random data.
Whichever approach you configure, the DEK or seed is also rotated whenever the KEK is rotated
(see `Understanding key_id and Key Rotation` section below for more details).
The KMS provider uses gRPC to communicate with a specific KMS plugin over a UNIX domain socket.
The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s)
as the Kubernetes control plane, is responsible for all communication with the remote KMS.
{{< caution >}}
If you are running virtual machine (VM) based nodes that leverage VM state store with this feature, you must not use KMS v2.
If you are running virtual machine (VM) based nodes that leverage VM state store with this feature,
using KMS v2 is **insecure** and an information security risk unless you also explicitly enable
the `KMSv2KDF`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
With KMS v2, the API server uses AES-GCM with a 12 byte nonce (8 byte atomic counter and 4 bytes random data) for encryption.
The following issues could occur if the VM is saved and restored:
1. The counter value may be lost or corrupted if the VM is saved in an inconsistent state or restored improperly.
This can lead to a situation where the same counter value is used twice, resulting in the same nonce being used
for two different messages.
@ -72,6 +100,13 @@ resulting in the same nonce being used again.
Although both of these cases are partially mitigated by the 4 byte random nonce, this can compromise
the security of the encryption.
If you have enabled the `KMSv2KDF`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) _and_ are using KMS v2
(not KMS v1), the API server generates single use data encryption keys from a secret seed.
This eliminates the need for a counter based nonce while avoiding nonce collision concerns.
It also removes any specific concerns with using KMS v2 and VM state store.
{{< /caution >}}
## Configuring the KMS provider
@ -108,7 +143,7 @@ See [Understanding the encryption at rest configuration](/docs/tasks/administer-
To implement a KMS plugin, you can develop a new plugin gRPC server or enable a KMS plugin
already provided by your cloud provider.
You then integrate the plugin with the remote KMS and deploy it on the Kubernetes master.
You then integrate the plugin with the remote KMS and deploy it on the Kubernetes control plane.
### Enabling the KMS supported by your cloud provider
@ -222,8 +257,9 @@ Then use the functions and data structures in the stub file to develop the serve
desired buffer to allow config changes to be processed - a minimum `M` of five minutes is recommend). Note that no
API server restart is required to perform KEK rotation.
{{< caution >}}
Because you don't control the number of writes performed with the DEK, we recommend rotating the KEK at least every 90 days.
{{< caution >}}
Because you don't control the number of writes performed with the DEK,
the Kubernetes project recommends rotating the KEK at least every 90 days.
{{< /caution >}}
* protocol: UNIX domain socket (`unix`)
@ -248,7 +284,7 @@ for decryption (KMS v2 makes this process easier by providing a dedicated `annot
### Deploying the KMS plugin
Ensure that the KMS plugin runs on the same host(s) as the Kubernetes master(s).
Ensure that the KMS plugin runs on the same host(s) as the Kubernetes API server(s).
## Encrypting your data with the KMS provider

158
content/en/docs/tasks/administer-cluster/kubeadm/change-package-repository.md Normal file
View File

@ -0,0 +1,158 @@
---
title: Changing The Kubernetes Package Repository
content_type: task
weight: 120
---
<!-- overview -->
This page explains how to switch from one Kubernetes package repository to another
when upgrading Kubernetes minor releases. Unlike deprecated Google-hosted
repositories, the Kubernetes package repositories are structured in a way that
there's a dedicated package repository for each Kubernetes minor version.
## {{% heading "prerequisites" %}}
This document assumes that you're already using the Kubernetes community-owned
package repositories. If that's not the case, it's strongly recommend to migrate
to the Kubernetes package repositories.
### Verifying if the Kubernetes package repositories are used
If you're unsure if you're using the Kubernetes package repositories or the
Google-hosted repository, take the following steps to verify:
{{< tabs name="k8s_install_versions" >}}
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
Print the contents of the file that defines the Kubernetes `apt` repository:
```shell
# On your system, this configuration file could have a different name
pager /etc/apt/sources.list.d/kubernetes.list
```
If you see a line similar to:
```
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v{{< skew currentVersionAddMinor -1 "." >}}/deb/ /
```
**You're using the Kubernetes package repositories and this guide applies to you.**
Otherwise, it's strongly recommend to migrate to the Kubernetes package repositories.
{{% /tab %}}
{{% tab name="CentOS, RHEL or Fedora" %}}
Print the contents of the file that defines the Kubernetes `yum` repository:
```shell
# On your system, this configuration file could have a different name
cat /etc/yum.repos.d/kubernetes.repo
```
If you see `baseurl` similar to the `baseurl` in the output below:
```
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v{{< skew currentVersionAddMinor -1 "." >}}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v{{< skew currentVersionAddMinor -1 "." >}}/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl
```
**You're using the Kubernetes package repositories and this guide applies to you.**
Otherwise, it's strongly recommend to migrate to the Kubernetes package repositories.
{{% /tab %}}
{{< /tabs >}}
{{< note >}}
The URL used for the Kubernetes package repositories is not limited to `pkgs.k8s.io`,
it can also be one of:
- `pkgs.k8s.io`
- `pkgs.kubernetes.io`
- `packages.kubernetes.io`
- `packages.kubernetes.io`
{{</ note >}}
<!-- steps -->
## Switching to another Kubernetes package repository
This step should be done upon upgrading from one to another Kubernetes minor
release in order to get access to the packages of the desired Kubernetes minor
version.
{{< tabs name="k8s_install_versions" >}}
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
1. Open the file that defines the Kubernetes `apt` repository using a text editor of your choice:
```shell
nano /etc/apt/sources.list.d/kubernetes.list
```
You should see a single line with the URL that contains your current Kubernetes
minor version. For example, if you're using v{{< skew currentVersionAddMinor -1 "." >}},
you should see this:
```
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v{{< skew currentVersionAddMinor -1 "." >}}/deb/ /
```
2. Change the version in the URL to **the next available minor release**, for example:
```
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/deb/ /
```
3. Save the file and exit your text editor. Continue following the relevant upgrade instructions.
{{% /tab %}}
{{% tab name="CentOS, RHEL or Fedora" %}}
1. Open the file that defines the Kubernetes `yum` repository using a text editor of your choice:
```shell
nano /etc/yum.repos.d/kubernetes.repo
```
You should see a file with two URLs that contain your current Kubernetes
minor version. For example, if you're using v{{< skew currentVersionAddMinor -1 "." >}},
you should see this:
```
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v{{< skew currentVersionAddMinor -1 "." >}}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v{{< skew currentVersionAddMinor -1 "." >}}/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
```
2. Change the version in these URLs to **the next available minor release**, for example:
```
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v{{< param "version" >}}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v{{< param "version" >}}/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
```
3. Save the file and exit your text editor. Continue following the relevant upgrade instructions.
{{% /tab %}}
{{< /tabs >}}
## {{% heading "whatsnext" %}}
* See how to [Upgrade Linux nodes](/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/).
* See how to [Upgrade Windows nodes](/docs/tasks/administer-cluster/kubeadm/upgrading-windows-nodes/).

5
content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md
View File

@ -36,6 +36,11 @@ driver of the kubelet.
{{< note >}}
In v1.22 and later, if the user does not set the `cgroupDriver` field under `KubeletConfiguration`,
kubeadm defaults it to `systemd`.
In Kubernetes v1.28, you can enable automatic detection of the
cgroup driver as an alpha feature.
See [systemd cgroup driver](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver)
for more details.
{{< /note >}}
A minimal example of configuring the field explicitly:

41
content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md
View File

@ -52,6 +52,13 @@ The upgrade workflow at high level is the following:
<!-- steps -->
## Changing the package repository
If you're using the Kubernetes community-owned repositories, you need to change
the package repository to one that contains packages for your desired Kubernetes
minor version. This is explained in [Changing the Kubernetes package repository](/docs/tasks/administer-cluster/kubeadm/change-package-repository/)
document.
## Determine which version to upgrade to
Find the latest patch release for Kubernetes {{< skew currentVersion >}} using the OS package manager:
@ -61,7 +68,7 @@ Find the latest patch release for Kubernetes {{< skew currentVersion >}} using t
```shell
# Find the latest {{< skew currentVersion >}} version in the list.
# It should look like {{< skew currentVersion >}}.x-00, where x is the latest patch.
# It should look like {{< skew currentVersion >}}.x-*, where x is the latest patch.
apt update
apt-cache madison kubeadm
```
@ -71,7 +78,7 @@ apt-cache madison kubeadm
```shell
# Find the latest {{< skew currentVersion >}} version in the list.
# It should look like {{< skew currentVersion >}}.x-0, where x is the latest patch.
# It should look like {{< skew currentVersion >}}.x-*, where x is the latest patch.
yum list --showduplicates kubeadm --disableexcludes=kubernetes
```
@ -93,9 +100,9 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-00 with the latest patch version
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \
apt-get update && apt-get install -y kubeadm='{{< skew currentVersion >}}.x-*' && \
apt-mark hold kubeadm
```
@ -103,8 +110,8 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc
{{% tab name="CentOS, RHEL or Fedora" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-0 with the latest patch version
yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
yum install -y kubeadm-'{{< skew currentVersion >}}.x-*' --disableexcludes=kubernetes
```
{{% /tab %}}
@ -152,6 +159,20 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
```
{{< note >}}
For versions earlier than v1.28, kubeadm defaulted to a mode that upgrades the addons (including CoreDNS and kube-proxy)
immediately during `kubeadm upgrade apply`, regardless of whether there are other control plane instances that have not
been upgraded. This may cause compatibility problems. Since v1.28, kubeadm defaults to a mode that checks whether all
the control plane instances have been upgraded before starting to upgrade the addons. You must perform control plane
instances upgrade sequentially or at least ensure that the last control plane instance upgrade is not started until all
the other control plane instances have been upgraded completely, and the addons upgrade will be performed after the last
control plane instance is upgraded. If you want to keep the old upgrade behavior, please enable the `UpgradeAddonsBeforeControlPlane`
feature gate by `kubeadm upgrade apply --feature-gates=UpgradeAddonsBeforeControlPlane=true`. The Kubernetes project does
not in general recommend enabling this feature gate, you should instead change your upgrade process or cluster addons so
that you do not need to enable the legacy behavior. The `UpgradeAddonsBeforeControlPlane` feature gate will be removed in
a future release.
{{</ note >}}
1. Manually upgrade your CNI provider plugin.
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow.
@ -193,9 +214,9 @@ kubectl drain <node-to-drain> --ignore-daemonsets
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-00 with the latest patch version
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \
apt-get update && apt-get install -y kubelet='{{< skew currentVersion >}}.x-*' kubectl='{{< skew currentVersion >}}.x-*' && \
apt-mark hold kubelet kubectl
```
@ -203,8 +224,8 @@ kubectl drain <node-to-drain> --ignore-daemonsets
{{% tab name="CentOS, RHEL or Fedora" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-0 with the latest patch version
yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
yum install -y kubelet-'{{< skew currentVersion >}}.x-*' kubectl-'{{< skew currentVersion >}}.x-*' --disableexcludes=kubernetes
```
{{% /tab %}}

23
content/en/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes.md
View File

@ -17,6 +17,13 @@ upgrade the control plane nodes before upgrading your Linux Worker nodes.
<!-- steps -->
## Changing the package repository
If you're using the Kubernetes community-owned repositories, you need to change
the package repository to one that contains packages for your desired Kubernetes
minor version. This is explained in [Changing the Kubernetes package repository](/docs/tasks/administer-cluster/kubeadm/change-package-repository/)
document.
## Upgrading worker nodes
### Upgrade kubeadm
@ -26,16 +33,16 @@ Upgrade kubeadm:
{{< tabs name="k8s_install_kubeadm_worker_nodes" >}}
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-00 with the latest patch version
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \
apt-get update && apt-get install -y kubeadm='{{< skew currentVersion >}}.x-*' && \
apt-mark hold kubeadm
```
{{% /tab %}}
{{% tab name="CentOS, RHEL or Fedora" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-0 with the latest patch version
yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
# replace x in {{< skew currentVersion >}}.x- with the latest patch version
yum install -y kubeadm-'{{< skew currentVersion >}}.x-*' --disableexcludes=kubernetes
```
{{% /tab %}}
{{< /tabs >}}
@ -64,16 +71,16 @@ kubectl drain <node-to-drain> --ignore-daemonsets
{{< tabs name="k8s_kubelet_and_kubectl" >}}
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-00 with the latest patch version
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \
apt-get update && apt-get install -y kubelet='{{< skew currentVersion >}}.x-*' kubectl='{{< skew currentVersion >}}.x-*' && \
apt-mark hold kubelet kubectl
```
{{% /tab %}}
{{% tab name="CentOS, RHEL or Fedora" %}}
```shell
# replace x in {{< skew currentVersion >}}.x-0 with the latest patch version
yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
# replace x in {{< skew currentVersion >}}.x-* with the latest patch version
yum install -y kubelet-'{{< skew currentVersion >}}.x-*' kubectl-'{{< skew currentVersion >}}.x-*' --disableexcludes=kubernetes
```
{{% /tab %}}
{{< /tabs >}}

46
content/en/docs/tasks/administer-cluster/kubelet-config-file.md
View File

@ -73,11 +73,53 @@ If `--config` is provided and the values are not specified via the command line,
defaults for the `KubeletConfiguration` version apply.
In the above example, this version is `kubelet.config.k8s.io/v1beta1`.
## Drop-in directory for Kubelet configuration files {#kubelet-conf-d}
As of Kubernetes v1.28.0, the Kubelet has been extended to support a drop-in configuration directory. The location of it can be specified with
`--config-dir` flag, and it defaults to `""`, or disabled, by default.
You can only set `--config-dir` if you set the environment variable `KUBELET_CONFIG_DROPIN_DIR_ALPHA` for the kubelet process (the value of that variable does not matter).
For Kubernetes v{{< skew currentVersion >}}, the kubelet returns an error if you specify `--config-dir` without that variable set, and startup fails.
You cannot specify the drop-in configuration directory using the kubelet configuration file; only the CLI argument `--config-dir` can set it.
One can use the Kubelet configuration directory in a similar way to the Kubelet config file.
{{< note >}}
The suffix of a valid Kubelet drop-in configuration file must be `.conf`. For instance: `99-kubelet-address.conf`
{{< /note >}}
For instance, you may want a baseline Kubelet configuration for all nodes, but you may want to customize the `address` field. This can be done as follows:
Main Kubelet configuration file contents:
```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
port: 20250
serializeImagePulls: false
evictionHard:
memory.available: "200Mi"
```
Contents of a file in `--config-dir` directory:
```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: "192.168.0.8"
```
On startup, the Kubelet merges configuration from:
* Command line arguments (lowest precedence).
* the Kubelet configuration
* Drop-in configuration files, according to sort order.
* Feature gates specified over the command line (highest precedence).
This produces the same outcome as if you used the [single configuration file](#create-the-config-file) used in the earlier example.
<!-- discussion -->
## {{% heading "whatsnext" %}}
- Learn more about kubelet configuration by checking the
[`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/)
reference.
reference.

24
content/en/docs/tasks/administer-cluster/topology-manager.md
View File

@ -28,7 +28,7 @@ disjoint set of components.
_Topology Manager_ is a Kubelet component that aims to coordinate the set of components that are
responsible for these optimizations.
## {{% heading "prerequisites" %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
@ -174,7 +174,7 @@ each Hint Provider to discover their resource availability. Using this informati
Manager stores the preferred NUMA Node affinity for that container. If the affinity is not
preferred, Topology Manager will store this and admit the pod to the node anyway.
The *Hint Providers* can then use this information when making the
The *Hint Providers* can then use this information when making the
resource allocation decision.
### restricted policy {#policy-restricted}
@ -190,7 +190,7 @@ reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigg
the pod. An external control loop could be also implemented to trigger a redeployment of pods that
have the `Topology Affinity` error.
If the pod is admitted, the *Hint Providers* can then use this information when making the
If the pod is admitted, the *Hint Providers* can then use this information when making the
resource allocation decision.
### single-numa-node policy {#policy-single-numa-node}
@ -211,14 +211,18 @@ that have the `Topology Affinity` error.
### Topology manager policy options
Support for the Topology Manager policy options requires `TopologyManagerPolicyOptions`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled.
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled
(it is enabled by default).
You can toggle groups of options on and off based upon their maturity level using the following feature gates:
* `TopologyManagerPolicyBetaOptions` default disabled. Enable to show beta-level options. Currently there are no beta-level options.
* `TopologyManagerPolicyAlphaOptions` default disabled. Enable to show alpha-level options. You will still have to enable each option using the `TopologyManagerPolicyOptions` kubelet option.
* `TopologyManagerPolicyBetaOptions` default enabled. Enable to show beta-level options.
* `TopologyManagerPolicyAlphaOptions` default disabled. Enable to show alpha-level options.
You will still have to enable each option using the `TopologyManagerPolicyOptions` kubelet option.
The following policy options exists:
* `prefer-closest-numa-nodes` (alpha, invisible by default, `TopologyManagerPolicyOptions` and `TopologyManagerPolicyAlphaOptions` feature gates have to be enabled)(1.26 or higher)
* `prefer-closest-numa-nodes` (beta, visible by default; `TopologyManagerPolicyOptions` and `TopologyManagerPolicyBetaOptions` feature gates have to be enabled).
The `prefer-closest-numa-nodes` policy option is beta in Kubernetes {{< skew currentVersion >}}.
If the `prefer-closest-numa-nodes` policy option is specified, the `best-effort` and `restricted`
policies will favor sets of NUMA nodes with shorter distance between them when making admission decisions.
@ -318,7 +322,7 @@ spec:
This pod runs in the `BestEffort` QoS class because there are no CPU and memory requests.
The Topology Manager would consider the above pods. The Topology Manager would consult the Hint
Providers, which are CPU and Device Manager to get topology hints for the pods.
Providers, which are CPU and Device Manager to get topology hints for the pods.
In the case of the `Guaranteed` pod with integer CPU request, the `static` CPU Manager policy
would return topology hints relating to the exclusive CPU and the Device Manager would send back
@ -337,7 +341,7 @@ of the requested devices.
Using this information the Topology Manager calculates the optimal hint for the pod and stores
this information, which will be used by the Hint Providers when they are making their resource
assignments.
assignments.
### Known Limitations
@ -346,4 +350,4 @@ assignments.
generating their hints.
2. The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail
on the node due to the Topology Manager.
on the node due to the Topology Manager.

18
content/en/docs/tasks/configure-pod-container/user-namespaces.md
View File

@ -9,9 +9,8 @@ min-kubernetes-server-version: v1.25
<!-- overview -->
{{< feature-state for_k8s_version="v1.25" state="alpha" >}}
This page shows how to configure a user namespace for stateless pods. This
allows to isolate the user running inside the container from the one in the
host.
This page shows how to configure a user namespace for pods. This allows you to
isolate the user running inside the container from the one in the host.
A process running as root in a container can run as a different (non-root) user
in the host; in other words, the process has full privileges for operations
@ -41,7 +40,14 @@ this is true when user namespaces are used.
* The node OS needs to be Linux
* You need to exec commands in the host
* You need to be able to exec into pods
* Feature gate `UserNamespacesStatelessPodsSupport` need to be enabled.
* You need to enable the `UserNamespacesSupport`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
{{< note >}}
The feature gate to enable user namespaces was previously named
`UserNamespacesStatelessPodsSupport`, when only stateless pods were supported.
Only Kubernetes v1.25 through to v1.27 recognise `UserNamespacesStatelessPodsSupport`.
{{</ note >}}
The cluster that you're using **must** include at least one node that meets the
[requirements](/docs/concepts/workloads/pods/user-namespaces/#before-you-begin)
@ -59,8 +65,8 @@ created without user namespaces.**
## Run a Pod that uses a user namespace {#create-pod}
A user namespace for a stateless pod is enabled setting the `hostUsers` field of
`.spec` to `false`. For example:
A user namespace for a pod is enabled setting the `hostUsers` field of `.spec`
to `false`. For example:
{{% code file="pods/user-namespaces-stateless.yaml" %}}

107
content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md
View File

@ -717,6 +717,57 @@ And create it:
kubectl apply -f my-crontab.yaml
crontab "my-new-cron-object" created
```
### Validation ratcheting
{{< feature-state state="alpha" for_k8s_version="v1.28" >}}
You need to enable the `CRDValidationRatcheting`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to
use this behavior, which then applies to all CustomResourceDefinitions in your
cluster.
Provided you enabled the feature gate, Kubernetes implements _validation racheting_
for CustomResourceDefinitions. The API server is willing accept updates to resources that
are not valid after the update, provided that each part of the resource that failed to validate
was not changed by the update operation. In other words, any invalid part of the resource
that remains invalid must have already been wrong.
You cannot use this mechanism to update a valid resource so that it becomes invalid.
This feature allows authors of CRDs to confidently add new validations to the
OpenAPIV3 schema under certain conditions. Users can update to the new schema
safely without bumping the version of the object or breaking workflows.
While most validations placed in the OpenAPIV3 schema of a CRD are support
ratcheting, there are a few exceptions. The following OpenAPIV3 schema
validations are not supported by ratcheting under the implementation in Kubernetes
{{< skew currentVersion >}} and if violated will continue to throw an error as normally:
- Quantors
- `allOf`
- `oneOf`
- `anyOf`
- `not`
- any validations in a descendent of one of these fields
- `x-kubernetes-validations`
For Kubernetes {{< skew currentVersion >}}, CRD validation rules](#validation-rules) are ignored by
ratcheting. This may change in later Kubernetes releases.
- `x-kubernetes-list-type`
Errors arising from changing the list type of a subschema will not be
ratcheted. For example adding `set` onto a list with duplicates will always
result in an error.
- `x-kubernetes-map-keys`
Errors arising from changing the map keys of a list schema will not be
ratcheted.
- `required`
Errors arising from changing the list of required fields will not be ratcheted.
- `properties`
Adding/removing/modifying the names of properties is not ratcheted, but
changes to validations in each properties' schemas and subschemas may be ratcheted
if the name of the property stays the same.
- `additionalProperties`
To remove a previously specified `additionalProperties` validation will not be
ratcheted.
## Validation rules
@ -1066,6 +1117,62 @@ message will be used instead.
`messageExpression` is a CEL expression, so the restrictions listed in [Resource use by validation functions](#resource-use-by-validation-functions) apply. If evaluation halts due to resource constraints
during `messageExpression` execution, then no further validation rules will be executed.
Setting `messageExpression` is optional.
#### The `message` field {#field-message}
If you want to set a static message, you can supply `message` rather than `messageExpression`.
The value of `message` is used as an opaque error string if validation fails.
Setting `message` is optional.
#### The `reason` field {#field-reason}
You can add a machine-readable validation failure reason within a `validation`, to be returned
whenever a request fails this validation rule.
For example:
```yaml
x-kubernetes-validations:
- rule: "self.x <= self.maxLimit"
reason: "FieldValueInvalid"
```
The HTTP status code returned to the caller will match the reason of the first failed validation rule.
The currently supported reasons are: "FieldValueInvalid", "FieldValueForbidden", "FieldValueRequired", "FieldValueDuplicate".
If not set or unknown reasons, default to use "FieldValueInvalid".
Setting `reason` is optional.
#### The `fieldPath` field {#field-field-path}
You can specify the field path returned when the validation fails.
For example:
```yaml
x-kubernetes-validations:
- rule: "self.foo.test.x <= self.maxLimit"
fieldPath: ".foo.test.x"
```
In the example above, the validation checks the value of field `x` should be less than the value of `maxLimit`.
If no `fieldPath` specified, when validation fails, the fieldPath would be default to wherever `self` scoped.
With `fieldPath` specified, the returned error will have `fieldPath` properly refer to the location of field `x`.
The `fieldPath` value must be a relative JSON path that is scoped to the location of this x-kubernetes-validations extension in the schema.
Additionally, it should refer to an existing field within the schema.
For example when validation checks if a specific attribute `foo` under a map `testMap`, you could set
`fieldPath` to `".testMap.foo"` or `.testMap['foo']'`.
If the validation requires checking for unique attributes in two lists, the fieldPath can be set to either of the lists.
For example, it can be set to `.testList1` or `.testList2`.
It supports child operation to refer to an existing field currently.
Refer to [JSONPath support in Kubernetes](/docs/reference/kubectl/jsonpath/) for more info.
The `fieldPath` field does not support indexing arrays numerically.
Setting `fieldPath` is optional.
#### Validation functions {#available-validation-functions}
Functions available include:

42
content/en/docs/tasks/tools/install-kubectl-linux.md
View File

@ -145,28 +145,29 @@ The following methods exist for installing kubectl on Linux:
```shell
sudo apt-get update
sudo apt-get install -y ca-certificates curl
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl
```
If you use Debian 9 (stretch) or earlier you would also need to install `apt-transport-https`:
2. Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories so you can disregard the version in the URL:
```shell
sudo apt-get install -y apt-transport-https
curl -fsSL https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
```
2. Download the Google Cloud public signing key:
3. Add the appropriate Kubernetes `apt` repository. If you want to use Kubernetes version different than {{< param "version" >}},
replace {{< param "version" >}} with the desired minor version in the command below:
```shell
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
3. Add the Kubernetes `apt` repository:
{{< note >}}
To upgrade kubectl to another minor release, you'll need to bump the version in `/etc/apt/sources.list.d/kubernetes.list` before running `apt-get update` and `apt-get upgrade`. This procedure is described in more detail in [Changing The Kubernetes Package Repository](/docs/tasks/administer-cluster/kubeadm/change-package-repository/).
{{< /note >}}
```shell
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
4. Update `apt` package index with the new repository and install kubectl:
4. Update `apt` package index, then install kubectl:
```shell
sudo apt-get update
@ -180,15 +181,30 @@ In releases older than Debian 12 and Ubuntu 22.04, `/etc/apt/keyrings` does not
{{% /tab %}}
{{% tab name="Red Hat-based distributions" %}}
1. Add the Kubernetes `yum` repository. If you want to use Kubernetes version
different than {{< param "version" >}}, replace {{< param "version" >}} with
the desired minor version in the command below.
```bash
# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
baseurl=https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
gpgkey=https://pkgs.k8s.io/core:/stable:/{{< param "version" >}}/rpm/repodata/repomd.xml.key
EOF
```
{{< note >}}
To upgrade kubectl to another minor release, you'll need to bump the version in `/etc/yum.repos.d/kubernetes.repo` before running `yum update`. This procedure is described in more detail in [Changing The Kubernetes Package Repository](/docs/tasks/administer-cluster/kubeadm/change-package-repository/).
{{< /note >}}
1. Install kubectl using `yum`:
```bash
sudo yum install -y kubectl
```

28
content/en/examples/access/image-matches-namespace-environment.policy.yaml Normal file
View File

@ -0,0 +1,28 @@
# This policy enforces that all containers of a deployment has the image repo match the environment label of its namespace.
# Except for "exempt" deployments, or any containers that do not belong to the "example.com" organization (e.g. common sidecars).
# For example, if the namespace has a label of {"environment": "staging"}, all container images must be either staging.example.com/*
# or do not contain "example.com" at all, unless the deployment has {"exempt": "true"} label.
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
name: "image-matches-namespace-environment.policy.example.com"
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
variables:
- name: environment
expression: "'environment' in namespaceObject.metadata.labels ? namespaceObject.metadata.labels['environment'] : 'prod'"
- name: exempt
expression: "'exempt' in object.metadata.labels && object.metadata.labels['exempt'] == 'true'"
- name: containers
expression: "object.spec.template.spec.containers"
- name: containersToCheck
expression: "variables.containers.filter(c, c.image.contains('example.com/'))"
validations:
- expression: "variables.exempt || variables.containersToCheck.all(c, c.image.startsWith(variables.environment + '.'))"
messageExpression: "'only ' + variables.environment + ' images are allowed in namespace ' + namespaceObject.metadata.name"

49
content/en/examples/access/validating-webhook-configuration-match-conditions.yaml Normal file
View File

@ -0,0 +1,49 @@
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ['CREATE','UPDATE']
apiGroups: ['*']
apiVersions: ['*']
resources: ['*']
failurePolicy: 'Ignore' # Fail-open (optional)
sideEffects: None
clientConfig:
service:
namespace: my-namespace
name: my-webhook
caBundle: '<omitted>'
# You can have up to 64 matchConditions per webhook
matchConditions:
- name: 'exclude-leases' # Each match condition must have a unique name
expression: '!(request.resource.group == "coordination.k8s.io" && request.resource.resource == "leases")' # Match non-lease resources.
- name: 'exclude-kubelet-requests'
expression: '!("system:nodes" in request.userInfo.groups)' # Match requests made by non-node users.
- name: 'rbac' # Skip RBAC requests, which are handled by the second webhook.
expression: 'request.resource.group != "rbac.authorization.k8s.io"'
# This example illustrates the use of the 'authorizer'. The authorization check is more expensive
# than a simple expression, so in this example it is scoped to only RBAC requests by using a second
# webhook. Both webhooks can be served by the same endpoint.
- name: rbac.my-webhook.example.com
matchPolicy: Equivalent
rules:
- operations: ['CREATE','UPDATE']
apiGroups: ['rbac.authorization.k8s.io']
apiVersions: ['*']
resources: ['*']
failurePolicy: 'Fail' # Fail-closed (the default)
sideEffects: None
clientConfig:
service:
namespace: my-namespace
name: my-webhook
caBundle: '<omitted>'
# You can have up to 64 matchConditions per webhook
matchConditions:
- name: 'breakglass'
# Skip requests made by users authorized to 'breakglass' on this webhook.
# The 'breakglass' API verb does not need to exist outside this check.
expression: '!authorizer.group("admissionregistration.k8s.io").resource("validatingwebhookconfigurations").name("my-webhook.example.com").check("breakglass").allowed()'

34
content/en/examples/application/deployment-sidecar.yaml Normal file
View File

@ -0,0 +1,34 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
labels:
app: myapp
spec:
replicas: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: alpine:latest
command: ['sh', '-c', 'echo "logging" > /opt/logs.txt']
volumeMounts:
- name: data
mountPath: /opt
initContainers:
- name: logshipper
image: alpine:latest
restartPolicy: Always
command: ['sh', '-c', 'tail /opt/logs.txt']
volumeMounts:
- name: data
mountPath: /opt
volumes:
- name: data
emptyDir: {}

26
content/en/examples/application/job/job-sidecar.yaml Normal file
View File

@ -0,0 +1,26 @@
apiVersion: batch/v1
kind: Job
metadata:
name: myjob
spec:
template:
spec:
containers:
- name: myjob
image: alpine:latest
command: ['sh', '-c', 'echo "logging" > /opt/logs.txt']
volumeMounts:
- name: data
mountPath: /opt
initContainers:
- name: logshipper
image: alpine:latest
restartPolicy: Always
command: ['sh', '-c', 'tail /opt/logs.txt']
volumeMounts:
- name: data
mountPath: /opt
restartPolicy: Never
volumes:
- name: data
emptyDir: {}

26
content/en/examples/controllers/job-backoff-limit-per-index-example.yaml Normal file
View File

@ -0,0 +1,26 @@
apiVersion: batch/v1
kind: Job
metadata:
name: job-backoff-limit-per-index-example
spec:
completions: 10
parallelism: 3
completionMode: Indexed # required for the feature
backoffLimitPerIndex: 1 # maximal number of failures per index
maxFailedIndexes: 5 # maximal number of failed indexes before terminating the Job execution
template:
spec:
restartPolicy: Never # required for the feature
containers:
- name: example
image: python
command: # The jobs fails as there is at least one failed index
# (all even indexes fail in here), yet all indexes
# are executed as maxFailedIndexes is not exceeded.
- python3
- -c
- |
import os, sys
print("Hello world")
if int(os.environ.get("JOB_COMPLETION_INDEX")) % 2 == 0:
sys.exit(1)

11
content/en/examples/validatingadmissionpolicy/basic-example-binding.yaml Normal file
View File

@ -0,0 +1,11 @@
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "demo-binding-test.example.com"
spec:
policyName: "demo-policy.example.com"
validationActions: [Deny]
matchResources:
namespaceSelector:
matchLabels:
environment: test

14
content/en/examples/validatingadmissionpolicy/basic-example-policy.yaml Normal file
View File

@ -0,0 +1,14 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
name: "demo-policy.example.com"
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.spec.replicas <= 5"

17
content/en/examples/validatingadmissionpolicy/binding-with-param-prod.yaml Normal file
View File

@ -0,0 +1,17 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "replicalimit-binding-nontest"
spec:
policyName: "replicalimit-policy.example.com"
validationActions: [Deny]
paramRef:
name: "replica-limit-prod.example.com"
namespace: "default"
matchResources:
namespaceSelector:
matchExpressions:
- key: environment
operator: NotIn
values:
- test

14
content/en/examples/validatingadmissionpolicy/binding-with-param.yaml Normal file
View File

@ -0,0 +1,14 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "replicalimit-binding-test.example.com"
spec:
policyName: "replicalimit-policy.example.com"
validationActions: [Deny]
paramRef:
name: "replica-limit-test.example.com"
namespace: "default"
matchResources:
namespaceSelector:
matchLabels:
environment: test

7
content/en/examples/validatingadmissionpolicy/failure-policy-ignore.yaml Normal file
View File

@ -0,0 +1,7 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
spec:
...
failurePolicy: Ignore # The default is "Fail"
validations:
- expression: "object.spec.xyz == params.x"

18
content/en/examples/validatingadmissionpolicy/policy-with-param.yaml Normal file
View File

@ -0,0 +1,18 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
name: "replicalimit-policy.example.com"
spec:
failurePolicy: Fail
paramKind:
apiVersion: rules.example.com/v1
kind: ReplicaLimit
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.spec.replicas <= params.maxReplicas"
reason: Invalid

5
content/en/examples/validatingadmissionpolicy/replicalimit-param-prod.yaml Normal file
View File

@ -0,0 +1,5 @@
apiVersion: rules.example.com/v1
kind: ReplicaLimit
metadata:
name: "replica-limit-prod.example.com"
maxReplicas: 100

6
content/en/examples/validatingadmissionpolicy/replicalimit-param.yaml Normal file
View File

@ -0,0 +1,6 @@
apiVersion: rules.example.com/v1
kind: ReplicaLimit
metadata:
name: "replica-limit-test.example.com"
namesapce: "default"
maxReplicas: 3

15
content/en/examples/validatingadmissionpolicy/typechecking-multiple-match.yaml Normal file
View File

@ -0,0 +1,15 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
name: "replica-policy.example.com"
spec:
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments","replicasets"]
validations:
- expression: "object.replicas > 1" # should be "object.spec.replicas > 1"
message: "must be replicated"
reason: Invalid

15
content/en/examples/validatingadmissionpolicy/typechecking.yaml Normal file
View File

@ -0,0 +1,15 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
name: "deploy-replica-policy.example.com"
spec:
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.replicas > 1" # should be "object.spec.replicas > 1"
message: "must be replicated"
reason: Invalid

24
content/en/releases/version-skew-policy.md
View File

@ -45,12 +45,13 @@ Example:
### kubelet
`kubelet` must not be newer than `kube-apiserver`, and may be up to two minor versions older.
* `kubelet` must not be newer than `kube-apiserver`.
* `kubelet` may be up to three minor versions older than `kube-apiserver` (`kubelet` < 1.25 may only be up to two minor versions older than `kube-apiserver`).
Example:
* `kube-apiserver` is at **{{< skew currentVersion >}}**
* `kubelet` is supported at **{{< skew currentVersion >}}**, **{{< skew currentVersionAddMinor -1 >}}**, and **{{< skew currentVersionAddMinor -2 >}}**
* `kubelet` is supported at **{{< skew currentVersion >}}**, **{{< skew currentVersionAddMinor -1 >}}**, **{{< skew currentVersionAddMinor -2 >}}**, and **{{< skew currentVersionAddMinor -3 >}}**
{{< note >}}
If version skew exists between `kube-apiserver` instances in an HA cluster, this narrows the allowed `kubelet` versions.
@ -59,17 +60,18 @@ If version skew exists between `kube-apiserver` instances in an HA cluster, this
Example:
* `kube-apiserver` instances are at **{{< skew currentVersion >}}** and **{{< skew currentVersionAddMinor -1 >}}**
* `kubelet` is supported at **{{< skew currentVersionAddMinor -1 >}}**, and **{{< skew currentVersionAddMinor -2 >}}** (**{{< skew currentVersion >}}** is not supported because that would be newer than the `kube-apiserver` instance at version **{{< skew currentVersionAddMinor -1 >}}**)
* `kubelet` is supported at **{{< skew currentVersionAddMinor -1 >}}**, **{{< skew currentVersionAddMinor -2 >}}**, and **{{< skew currentVersionAddMinor -3 >}}** (**{{< skew currentVersion >}}** is not supported because that would be newer than the `kube-apiserver` instance at version **{{< skew currentVersionAddMinor -1 >}}**)
### kube-proxy
`kube-proxy` must not be newer than `kube-apiserver`, and may be up to two minor versions older.
`kube-proxy` may be up to two minor versions older or newer than the `kubelet` instance it runs alongside.
* `kube-proxy` must not be newer than `kube-apiserver`.
* `kube-proxy` may be up to three minor versions older than `kube-apiserver` (`kube-proxy` < 1.25 may only be up to two minor versions older than `kube-apiserver`).
* `kube-proxy` may be up to three minor versions older or newer than the `kubelet` instance it runs alongside (`kube-proxy` < 1.25 may only be up to two minor versions older or newer than the `kubelet` instance it runs alongside).
Example:
* `kube-apiserver` is at **{{< skew currentVersion >}}**
* `kube-proxy` is supported at **{{< skew currentVersion >}}**, **{{< skew currentVersionAddMinor -1 >}}**, and **{{< skew currentVersionAddMinor -2 >}}**
* `kube-proxy` is supported at **{{< skew currentVersion >}}**, **{{< skew currentVersionAddMinor -1 >}}**, **{{< skew currentVersionAddMinor -2 >}}**, and **{{< skew currentVersionAddMinor -3 >}}**
{{< note >}}
If version skew exists between `kube-apiserver` instances in an HA cluster, this narrows the allowed `kube-proxy` versions.
@ -78,7 +80,7 @@ If version skew exists between `kube-apiserver` instances in an HA cluster, this
Example:
* `kube-apiserver` instances are at **{{< skew currentVersion >}}** and **{{< skew currentVersionAddMinor -1 >}}**
* `kube-proxy` is supported at **{{< skew currentVersionAddMinor -1 >}}**, and **{{< skew currentVersionAddMinor -2 >}}** (**{{< skew currentVersion >}}** is not supported because that would be newer than the `kube-apiserver` instance at version **{{< skew currentVersionAddMinor -1 >}}**)
* `kube-proxy` is supported at **{{< skew currentVersionAddMinor -1 >}}**, **{{< skew currentVersionAddMinor -2 >}}**, and **{{< skew currentVersionAddMinor -3 >}}** (**{{< skew currentVersion >}}** is not supported because that would be newer than the `kube-apiserver` instance at version **{{< skew currentVersionAddMinor -1 >}}**)
### kube-controller-manager, kube-scheduler, and cloud-controller-manager
@ -173,7 +175,7 @@ Pre-requisites:
* The `kube-apiserver` instances the `kubelet` communicates with are at **{{< skew currentVersion >}}**
Optionally upgrade `kubelet` instances to **{{< skew currentVersion >}}** (or they can be left at **{{< skew currentVersionAddMinor -1 >}}** or **{{< skew currentVersionAddMinor -2 >}}**)
Optionally upgrade `kubelet` instances to **{{< skew currentVersion >}}** (or they can be left at **{{< skew currentVersionAddMinor -1 >}}**, **{{< skew currentVersionAddMinor -2 >}}**, or **{{< skew currentVersionAddMinor -3 >}}**)
{{< note >}}
Before performing a minor version `kubelet` upgrade, [drain](/docs/tasks/administer-cluster/safely-drain-node/) pods from that node.
@ -181,7 +183,7 @@ In-place minor version `kubelet` upgrades are not supported.
{{</ note >}}
{{< warning >}}
Running a cluster with `kubelet` instances that are persistently two minor versions behind `kube-apiserver` means they must be upgraded before the control plane can be upgraded.
Running a cluster with `kubelet` instances that are persistently three minor versions behind `kube-apiserver` means they must be upgraded before the control plane can be upgraded.
{{</ warning >}}
### kube-proxy
@ -190,8 +192,8 @@ Pre-requisites:
* The `kube-apiserver` instances `kube-proxy` communicates with are at **{{< skew currentVersion >}}**
Optionally upgrade `kube-proxy` instances to **{{< skew currentVersion >}}** (or they can be left at **{{< skew currentVersionAddMinor -1 >}}** or **{{< skew currentVersionAddMinor -2 >}}**)
Optionally upgrade `kube-proxy` instances to **{{< skew currentVersion >}}** (or they can be left at **{{< skew currentVersionAddMinor -1 >}}**, **{{< skew currentVersionAddMinor -2 >}}**, or **{{< skew currentVersionAddMinor -3 >}}**)
{{< warning >}}
Running a cluster with `kube-proxy` instances that are persistently two minor versions behind `kube-apiserver` means they must be upgraded before the control plane can be upgraded.
Running a cluster with `kube-proxy` instances that are persistently three minor versions behind `kube-apiserver` means they must be upgraded before the control plane can be upgraded.
{{</ warning >}}

26
hugo.toml
View File

@ -138,9 +138,9 @@ time_format_default = "January 02, 2006 at 3:04 PM PST"
description = "Production-Grade Container Orchestration"
showedit = true
latest = "v1.27"
latest = "v1.28"
version = "v1.27"
version = "v1.28"
githubbranch = "main"
docsbranch = "main"
deprecated = false
@ -180,35 +180,35 @@ js = [
]
[[params.versions]]
version = "v1.27"
githubbranch = "v1.27.0"
version = "v1.28"
githubbranch = "v1.28.0"
docsbranch = "main"
url = "https://kubernetes.io"
[[params.versions]]
version = "v1.27"
githubbranch = "v1.27.4"
docsbranch = "release-1.27"
url = "https://v1-27.docs.kubernetes.io"
[[params.versions]]
version = "v1.26"
githubbranch = "v1.26.3"
githubbranch = "v1.26.7"
docsbranch = "release-1.26"
url = "https://v1-26.docs.kubernetes.io"
[[params.versions]]
version = "v1.25"
githubbranch = "v1.25.8"
githubbranch = "v1.25.12"
docsbranch = "release-1.25"
url = "https://v1-25.docs.kubernetes.io"
[[params.versions]]
version = "v1.24"
githubbranch = "v1.24.12"
githubbranch = "v1.24.16"
docsbranch = "release-1.24"
url = "https://v1-24.docs.kubernetes.io"
[[params.versions]]
version = "v1.23"
githubbranch = "v1.23.17"
docsbranch = "release-1.23"
url = "https://v1-23.docs.kubernetes.io"
# User interface configuration
[params.ui]
# Enable to show the side bar menu in its compact state.