Merge pull request #28290 from shannonxtreme/api-eviction
Add information to API Evictions from Safely Drain a Node
This commit is contained in:
Kubernetes Prow Robot
GitHub
commit
ee0389d360
|
|
@ -6,14 +6,117 @@ weight: 70
|
|||
|
||||
{{< glossary_definition term_id="api-eviction" length="short" >}} </br>
|
||||
|
||||
You can request eviction by directly calling the Eviction API
|
||||
using a client of the kube-apiserver, like the `kubectl drain` command.
|
||||
This creates an `Eviction` object, which causes the API server to terminate the Pod.
|
||||
You can request eviction by calling the Eviction API directly, or programmatically
|
||||
using a client of the {{<glossary_tooltip term_id="kube-apiserver" text="API server">}}, like the `kubectl drain` command. This
|
||||
creates an `Eviction` object, which causes the API server to terminate the Pod.
|
||||
|
||||
API-initiated evictions respect your configured [`PodDisruptionBudgets`](/docs/tasks/run-application/configure-pdb/)
|
||||
and [`terminationGracePeriodSeconds`](/docs/concepts/workloads/pods/pod-lifecycle#pod-termination).
|
||||
|
||||
Using the API to create an Eviction object for a Pod is like performing a
|
||||
policy-controlled [`DELETE` operation](/docs/reference/kubernetes-api/workload-resources/pod-v1/#delete-delete-a-pod)
|
||||
on the Pod.
|
||||
|
||||
## Calling the Eviction API
|
||||
|
||||
You can use a [Kubernetes language client](/docs/tasks/administer-cluster/access-cluster-api/#programmatic-access-to-the-api)
|
||||
to access the Kubernetes API and create an `Eviction` object. To do this, you
|
||||
POST the attempted operation, similar to the following example:
|
||||
|
||||
{{< tabs name="Eviction_example" >}}
|
||||
{{% tab name="policy/v1" %}}
|
||||
{{< note >}}
|
||||
`policy/v1` Eviction is available in v1.22+. Use `policy/v1beta1` with prior releases.
|
||||
{{< /note >}}
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
"namespace": "default"
|
||||
}
|
||||
}
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="policy/v1beta1" %}}
|
||||
{{< note >}}
|
||||
Deprecated in v1.22 in favor of `policy/v1`
|
||||
{{< /note >}}
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1beta1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
"namespace": "default"
|
||||
}
|
||||
}
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
Alternatively, you can attempt an eviction operation by accessing the API using
|
||||
`curl` or `wget`, similar to the following example:
|
||||
|
||||
```bash
|
||||
curl -v -H 'Content-type: application/json' https://your-cluster-api-endpoint.example/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
|
||||
```
|
||||
|
||||
## How API-initiated eviction works
|
||||
|
||||
When you request an eviction using the API, the API server performs admission
|
||||
checks and responds in one of the following ways:
|
||||
|
||||
* `200 OK`: the eviction is allowed, the `Eviction` subresource is created, and
|
||||
the Pod is deleted, similar to sending a `DELETE` request to the Pod URL.
|
||||
* `429 Too Many Requests`: the eviction is not currently allowed because of the
|
||||
configured {{<glossary_tooltip term_id="pod-disruption-budget" text="PodDisruptionBudget">}}.
|
||||
You may be able to attempt the eviction again later. You might also see this
|
||||
response because of API rate limiting.
|
||||
* `500 Internal Server Error`: the eviction is not allowed because there is a
|
||||
misconfiguration, like if multiple PodDisruptionBudgets reference the same Pod.
|
||||
|
||||
If the Pod you want to evict isn't part of a workload that has a
|
||||
PodDisruptionBudget, the API server always returns `200 OK` and allows the
|
||||
eviction.
|
||||
|
||||
If the API server allows the eviction, the Pod is deleted as follows:
|
||||
|
||||
1. The `Pod` resource in the API server is updated with a deletion timestamp,
|
||||
after which the API server considers the `Pod` resource to be terminated. The
|
||||
`Pod` resource is also marked with the configured grace period.
|
||||
1. The {{<glossary_tooltip term_id="kubelet" text="kubelet">}} on the node where the local Pod is running notices that the `Pod`
|
||||
resource is marked for termination and starts to gracefully shut down the
|
||||
local Pod.
|
||||
1. While the kubelet is shutting the Pod down, the control plane removes the Pod
|
||||
from {{<glossary_tooltip term_id="endpoint" text="Endpoint">}} and
|
||||
{{<glossary_tooltip term_id="endpoint-slice" text="EndpointSlice">}}
|
||||
objects. As a result, controllers no longer consider the Pod as a valid object.
|
||||
1. After the grace period for the Pod expires, the kubelet forcefully terminates
|
||||
the local Pod.
|
||||
1. The kubelet tells the API server to remove the `Pod` resource.
|
||||
1. The API server deletes the `Pod` resource.
|
||||
|
||||
## Troubleshooting stuck evictions
|
||||
|
||||
In some cases, your applications may enter a broken state, where the Eviction
|
||||
API will only return `429` or `500` responses until you intervene. This can
|
||||
happen if, for example, a ReplicaSet creates pods for your application but new
|
||||
pods do not enter a `Ready` state. You may also notice this behavior in cases
|
||||
where the last evicted Pod had a long termination grace period.
|
||||
|
||||
If you notice stuck evictions, try one of the following solutions:
|
||||
|
||||
* Abort or pause the automated operation causing the issue. Investigate the stuck
|
||||
application before you restart the operation.
|
||||
* Wait a while, then directly delete the Pod from your cluster control plane
|
||||
instead of using the Eviction API.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
* Learn about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
|
||||
* Learn about [Pod Priority and Preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/)
|
||||
* Learn how to protect your applications with a [Pod Disruption Budget](/docs/tasks/run-application/configure-pdb/).
|
||||
* Learn about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/).
|
||||
* Learn about [Pod Priority and Preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/).
|
||||
|
|
|
|||
|
|
@ -23,8 +23,6 @@ This task also assumes that you have met the following prerequisites:
|
|||
and have [configured PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb/) for
|
||||
applications that need them.
|
||||
|
||||
|
||||
|
||||
<!-- steps -->
|
||||
|
||||
## (Optional) Configure a disruption budget {#configure-poddisruptionbudget}
|
||||
|
|
@ -100,95 +98,12 @@ replicas to fall below the specified budget are blocked.
|
|||
|
||||
If you prefer not to use [kubectl drain](/docs/reference/generated/kubectl/kubectl-commands/#drain) (such as
|
||||
to avoid calling to an external command, or to get finer control over the pod
|
||||
eviction process), you can also programmatically cause evictions using the eviction API.
|
||||
eviction process), you can also programmatically cause evictions using the
|
||||
eviction API.
|
||||
|
||||
You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api/#programmatic-access-to-the-api) to access the API.
|
||||
|
||||
The eviction subresource of a
|
||||
Pod can be thought of as a kind of policy-controlled DELETE operation on the Pod
|
||||
itself. To attempt an eviction (more precisely: to attempt to
|
||||
*create* an Eviction), you POST an attempted operation. Here's an example:
|
||||
|
||||
{{< tabs name="Eviction_example" >}}
|
||||
{{% tab name="policy/v1" %}}
|
||||
{{< note >}}
|
||||
`policy/v1` Eviction is available in v1.22+. Use `policy/v1beta1` with prior releases.
|
||||
{{< /note >}}
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
"namespace": "default"
|
||||
}
|
||||
}
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="policy/v1beta1" %}}
|
||||
{{< note >}}
|
||||
Deprecated in v1.22 in favor of `policy/v1`
|
||||
{{< /note >}}
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1beta1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
"namespace": "default"
|
||||
}
|
||||
}
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
You can attempt an eviction using `curl`:
|
||||
|
||||
```bash
|
||||
curl -v -H 'Content-type: application/json' https://your-cluster-api-endpoint.example/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
|
||||
```
|
||||
|
||||
The API can respond in one of three ways:
|
||||
|
||||
- If the eviction is granted, then the Pod is deleted as if you sent
|
||||
a `DELETE` request to the Pod's URL and received back `200 OK`.
|
||||
- If the current state of affairs wouldn't allow an eviction by the rules set
|
||||
forth in the budget, you get back `429 Too Many Requests`. This is
|
||||
typically used for generic rate limiting of *any* requests, but here we mean
|
||||
that this request isn't allowed *right now* but it may be allowed later.
|
||||
- If there is some kind of misconfiguration; for example multiple PodDisruptionBudgets
|
||||
that refer the same Pod, you get a `500 Internal Server Error` response.
|
||||
|
||||
For a given eviction request, there are two cases:
|
||||
|
||||
- There is no budget that matches this pod. In this case, the server always
|
||||
returns `200 OK`.
|
||||
- There is at least one budget. In this case, any of the three above responses may
|
||||
apply.
|
||||
|
||||
## Stuck evictions
|
||||
|
||||
In some cases, an application may reach a broken state, one where unless you intervene the
|
||||
eviction API will never return anything other than 429 or 500.
|
||||
|
||||
For example: this can happen if ReplicaSet is creating Pods for your application but
|
||||
the replacement Pods do not become `Ready`. You can also see similar symptoms if the
|
||||
last Pod evicted has a very long termination grace period.
|
||||
|
||||
In this case, there are two potential solutions:
|
||||
|
||||
- Abort or pause the automated operation. Investigate the reason for the stuck application,
|
||||
and restart the automation.
|
||||
- After a suitably long wait, `DELETE` the Pod from your cluster's control plane, instead
|
||||
of using the eviction API.
|
||||
|
||||
Kubernetes does not specify what the behavior should be in this case; it is up to the
|
||||
application owners and cluster owners to establish an agreement on behavior in these cases.
|
||||
For more information, see [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/).
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
|
||||
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application/configure-pdb/).
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue