document dynamic resource allocation concepts
The documentation intentionally only gives an overview. The API documentation and the KEP have more details. Specific examples will be provided by hardware vendors that provide resource drivers. The new page gets placed between "Scheduling Framework" (because that introduces the concept of a scheduler plugin) and "Scheduler Performance Tuning" (which seems like a more advanced topic).
This commit is contained in:
Patrick Ohly
parent
b5aafaaac4
commit
197de4f5ba
|
|
@ -26,6 +26,7 @@ of terminating one or more Pods on Nodes.
|
|||
* [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/)
|
||||
* [Taints and Tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/)
|
||||
* [Scheduling Framework](/docs/concepts/scheduling-eviction/scheduling-framework)
|
||||
* [Dynamic Resource Allocation](/docs/concepts/scheduling-eviction/dynamic-resource-allocation)
|
||||
* [Scheduler Performance Tuning](/docs/concepts/scheduling-eviction/scheduler-perf-tuning/)
|
||||
* [Resource Bin Packing for Extended Resources](/docs/concepts/scheduling-eviction/resource-bin-packing/)
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,215 @@
|
|||
---
|
||||
reviewers:
|
||||
- klueska
|
||||
- pohly
|
||||
title: Dynamic Resource Allocation
|
||||
content_type: concept
|
||||
weight: 65
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
{{< feature-state for_k8s_version="v1.26" state="alpha" >}}
|
||||
|
||||
Dynamic resource allocation is a new API for requesting and sharing resources
|
||||
between pods and containers inside a pod. It is a generalization of the
|
||||
persistent volumes API for generic resources. Third-party resource drivers are
|
||||
responsible for tracking and allocating resources. Different kinds of
|
||||
resources support arbitrary parameters for defining requirements and
|
||||
initialization.
|
||||
|
||||
## {{% heading "prerequisites" %}}
|
||||
|
||||
Kubernetes v{{< skew currentVersion >}} includes cluster-level API support for
|
||||
dynamic resource allocation, but it [needs to be
|
||||
enabled](#enabling-dynamic-resource-allocation) explicitly. You also must
|
||||
install a resource driver for specific resources that are meant to be managed
|
||||
using this API. If you are not running Kubernetes v{{< skew currentVersion>}},
|
||||
check the documentation for that version of Kubernetes.
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## API
|
||||
|
||||
The new `resource.k8s.io/v1alpha1` {{< glossary_tooltip text="API group"
|
||||
term_id="api-group" >}} provides four new types:
|
||||
|
||||
ResourceClass
|
||||
: Defines which resource driver handles a certain kind of
|
||||
resource and provides common parameters for it. ResourceClasses
|
||||
are created by a cluster administrator when installing a resource
|
||||
driver.
|
||||
|
||||
ResourceClaim
|
||||
: Defines a particular resource instances that is required by a
|
||||
workload. Created by a user (lifecycle managed manually, can be shared
|
||||
between different Pods) or for individual Pods by the control plane based on
|
||||
a ResourceClaimTemplate (automatic lifecycle, typically used by just one
|
||||
Pod).
|
||||
|
||||
ResourceClaimTemplate
|
||||
: Defines the spec and some meta data for creating
|
||||
ResourceClaims. Created by a user when deploying a workload.
|
||||
|
||||
PodScheduling
|
||||
: Used internally by the control plane and resource drivers
|
||||
to coordinate pod scheduling when ResourceClaims need to be allocated
|
||||
for a Pod.
|
||||
|
||||
Parameters for ResourceClass and ResourceClaim are stored in separate objects,
|
||||
typically using the type defined by a {{< glossary_tooltip
|
||||
term_id="CustomResourceDefinition" text="CRD" >}} that was created when
|
||||
installing a resource driver.
|
||||
|
||||
The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a new
|
||||
`resourceClaims` field. Entries in that list reference either a ResourceClaim
|
||||
or a ResourceClaimTemplate. When referencing a ResourceClaim, all Pods using
|
||||
this PodSpec (for example, inside a Deployment or StatefulSet) share the same
|
||||
ResourceClaim instance. When referencing a ResourceClaimTemplate, each Pod gets
|
||||
its own instance.
|
||||
|
||||
The `resources.claims` list for container resources defines whether a container gets
|
||||
access to these resource instances, which makes it possible to share resources
|
||||
between one or more containers.
|
||||
|
||||
Here is an example for a fictional resource driver. Two ResourceClaim objects
|
||||
will get created for this Pod and each container gets access to one of them.
|
||||
|
||||
```yaml
|
||||
apiVersion: resource.k8s.io/v1alpha1
|
||||
kind: ResourceClass
|
||||
name: resource.example.com
|
||||
driverName: resource-driver.example.com
|
||||
---
|
||||
apiVersion: cats.resource.example.com/v1
|
||||
kind: ClaimParameters
|
||||
name: large-black-cat-claim-parameters
|
||||
spec:
|
||||
color: black
|
||||
size: large
|
||||
---
|
||||
apiVersion: resource.k8s.io/v1alpha1
|
||||
kind: ResourceClaimTemplate
|
||||
metadata:
|
||||
name: large-black-cat-claim-template
|
||||
spec:
|
||||
spec:
|
||||
resourceClassName: resource.example.com
|
||||
parametersRef:
|
||||
apiGroup: cats.resource.example.com
|
||||
kind: ClaimParameters
|
||||
name: large-black-cat-claim-parameters
|
||||
–--
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: pod-with-cats
|
||||
spec:
|
||||
containers:
|
||||
- name: container0
|
||||
image: ubuntu:20.04
|
||||
command: ["sleep", "9999"]
|
||||
resources:
|
||||
claims:
|
||||
- name: cat-0
|
||||
- name: container1
|
||||
image: ubuntu:20.04
|
||||
command: ["sleep", "9999"]
|
||||
resources:
|
||||
claims:
|
||||
- name: cat-1
|
||||
resourceClaims:
|
||||
- name: cat-0
|
||||
source:
|
||||
resourceClaimTemplateName: large-black-cat-claim-template
|
||||
- name: cat-1
|
||||
source:
|
||||
resourceClaimTemplateName: large-black-cat-claim-template
|
||||
```
|
||||
|
||||
## Scheduling
|
||||
|
||||
In contrast to native resources (CPU, RAM) and extended resources (managed by a
|
||||
device plugin, advertised by kubelet), the scheduler has no knowledge of what
|
||||
dynamic resources are available in a cluster or how they could be split up to
|
||||
satisfy the requirements of a specific ResourceClaim. Resource drivers are
|
||||
responsible for that. They mark ResourceClaims as "allocated" once resources
|
||||
for it are reserved. This also then tells the scheduler where in the cluster a
|
||||
ResourceClaim is available.
|
||||
|
||||
ResourceClaims can get allocated as soon as they are created ("immediate
|
||||
allocation"), without considering which Pods will use them. The default is to
|
||||
delay allocation until a Pod gets scheduled which needs the ResourceClaim
|
||||
(i.e. "wait for first consumer").
|
||||
|
||||
In that mode, the scheduler checks all ResourceClaims needed by a Pod and
|
||||
creates a PodScheduling object where it informs the resource drivers
|
||||
responsible for those ResourceClaims about nodes that the scheduler considers
|
||||
suitable for the Pod. The resource drivers respond by excluding nodes that
|
||||
don't have enough of the driver's resources left. Once the scheduler has that
|
||||
information, it selects one node and stores that choice in the PodScheduling
|
||||
object. The resource drivers then allocate their ResourceClaims so that the
|
||||
resources will be available on that node. Once that is complete, the Pod
|
||||
gets scheduled.
|
||||
|
||||
As part of this process, ResourceClaims also get reserved for the
|
||||
Pod. Currently ResourceClaims can either be used exclusively by a single Pod or
|
||||
an unlimited number of Pods.
|
||||
|
||||
One key feature is that Pods do not get scheduled to a node unless all of
|
||||
their resources are allocated and reserved. This avoids the scenario where a Pod
|
||||
gets scheduled onto one node and then cannot run there, which is bad because
|
||||
such a pending Pod also blocks all other resources like RAM or CPU that were
|
||||
set aside for it.
|
||||
|
||||
## Limitations
|
||||
|
||||
The scheduler plugin must be involved in scheduling Pods which use
|
||||
ResourceClaims. Bypassing the scheduler by setting the `nodeName` field leads
|
||||
to Pods that the kubelet refuses to start because the ResourceClaims are not
|
||||
reserved or not even allocated. It may be possible to [remove this
|
||||
limitation](https://github.com/kubernetes/kubernetes/issues/114005) in the
|
||||
future.
|
||||
|
||||
## Enabling dynamic resource allocation
|
||||
|
||||
Dynamic resource allocation is an *alpha feature* and only enabled when the
|
||||
`DynamicResourceAllocation` [feature
|
||||
gate](/docs/reference/command-line-tools-reference/feature-gates/) and the
|
||||
`resource.k8s.io/v1alpha1` {{< glossary_tooltip text="API group"
|
||||
term_id="api-group" >}} are enabled. For details on that, see the
|
||||
`--feature-gates` and `--runtime-config` [kube-apiserver
|
||||
parameters](/docs/reference/command-line-tools-reference/kube-apiserver/).
|
||||
kube-scheduler, kube-controller-manager and kubelet also need the feature gate.
|
||||
|
||||
A quick check whether a Kubernetes cluster supports the feature is to list
|
||||
ResourceClass objects with:
|
||||
|
||||
```shell
|
||||
kubectl get resourceclasses
|
||||
```
|
||||
|
||||
If your cluster supports dynamic resource allocation, the response is either a
|
||||
list of ResourceClass objects or:
|
||||
|
||||
```
|
||||
No resources found
|
||||
```
|
||||
|
||||
If not supported, this error is printed instead:
|
||||
|
||||
```
|
||||
error: the server doesn't have a resource type "resourceclasses"
|
||||
```
|
||||
|
||||
The default configuration of kube-scheduler enables the "DynamicResources"
|
||||
plugin if and only if the feature gate is enabled. Custom configurations may
|
||||
have to be modified to include it.
|
||||
|
||||
In addition to enabling the feature in the cluster, a resource driver also has to
|
||||
be installed. Please refer to the driver's documentation for details.
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
- For more information on the design, see the
|
||||
[Dynamic Resource Allocation KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3063-dynamic-resource-allocation/README.md).
|
||||
|
|
@ -87,6 +87,7 @@ For a reference to old feature gates that are removed, please refer to
|
|||
| `DownwardAPIHugePages` | `false` | Alpha | 1.20 | 1.20 |
|
||||
| `DownwardAPIHugePages` | `false` | Beta | 1.21 | 1.21 |
|
||||
| `DownwardAPIHugePages` | `true` | Beta | 1.22 | |
|
||||
| `DynamicResourceAllocation` | `false` | Alpha | 1.26 | |
|
||||
| `EndpointSliceTerminatingCondition` | `false` | Alpha | 1.20 | 1.21 |
|
||||
| `EndpointSliceTerminatingCondition` | `true` | Beta | 1.22 | |
|
||||
| `ExpandedDNSConfig` | `false` | Alpha | 1.22 | |
|
||||
|
|
|
|||
Loading…
Reference in New Issue