kubernetes
/
autoscaler
mirror of https://github.com/kubernetes/autoscaler.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Introduce AEP with Provisioning Request CRD

This commit is contained in:
Daniel Gutowski 2023-06-13 02:11:26 -07:00
parent 1aaf4d0152
commit 09fb9591cf
1 changed files with 326 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

326
cluster-autoscaler/proposals/provisioning-request.md Normal file
View File

@ -0,0 +1,326 @@
# Provisioning Request CRD
author: kisieland
## Background
Currently CA does not provide any way to express that a group of pods would like
to have a capacity available.
This is caused by the fact that each CA loop picks a group of unschedulable pods
and works on provisioning capacity for them, meaning that the grouping is random
(as it depends on the kube-scheduler and CA loop interactions).
This is especially problematic in couple of cases:
- Users would like to have all-or-nothing semantics for their workloads.
Currently CA will try to provision this capacity and if it is partially
successful it will leave it in cluster until user removes the workload.
- Users would like to lower e2e scale-up latency for huge scale-ups (100
nodes+). Due to CA nature and kube-scheduler throughput, CA will create
partial scale-ups, e.g. `0->200->400->600` rather than one `0->600`. This
significantly increases the e2e latency as there is non-negligible time tax
on each scale-up operation.
## Proposal
### High level
Provisioning Request (abbr. ProvReq) is a new namespaced Custom Resource that
aims to allow users to ask CA for capacity for groups of pods.
It allows users to express the fact that group of pods is connected and should
be threated as one entity.
This AEP proposes an API that can have multiple provisioning classes and can be
extended by cloud provider specific ones.
This object is meant as one-shot request to CA, so that if CA fails to provision
the capacity it is up to users to retry (such retry functionality can be added
later on).
### ProvisioningRequest CRD
The following code snippets assume [kubebuilder](https://book.kubebuilder.io/)
is used to generate the CRD:
```go
// ProvisioningRequest is a way to express additional capacity
// that we would like to provision in the cluster. Cluster Autoscaler
// can use this information in its calculations and signal if the capacity
// is available in the cluster or actively add capacity if needed.
type ProvisioningRequest struct {
metav1.TypeMeta `json:",inline"`
// Standard object metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata
//
// +optional
metav1.ObjectMeta `json:"metadata,omitempty"`
// Spec contains specification of the ProvisioningRequest object.
// More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.
//
// +kubebuilder:validation:Required
Spec ProvisioningRequestSpec `json:"spec"`
// Status of the ProvisioningRequest. CA constantly reconciles this field.
//
// +optional
Status ProvisioningRequestStatus `json:"status,omitempty"`
}
// ProvisioningRequestList is a object for list of ProvisioningRequest.
type ProvisioningRequestList struct {
metav1.TypeMeta `json:",inline"`
// Standard list metadata.
//
// +optional
metav1.ListMeta `json:"metadata"`
// Items, list of ProvisioningRequest returned from API.
//
// +optional
Items []ProvisioningRequest `json:"items"`
}
// ProvisioningRequestSpec is a specification of additional pods for which we
// would like to provision additional resources in the cluster.
type ProvisioningRequestSpec struct {
// PodSets lists groups of pods for which we would like to provision
// resources.
//
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinItems=1
// +kubebuilder:validation:MaxItems=32
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
PodSets []PodSet `json:"podSets"`
// ProvisioningClass describes the different modes of provisioning the resources.
// Supported values:
// * check-capacity.kubernetes.io - check if current cluster state can fullfil this request,
// do not reserve the capacity.
// * atomic-scale-up.kubernetes.io - provision the resources in an atomic manner
// * ... - potential other classes that are specific to the cloud providers
//
// +kubebuilder:validation:Required
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
ProvisioningClass string `json:"provisioningClass"`
// AdditionalParameters contains all other parameters custom classes may require.
//
// +optional
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
AdditionalParameters map[string]string `json:"additionalParameters"`
}
type PodSet struct {
// PodTemplateRef is a reference to a PodTemplate object that is representing pods
// that will consume this reservation (must be within the same namespace).
// Users need to make sure that the fields relevant to scheduler (e.g. node selector tolerations)
// are consistent between this template and actual pods consuming the Provisioning Request.
//
// +kubebuilder:validation:Required
PodTemplateRef Reference `json:"podTemplateRef"`
// Count contains the number of pods that will be created with a given
// template.
//
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=16384
Count int32 `json:"count"`
}
type Reference struct {
// Name of the referenced object.
// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names
//
// +kubebuilder:validation:Required
Name string `json:"name,omitempty"`
}
// ProvisioningRequestStatus represents the status of the resource reservation.
type ProvisioningRequestStatus struct {
// Conditions represent the observations of a Provisioning Request's
// current state. Those will contain information whether the capacity
// was found/created or if there were any issues. The condition types
// may differ between different provisioning classes.
//
// +listType=map
// +listMapKey=type
// +patchStrategy=merge
// +patchMergeKey=type
// +optional
Conditions []metav1.Condition `json:"conditions"`
// AdditionalStatus contains all other status values custom provisioning classes may require.
//
// +optional
// +kubebuilder:validation:MaxItems=64
AdditionalStatus map[string]string `json:"additionalStatus"`
}
```
### Provisioning Classes
#### check-capacity.kubernetes.io class
The `check-capacity.kubernetes.io` is one-off check to verify that the in the cluster
there is enough capacity to provision given set of pods.
Note: If two of such objects are created around the same time, CA will consider
them independently and place no guards for the capacity.
Also the capacity is not reserved in any manner so it may be scaled-down.
#### atomic-scale-up.kubernetes.io class
The `atomic-scale-up.kubernetes.io` aims to provision the resources required for the
specified pods in an atomic way. The proposed logic is to:
1. Try to provision required VMs in one loop.
2. If it failed, remove the partially provisioned VMs and back-off.
3. Stop the back-off after a given duration (optional), which would be passed
via `AdditionalParameters` field, using `ValidUntilSeconds` key and would contain string
denoting duration for which we should retry (measured since creation fo the CR).
Note: that the VMs created in this mode are subject to the scale-down logic.
So the duration during which users need to create the Pods is equal to the
value of `--scale-down-unneeded-time` flag.
### Adding pods that consume given ProvisioningRequest
To avoid generating double scale-ups and exclude pods that are meant to consume
given capacity CA should be able to differentiate those from all other pods.
To do so users need to specify the following pod annotation (it is not required
in ProvReq’s template, though it can be specified):
```yaml
annotations:
"cluster-autoscaler.kubernetes.io/consume-provisioning-request": "provreq-name"
```
If it is provided for the pods that consume the ProvReq with `check-capacity.kubernetes.io` class,
the CA will not provision the capacity, even if it was needed (as some other pods might have been
scheduled on it) and will result in visibility events passed to the ProvReq and pods.
If it is not passed the CA will behave normally and just provision the capacity if it needed.
Note: CA will match all pods with this annotation to a corresponding ProvReq and
ignore them when executing a scale-up loop (so that is up to users to make sure
that the ProvReq count is matching the number of created pods).
If the ProvReq is missing, all of the pods that consume it will be unschedulable indefinitely.
### CRD lifecycle
1. A ProvReq will be created either by the end user or by a framework.
At this point needed PodTemplate objects should be also created.
2. CA will pick it up, choose a nodepool (or create a new one if NAP is
enabled), and try to create nodes.
3. If CA successfully creates capacity, ProvReq will receive information about
this fact in `Conditions` field.
4. At this moment, users can create pods in that will consume the ProvReq (in
the same namespace), those will be scheduled on the capacity that was
created by the CA.
5. Once all of the pods are scheduled users can delete the ProvReq object,
otherwise it will be garbage collected after some time.
6. When pods finish the work and nodes become unused the CA will scale them
down.
Note: Users can create a ProvReq and pods consuming them at the same time (in a
"fire and forget" manner), but this may result in the pods being unschedulable
and triggering user configured alerts.
### Canceling the requests
To cancel a pending Provisioning Request with atomic class, all that the users need to do is
to delete the Provisioning Request object.
After that the CA will no longer guard the nodes from deletion and proceed with standard scale-down logic.
### Conditions
The following Condition states should encode the states of the ProvReq:
- Provisioned - VMs were created successfully (Atomic class)
- CapacityAvailable - cluster contains enough capacity to schedule pods (Check
class)
* `CapacityAvailable=true` will denote that cluster contains enough capacity to schedule pods
* `CapacityAvailable=false` will denote that cluster does not contain enough capacity to schedule pods
- Failed - failed to create or check capacity (both classes)
The Reasons and Messages will contain more details about why the specific
condition was triggered.
Providers of the custom classes should reuse the conditions where available or create their own ones
if items from the above list cannot be used to denote a specific situation.
### CA implementation details
The proposed implementation is to handle each ProvReq in a separate scale-up
loop. This will require changes in multiple parts of CA:
1. Listing unschedulable pods where:
- pods that consume ProvReq need to filtered-out
- pods that are represented by the ProvReq need to be injected (we need to
ensure those are threated as one group by the sharding logic)
2. Scale-up logic, which as of now has no notion atomicity and grouping of
pods. This is simplified as the ScaleUp logic was recently put [behind an
interface](https://github.com/kubernetes/autoscaler/pull/5597).
- This is a place where the biggest part of the change will be made. Here
many parts of the logic are assuming best-effort semantics and the scale
up size is lowered in many situations:
- Estimation logic, which stops after some time-out or number of
pods/nodes.
- Size limiting, which caps the scale-up to match the size
restrictions (on node group or cluster level).
3. Node creation, which needs to support atomic resize. Either via native cloud
provider APIs or best effort with node removal if CA is unable to fulfill
the scale-up.
- This is also quite substantial change, we can provide a generic
best-effort implementation that will try to scale up and clean-up nodes
if it is unsuccessful, but it is up to cloud providers to integrate with
provider specific APIs.
4. Scale down path is not expected to change much. But users should follow
[best
practices](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node)
to avoid CA disturbing their workloads.
## Testing
The following e2e test scenarios will be created to check whether ProvReq
handling works as expected:
1. A new ProvReq with `check-capacity.kubernetes.io` provisioning class is created, CA
checks if there is enough capacity in cluster to provision specified pods.
2. A new ProvReq with `atomic-scale-up.kubernetes.io` provisioning class is created, CA
picks an appropriate node group scales it up atomically.
3. A new atomic ProvReq is created for which a NAP needs to provision a new
node group. NAP creates it CA scales it atomically.
- Here we should cover some of the different reasons why NAP may be
required.
4. An atomic ProvReq fails due to node group size limits and NAP CPU and/or RAM
limits.
5. Scalability tests.
- Scenario in which many small ProvReqs are created (strain on the number
of scale-up loops).
- Scenario in which big ProvReq is created (strain on a single scale-up
loop).
## Limitations
The current Cluster Autoscaler implementation is not taking into account [Resource Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/). \
The current proposal is to not include handling of the Resource Quotas, but it could be added later on.
## Future Expansions
### ProvisioningClass CRD
One of the expansion of this approach is to introduce the ProvisioningClass CRD,
which follows the same approach as
[StorageClass object](https://kubernetes.io/docs/concepts/storage/storage-classes/).
Such approach would allow administrators of the cluster to introduce a list of allowed
ProvisioningClasses. Such CRD can also contain a pre set configuration, i.e.
administrators may set that `atomic-scale-up.kubernetes.io` would retry up to `2h`.
Possible CRD definition:
```go
// ProvisioningClass is a way to express provisioning classes available in the cluster.
type ProvisioningClass struct {
// Name denotes the name of the object, which is to be used in the ProvisioningClass
// field in Provisioning Request CRD.
//
// +kubebuilder:validation:Required
Name string `json:"name"`
// AdditionalParameters contains all other parameters custom classes may require.
//
// +optional
AdditionalParameters map[string]string `json:"additionalParameters"`
}
```