Add multiple scheduling group proposal
Signed-off-by: RainbowMango <qdurenhongcai@gmail.com>
This commit is contained in:
RainbowMango
parent
4641e88fd2
commit
ac662db45a
|
|
@ -0,0 +1,371 @@
|
||||||
|
---
|
||||||
|
title: Multiple scheduling group
|
||||||
|
|
||||||
|
authors:
|
||||||
|
- "@RainbowMango"
|
||||||
|
|
||||||
|
reviewers:
|
||||||
|
- "@chaunceyjiang"
|
||||||
|
- "@Garrybest"
|
||||||
|
- "@lonelyCZ"
|
||||||
|
- "@Poor12"
|
||||||
|
- "@XiShanYongYe-Chang"
|
||||||
|
|
||||||
|
approvers:
|
||||||
|
- "@kevin-wangzefeng"
|
||||||
|
|
||||||
|
creation-date: 2023-02-06
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# Multiple scheduling group
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
The current PropagationPolicy supports declaring `only one` group of clusters, that
|
||||||
|
is the `.spec.placement.clusterAffinity`, e.g.
|
||||||
|
```yaml
|
||||||
|
apiVersion: policy.karmada.io/v1alpha1
|
||||||
|
kind: PropagationPolicy
|
||||||
|
metadata:
|
||||||
|
name: foo
|
||||||
|
spec:
|
||||||
|
resourceSelectors:
|
||||||
|
- apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: foo
|
||||||
|
placement:
|
||||||
|
clusterAffinity:
|
||||||
|
clusterNames:
|
||||||
|
- member1
|
||||||
|
- member2
|
||||||
|
```
|
||||||
|
The `clusterAffinity` supplies a group of candidate clusters to the `karmada-scheduler`.
|
||||||
|
The `karmada-scheduler` makes schedule decisions among the candidate clusters
|
||||||
|
according to relevant restrictions(like spreadConstraint, filter plugins, and so on),
|
||||||
|
and the scheduling results are either successful or failure.
|
||||||
|
- successful: successfully selected a group of clusters(probably a subset of candidates) for the referencing resource.
|
||||||
|
- failure: failed to select a group of clusters that satisfied the restrictions.
|
||||||
|
|
||||||
|
This proposal proposes a strategy that multiple `clusterAffinity` could be declared,
|
||||||
|
and the `karmada-scheduler` could make the decision by evaluating each `clusterAffinity`
|
||||||
|
in a specified order.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
The cluster administrators usually classify their clusters by categories(like,
|
||||||
|
by provider, usage, and so on) into different groups, and hoping to deploy
|
||||||
|
workload to the preferred groups and leave a backup group in case of preferred
|
||||||
|
group doesn't satisfy scheduling restrictions like lack of resources.
|
||||||
|
|
||||||
|
The Karmada community received a lot of feedback about that from end users, like
|
||||||
|
issue [#780](https://github.com/karmada-io/karmada/issues/780) and
|
||||||
|
[#2085](https://github.com/karmada-io/karmada/issues/2085).
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
|
||||||
|
- Extend the API of PropagationPolicy to hold multiple affinity groups declaration.
|
||||||
|
- Extend the API of ResourceBinding to annotate the affinity group that the scheduler is inspecting.
|
||||||
|
- Propose the implementation ideas for involved components, including `karmada-controller-manager`, `karmada-webhook` and `karmada-scheduler`.
|
||||||
|
|
||||||
|
### Non-Goals
|
||||||
|
|
||||||
|
- **The relative priority of clusters in the same group**
|
||||||
|
|
||||||
|
This proposal focuses on introducing multiple affinity groups, for the relative
|
||||||
|
priority of clusters in the same group topic is out of scope. And such requirements
|
||||||
|
could be done by `weightPreference`(.spec.placement.replicaScheduling.weightPreference) or
|
||||||
|
define a strategy for `karmada-scheduler` to score specified clusters.
|
||||||
|
|
||||||
|
- **Scheduling re-balance**
|
||||||
|
|
||||||
|
Indeed, oftentimes people want the `karmada-scheduler` to perform an extra schedule
|
||||||
|
for different purposes, like the question addressed by the discussion
|
||||||
|
[#3069](https://github.com/karmada-io/karmada/discussions/3069), these kind of
|
||||||
|
issues should be tracked by another proposal.
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
### User Stories (Optional)
|
||||||
|
|
||||||
|
#### As a user, I prefer to deploy my applications on clusters in local data center to save costs.
|
||||||
|
|
||||||
|
I have some clusters in my data center as well as some managed clusters from cloud
|
||||||
|
providers(like AWS and Google Cloud). Since the cost of the managed clusters is higher
|
||||||
|
than private clusters, so I prefer to deploy applications on private clusters and
|
||||||
|
take the managed clusters as a backup.
|
||||||
|
|
||||||
|
#### As a user, I want to deploy applications on the primary cluster and leave a backup cluster for disaster recovery cases.
|
||||||
|
|
||||||
|
I have two clusters, the primary, and the backup cluster, I want Karmada to deploy
|
||||||
|
my applications on the primary cluster and migrate applications on the backup cluster
|
||||||
|
when the primary cluster becomes unavailable due to like data center power off or
|
||||||
|
maintenance task.
|
||||||
|
|
||||||
|
### Notes/Constraints/Caveats (Optional)
|
||||||
|
|
||||||
|
This proposal mainly focuses on the changes in `PropagationPolicy` and `ResourceBinding`,
|
||||||
|
but it will also be applied to `ClusterPropagationPolicy` and `ClusterResourceBinding`.
|
||||||
|
|
||||||
|
### Risks and Mitigations
|
||||||
|
|
||||||
|
This proposal maintains the backward compatibility, the system built with previous
|
||||||
|
versions of Karmada can be seamlessly migrated to the new version.
|
||||||
|
The previous configurations(yamls) could be applied to the new version of Karmada and
|
||||||
|
without any behavior change.
|
||||||
|
|
||||||
|
## Design Details
|
||||||
|
|
||||||
|
### API change
|
||||||
|
|
||||||
|
#### PropagationPolicy API change
|
||||||
|
|
||||||
|
This proposal proposes a new field `ClusterAffinities` for declaring
|
||||||
|
multiple affinity terms in `.spec.placement` of `PropagationPolicy`.
|
||||||
|
```go
|
||||||
|
// Placement represents the rule for select clusters.
|
||||||
|
type Placement struct {
|
||||||
|
// ClusterAffinity represents scheduling restrictions to a certain set of clusters.
|
||||||
|
// If not set, any cluster can be scheduling candidate.
|
||||||
|
// +optional
|
||||||
|
ClusterAffinity *ClusterAffinity `json:"clusterAffinity,omitempty"`
|
||||||
|
|
||||||
|
// ClusterAffinities represents scheduling restrictions to multiple cluster
|
||||||
|
// groups that indicated by ClusterAffinityTerm.
|
||||||
|
//
|
||||||
|
// The scheduler will evaluate these groups one by one in the order they
|
||||||
|
// appear in the spec, the group that does not satisfy scheduling restrictions
|
||||||
|
// will be ignored which means all clusters in this group will not be selected
|
||||||
|
// unless it also belongs to the next group(a cluster could belong to multiple
|
||||||
|
// groups).
|
||||||
|
//
|
||||||
|
// If none of the groups satisfy the scheduling restrictions, then scheduling
|
||||||
|
// fails, which means no cluster will be selected.
|
||||||
|
//
|
||||||
|
// Note:
|
||||||
|
// 1. ClusterAffinities can not co-exist with ClusterAffinity.
|
||||||
|
// 2. If both ClusterAffinity and ClusterAffinities are not set, any cluster
|
||||||
|
// can be scheduling candidates.
|
||||||
|
//
|
||||||
|
// Potential use case 1:
|
||||||
|
// The private clusters in the local data center could be the main group, and
|
||||||
|
// the managed clusters provided by cluster providers could be the secondary
|
||||||
|
// group. So that the Karmada scheduler would prefer to schedule workloads
|
||||||
|
// to the main group and the second group will only be considered in case of
|
||||||
|
// the main group does not satisfy restrictions(like, lack of resources).
|
||||||
|
//
|
||||||
|
// Potential use case 2:
|
||||||
|
// For the disaster recovery scenario, the clusters could be organized to
|
||||||
|
// primary and backup groups, the workloads would be scheduled to primary
|
||||||
|
// clusters firstly, and when primary cluster fails(like data center power off),
|
||||||
|
// Karmada scheduler could migrate workloads to the backup clusters.
|
||||||
|
//
|
||||||
|
// +optional
|
||||||
|
ClusterAffinities []ClusterAffinityTerm `json:"clusterAffinities,omitempty"`
|
||||||
|
|
||||||
|
// ClusterTolerations represents the tolerations.
|
||||||
|
// +optional
|
||||||
|
ClusterTolerations []corev1.Toleration `json:"clusterTolerations,omitempty"`
|
||||||
|
|
||||||
|
// SpreadConstraints represents a list of the scheduling constraints.
|
||||||
|
// +optional
|
||||||
|
SpreadConstraints []SpreadConstraint `json:"spreadConstraints,omitempty"`
|
||||||
|
|
||||||
|
// ReplicaScheduling represents the scheduling policy on dealing with the number of replicas
|
||||||
|
// when propagating resources that have replicas in spec (e.g. deployments, statefulsets) to member clusters.
|
||||||
|
// +optional
|
||||||
|
ReplicaScheduling *ReplicaSchedulingStrategy `json:"replicaScheduling,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ClusterAffinityTerm selects a set of cluster.
|
||||||
|
type ClusterAffinityTerm struct {
|
||||||
|
// AffinityName is the name of the cluster group.
|
||||||
|
// +required
|
||||||
|
AffinityName string `json:"affinityName"`
|
||||||
|
|
||||||
|
ClusterAffinity `json:",inline"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Each affinity term essentially is the named `ClusterAffinity`. During the scheduling
|
||||||
|
phase, the scheduler evaluates terms in the order they appear in the spec one by
|
||||||
|
one, and turns to the next term if the term does not satisfy restrictions.
|
||||||
|
|
||||||
|
The following configuration sample declares 3 affinity terms:
|
||||||
|
```yaml
|
||||||
|
apiVersion: policy.karmada.io/v1alpha1
|
||||||
|
kind: PropagationPolicy
|
||||||
|
metadata:
|
||||||
|
name: nginx
|
||||||
|
spec:
|
||||||
|
resourceSelectors:
|
||||||
|
- apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: nginx
|
||||||
|
placement:
|
||||||
|
clusterAffinities:
|
||||||
|
- affinityName: dc-shanghai
|
||||||
|
clusterNames:
|
||||||
|
- unavailable
|
||||||
|
- affinityName: dc-beijing
|
||||||
|
clusterNames:
|
||||||
|
- member1
|
||||||
|
- affinityName: dc-hongkong
|
||||||
|
clusterNames:
|
||||||
|
- member2
|
||||||
|
```
|
||||||
|
|
||||||
|
During the scheduling phase, the scheduler will first look at the affinity
|
||||||
|
named `dc-shanghai` and try to select a feasible cluster set from it.
|
||||||
|
|
||||||
|
If no feasible cluster is found from the term, the scheduler will try against the
|
||||||
|
next term which is the one named `dc-beijing`. And once the scheduler successfully
|
||||||
|
selected a feasible cluster set from this affinity term, the scheduler will not
|
||||||
|
continue to look at the following terms.
|
||||||
|
|
||||||
|
And, in case of cluster `member1` becomes unavailable, by leveraging the
|
||||||
|
[Failover feature](https://karmada.io/docs/userguide/failover/failover-overview)
|
||||||
|
the scheduler will start looking for alternatives `in current affinity term`, and
|
||||||
|
if that fails, it will look at the term named `dc-hongkong`.
|
||||||
|
|
||||||
|
**Note:** Each affinity term is completely independent, and the scheduler will
|
||||||
|
only select one term at a time during scheduling. But it is allowed for the same
|
||||||
|
cluster present in multiple affinity terms.
|
||||||
|
|
||||||
|
#### ResourceBinding API change
|
||||||
|
|
||||||
|
In order to track the affinity term that the scheduler last evaluated,
|
||||||
|
the proposal proposes a new field named `SchedulerObservedAffinityName` to
|
||||||
|
`.status` of `ResourceBinding`, so that the scheduler can continue the previous
|
||||||
|
scheduling cycle in any cases of re-schedule.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// ResourceBindingStatus represents the overall status of the strategy as well as the referenced resources.
|
||||||
|
type ResourceBindingStatus struct {
|
||||||
|
// SchedulerObservedGeneration is the generation(.metadata.generation) observed by the scheduler.
|
||||||
|
// If SchedulerObservedGeneration is less than the generation in metadata means the scheduler hasn't confirmed
|
||||||
|
// the scheduling result or hasn't done the schedule yet.
|
||||||
|
// +optional
|
||||||
|
SchedulerObservedGeneration int64 `json:"schedulerObservedGeneration,omitempty"`
|
||||||
|
|
||||||
|
// SchedulerObservedAffinityName is the affinity terms that
|
||||||
|
// scheduler looking at.
|
||||||
|
// +optional
|
||||||
|
SchedulerObservedAffinityName string `json:"schedulerObservingAffinityName,omitempty"`
|
||||||
|
|
||||||
|
// Conditions contain the different condition statuses.
|
||||||
|
// +optional
|
||||||
|
Conditions []metav1.Condition `json:"conditions,omitempty"`
|
||||||
|
// AggregatedStatus represents status list of the resource running in each member cluster.
|
||||||
|
// +optional
|
||||||
|
AggregatedStatus []AggregatedStatusItem `json:"aggregatedStatus,omitempty"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
E.g:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
status:
|
||||||
|
aggregatedStatus:
|
||||||
|
- applied: true
|
||||||
|
clusterName: member1
|
||||||
|
health: Healthy
|
||||||
|
status:
|
||||||
|
availableReplicas: 2
|
||||||
|
readyReplicas: 2
|
||||||
|
replicas: 2
|
||||||
|
updatedReplicas: 2
|
||||||
|
conditions:
|
||||||
|
- lastTransitionTime: "2023-02-04T09:38:20Z"
|
||||||
|
message: All works have been successfully applied
|
||||||
|
reason: FullyAppliedSuccess
|
||||||
|
status: "True"
|
||||||
|
type: FullyApplied
|
||||||
|
- lastTransitionTime: "2023-02-04T09:38:20Z"
|
||||||
|
message: Binding has been scheduled
|
||||||
|
reason: BindingScheduled
|
||||||
|
status: "True"
|
||||||
|
type: Scheduled
|
||||||
|
schedulerObservedGeneration: 2
|
||||||
|
SchedulerObservedAffinityName: ds-hongkong
|
||||||
|
```
|
||||||
|
|
||||||
|
The `SchedulerObservedAffinityName: ds-hongkong` means the scheduling result is based on
|
||||||
|
the affinity term named `ds-hongkong`, in case of re-scheduling, the scheduler should
|
||||||
|
continue to evaluate from this affinity term.
|
||||||
|
|
||||||
|
### Components change
|
||||||
|
|
||||||
|
|
||||||
|
#### karmada-controller-manager
|
||||||
|
|
||||||
|
When creating or updating `ResourceBidning`/`ClusterResourceBinding`, the added
|
||||||
|
`OrderedClusterAffinities` in `PropagationPolicy`/`ClusterPropagationPolicy` should
|
||||||
|
be synced.
|
||||||
|
|
||||||
|
#### karmada-scheduler
|
||||||
|
|
||||||
|
Currently, the karmada-scheduler only runs a single loop by accepting an affinity
|
||||||
|
term, that is [ScheduleAlgorithm interface](https://github.com/karmada-io/karmada/blob/32b8f21b79017e2f4154fbe0677cb63fb18b120c/pkg/scheduler/core/generic_scheduler.go#L22).
|
||||||
|
|
||||||
|
With this proposal, the `ScheduleAlgorithm interface` probably be invoked multi
|
||||||
|
time, and feed a different affinity term each time until the schedule succeeds.
|
||||||
|
|
||||||
|
#### karmada-webhook
|
||||||
|
|
||||||
|
Since it doesn't make sense for the newly introduced `ClusterAffinities`
|
||||||
|
co-exist with the previous `ClusterAffinity`, so the webhook should perform extra
|
||||||
|
validation work to prevent misleading configuration.
|
||||||
|
|
||||||
|
In addition, the `affinityName` of each affinity terms should be unique among all terms.
|
||||||
|
|
||||||
|
### Test Plan
|
||||||
|
|
||||||
|
- All current testing should be passed, no break change would be involved by this feature.
|
||||||
|
- Add new E2E tests to cover the feature, the scope should include:
|
||||||
|
* Workload propagating with scheduling type `Duplicated`.
|
||||||
|
* Workload propagating with scheduling type `Divided`
|
||||||
|
* Failover scenario
|
||||||
|
|
||||||
|
## Alternatives
|
||||||
|
|
||||||
|
Introducing a new field to specify cluster priority was one of the first suggested
|
||||||
|
ideas.(tracked by [#842](https://github.com/karmada-io/karmada/pull/842)).
|
||||||
|
This approach tried to reuse the terms in Kubernetes Pod affinity.
|
||||||
|
An example of this approach is as follows:
|
||||||
|
```yaml
|
||||||
|
apiVersion: policy.karmada.io/v1alpha1
|
||||||
|
kind: PropagationPolicy
|
||||||
|
metadata:
|
||||||
|
name: foo
|
||||||
|
spec:
|
||||||
|
resourceSelectors:
|
||||||
|
- apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: foo
|
||||||
|
placement:
|
||||||
|
clusterAffinity:
|
||||||
|
clusterNames:
|
||||||
|
- member1
|
||||||
|
- member2
|
||||||
|
propagatePriority:
|
||||||
|
- weight: 30
|
||||||
|
preference:
|
||||||
|
matchExpressions:
|
||||||
|
- key: topology
|
||||||
|
operator: In
|
||||||
|
values:
|
||||||
|
- us
|
||||||
|
- weight: 20
|
||||||
|
preference:
|
||||||
|
matchExpressions:
|
||||||
|
- key: topology
|
||||||
|
operator: In
|
||||||
|
values:
|
||||||
|
- cn
|
||||||
|
```
|
||||||
|
The `.spec.placement.clusterAffinity.propagatePriority` is used to specify the
|
||||||
|
cluster preference by reusing the [PreferredSchedulingTerm of Kubernetes](https://github.com/kubernetes/kubernetes/blob/fc002b2f07250a462bb1b471807708b542472c18/staging/src/k8s.io/api/core/v1/types.go#L3028-L3035).
|
||||||
|
|
||||||
|
However, the `PreferredSchedulingTerm` relies on group cluster by the label selector
|
||||||
|
and field selector, the `matchExpressions` involves too many nested layers that
|
||||||
|
probably make the configuration hard to maintain.
|
||||||
Loading…
Reference in New Issue