karmada-io
/
karmada
mirror of https://github.com/karmada-io/karmada.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #610 from RainbowMango/pr_runtime_value_retention 

Proposal: Configurable local value retention
This commit is contained in:
karmada-bot 2021-08-20 09:12:34 +08:00 committed by GitHub
parent f8873c840f 3640948239
commit c37bedc1cf
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 305 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

155
docs/proposals/configurable-local-value-retention/README.md Normal file
View File

@ -0,0 +1,155 @@
---
title: Configurable Local Value Retention
authors:
- "@RainbowMango"
reviewers:
- "@TBD"
approvers:
- "@TBD"
creation-date: 2021-08-12
---
# Configurable Local Value Retention
## Summary
For now, Karmada keeps watching the propagated resources in member clusters to ensure the resource is always in desire
state.
In many cases, the controllers running in member clusters will make changes to the resource, such as:
- The Kubernetes will assign a `clusterIP` for `Service` with type `ClusterIP`.
- The Kubernetes will assign a `nodeName` for `Pod` in the scheduling phase.
When Karmada users make changes to the resource template, Karmada will update the resource against the member cluster,
but before the update, Karmada should `retain` the changes made by member cluster controllers.
Karmada has implemented different `retain` methods for common resources, such as `Service`, `Pod`, `ServiceAccount`, and so on.
The `retain` feature works for most cases, but still has disadvantages:
- The `retain` methods are built-in and users can't customize them.
- No `retain` method for custom resources(declared by `CustomResourceDefinition`).
This proposal aims to provide a strategy to customize the `retain` methods for any kind of resource.
## Motivation
Nowadays, Kubernetes has provided dozens of resources, even though we don't need to implement the `retain` method for
each of them, but it is still a heavy burden to maintain, it's hard to meet different kinds of expectations.
On the other hand, we can't define the `retain` method for custom resources as you even don't know the fields in it.
### Goals
- Provide a strategy to support customize `retain` methods for any kind of resources.
* Support overrides built-in `retain` methods of Kubernetes resources.
* Support customize `retain` method for custom resources(declared by CustomResourceDefinition).
- Provide a general mechanism to customize karmada controller behaviors.
* The mechanism can be reused by other customized requirements.
### Non-Goals
- Define specific `retain` methods for Kubernetes resources.
- Deprecate the built-in `retain` methods.
* We should maintain built-in `retain` for the well-known resources to simplify user configuration.
## Proposal
### User Stories
#### As a user, I want to customize the built-in retain method because it can't fulfill my requirement.
The built-in retain methods aim to provide a default retain method for well-known resources, they are suitable for
most situations, but can't guarantee to meet all user's needs.
On the other hand, we usually maintain built-in retain methods for a preferred version of the resource, such as `Deployment`,
The `apps/v1` retain method might not suitable for `apps/v1beta1`.
#### As a user, I want to customize the retain method for my CRD resources.
Nowadays, it's getting pretty common that people extend Kubernetes by CRD, at the meanwhile people might implement
their controllers and the controllers probably make changes to these CRs when reconciling the changes.
In this case, people should define the retain method for the CR, otherwise, it will be a conflict when syncing changes
make on the Karmada control plane.(Karmada update--> controller update them back --> Karmada update ... endless update loop)
### Notes/Constraints/Caveats (Optional)
### Risks and Mitigations
## Design Details
### New Config API
We propose a new CR in `config.karmada.io` group.
```golang
// Config represents the configuration of Karmada.
type Config struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
// Spec represents the specification of the desired behavior of Karmada configuration.
// +required
Spec ConfigSpec `json:"spec"`
}
type ConfigSpec struct {
// Retentions represents a group of customized retention methods.
// +optional
Retentions []LocalValueRetention `json:"retentions,omitempty"`
}
// LocalValueRetention represents a customized retention method for specific API resource.
type LocalValueRetention struct {
// APIVersion represents the API version of the target resources.
// +required
APIVersion string `json:"apiVersion"`
// Kind represents the Kind of the target resources.
// +required
Kind string `json:"kind"`
// Fields indicates the fields that should be retained.
// Each field describes a field in JsonPath format.
// +optional
Fields []string `json:"fields,omitempty"`
// RetentionLua holds the Lua script that is used to retain runtime values to the desired specification.
// The script should implement a function just like:
//
// function Retain(desiredObj, runtimeObj)
// desiredObj.spec.fieldFoo = runtimeObj.spec.fieldFoo
// return desiredObj
// end
//
// RetentionLua only holds the function body part, take the `desiredObj` and `runtimeObj` as the function parameters or
// global parameters, and finally retains a retained specification.
// +optional
RetentionLua string `json:"retentionLua,omitempty"`
}
```
Each `LocalValueRetention` customizes one API resource referencing by `APIVersion` and `Kind`.
The `Fields` holds a collection of fields that should be retained.
The `RetentionLua` holds a fragment of [Lua](https://www.lua.org/) script which is used to implement the retention logic.
**Note:** The `Lua` might not be the best choice, another considered approach is [cue](https://github.com/cue-lang/cue/tree/master/doc/tutorial/kubernetes) as mentioned by @pigletfly at [this discussion](https://github.com/karmada-io/karmada/pull/592#issuecomment-895759570),
we have to do some investigation. But we also reserve the possibility that uses more than one script, people can select by themselves.
### Bundle well-known custom resources
There are a lot of famous projects that defined CRD in the Kubernetes ecosystem, for these widely adopted
CRDs, we can bundle them into Karmada, just like the `built-in` retain methods.
This part of the design will continue later.
### Test Plan
- Propose E2E test cases according to user stories above:
* Test we can customize the built-in method
* Test we can customize custom resource
- Propose a tool that people can test the script.
## Alternatives

150
docs/proposals/runtime-value-retention/README.md Normal file
View File

@ -0,0 +1,150 @@
---
title: Runtime value retention
authors:
- "@RainbowMango"
reviewers:
- "@TBD"
approvers:
- "@TBD"
creation-date: 2021-08-12
---
# Runtime value retention
## Summary
For now, Karmada keeps watching the propagated resources in member clusters to ensure the resource is always in desire
state.
In many cases, the controllers running in member clusters will make changes to the resource, such as:
- The Kubernetes will assign a `clusterIP` for `Service` with type `ClusterIP`.
- The Kubernetes will assign a `nodeName` for `Pod` in the scheduling phase.
When Karmada users make changes to the resource template, Karmada will update the resource against the member cluster,
but before the update, Karmada should `retain` the changes made by member cluster controllers.
Karmada has implemented different `retain` methods for common resources, such as `Service`, `Pod`, `ServiceAccount`, and so on.
The `retain` feature works for most cases, but still has disadvantages:
- The `retain` methods are built-in and users can't customize them.
- No `retain` method for custom resources(declared by `CustomResourceDefinition`).
This proposal aims to provide a strategy to customize the `retain` methods for any kind of resource.
## Motivation
Nowadays, Kubernetes has provided dozens of resources, even though we don't need to implement the `retain` method for
each of them, but it is still a heavy burden to maintain, it's hard to meet different kinds of expectations.
On the other hand, we can't define the `retain` method for custom resources as you even don't know the fields in it.
### Goals
- Provide a strategy to support customize `retain` methods for any kind of resources.
* Support customize built-in `retain` methods of Kubernetes resources.
* Support customize `retain` method for custom resources(declared by CustomResourceDefinition).
- Provide a general mechanism to customize karmada controller behaviors.
* The mechanism can be reused by other customized requirements.
### Non-Goals
- Define specific `retain` methods for Kubernetes resources.
- Deprecate the built-in `retain` methods.
* We should maintain built-in `retain` for the well-known resources to simplify user configuration.
## Proposal
### User Stories
#### As a user, I want to customize the built-in retain method because it can't fulfill my requirement.
The built-in retain methods aim to provide a default retain method for well-known resources, they are suitable for
most situations, but can't guarantee to meet all user's needs.
On the other hand, we usually maintain built-in retain methods for a preferred version of the resource, such as `Deployment`,
The `apps/v1` retain method might not suitable for `apps/v1beta1`.
#### As a user, I want to customize the retain method for my CRD resources.
Nowadays, it's getting pretty common that people extend Kubernetes by CRD, at the meanwhile people might implement
their controllers and the controllers probably make changes to these CRs when reconciling the changes.
In this case, people should define the retain method for the CR, otherwise, it will be a conflict when syncing changes
make on the Karmada control plane.(Karmada update--> controller update them back --> Karmada update ... endless update loop)
### Notes/Constraints/Caveats (Optional)
### Risks and Mitigations
## Design Details
### Retention configuration
We propose a new configuration for customize the retention rules:
```golang
// RuntimeValueRetention represents a customized retention method for specific API resource.
type RuntimeValueRetention struct {
// APIVersion represents the API version of the target resources.
// +required
APIVersion string `json:"apiVersion"`
// Kind represents the Kind of the target resources.
// +required
Kind string `json:"kind"`
// RetentionLua holds the Lua script that is used to retain runtime values to the desired specification.
// The script should implement a function just like:
//
// function Retain(desiredObj, runtimeObj)
// desiredObj.spec.fieldFoo = runtimeObj.spec.fieldFoo
// return desiredObj
// end
//
// RetentionLua only holds the function body part, take the `desiredObj` and `runtimeObj` as the function parameters or
// global parameters, and finally retains a retained specification.
// +optional
RetentionLua string `json:"health.lua,omitempty"`
}
```
Each `RuntimeValueRetention` customizes one API resource referencing by `APIVersion` and `Kind`.
The `RetentionLua` holds a fragment of [Lua](https://www.lua.org/) script which is used to implement the retention logic.
**Note:** The `Lua` might not be the best choice, another considered approach is [cue](https://github.com/cue-lang/cue/tree/master/doc/tutorial/kubernetes) as mentioned by @pigletfly at [this discussion](https://github.com/karmada-io/karmada/pull/592#issuecomment-895759570),
we have to do some investigation. But we also reserve the possibility that uses more than one script, people can select by themselves.
### Using ConfigMap to deploy the configuration
We propose to use `ConfigMap` to hold the configuration, and reserve a dedicated name `karmada-config`. The slices of
`RuntimeValueRetention` will be resident in the field `retention.customizations`, such as:
```yaml
data:
retention.customizations: |
<one or more RuntimeValueRetention>
```
The `karmada-controller-manager` will detect the `karmada-config` at the start phase. That means people have to deploy
`karmada-config` first before starting the `karmada-controller-manager`, otherwise `karmada-controller-manager` should be
restarted to load the `karmada-config`.
**Note:** Technically speaking, we can discover the `karmada-config` in real-time.
### Bundle well-known custom resources
There are a lot of famous projects that defined CRD in the Kubernetes ecosystem, for these widely adopted
CRDs, we can bundle them into Karmada, just like the `built-in` retain methods.
This part of the design will continue later.
### Test Plan
- Propose E2E test cases according to user stories above:
* Test we can customize the built-in method
* Test we can customize custom resource
- Propose a tool that people can test the script.
## Alternatives
### Define customization as an API resource
Instead of using a `ConfigMap`, we could also use an API resource. In this case, we should define a resource type for the configuration. But will be a bit heavy to maintain an API.