Merge pull request #913 from nikhita/customresources-subresources
Automatic merge from submit-queue. Proposal: SubResources for CustomResources [CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs. However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources. cc @sttts @deads2k @enisoc @bgrant0607 @erictune @lavalamp @brendandburns @philips @liggitt @mbohlool @fabxc @adohe @munnerz
This commit is contained in:
Kubernetes Submit Queue
GitHub
commit
ca27d20ba1
|
|
@ -0,0 +1,203 @@
|
|||
# Subresources for CustomResources
|
||||
|
||||
Authors: @nikhita, @sttts
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Abstract](#abstract)
|
||||
2. [Goals](#goals)
|
||||
3. [Non-Goals](#non-goals)
|
||||
4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition)
|
||||
1. [API Types](#api-types)
|
||||
2. [Feature Gate](#feature-gate)
|
||||
5. [Semantics](#semantics)
|
||||
1. [Validation Behavior](#validation-behavior)
|
||||
1. [Status](#status)
|
||||
2. [Scale](#scale)
|
||||
2. [Status Behavior](#status-behavior)
|
||||
3. [Scale Behavior](#scale-behavior)
|
||||
1. [Status Replicas Behavior](#status-replicas-behavior)
|
||||
2. [Selector Behavior](#selector-behavior)
|
||||
4. [Implementation Plan](#implementation-plan)
|
||||
5. [Alternatives](#alternatives)
|
||||
1. [Scope](#scope)
|
||||
|
||||
## Abstract
|
||||
|
||||
[CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs.
|
||||
|
||||
However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Support status/spec split for CustomResources:
|
||||
1. Status changes are ignored on the main resource endpoint.
|
||||
2. Support a `/status` subresource HTTP path for status changes.
|
||||
3. `metadata.Generation` is increased only on spec changes.
|
||||
2. Support a `/scale` subresource for CustomResources.
|
||||
3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources.
|
||||
4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint.
|
||||
5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`.
|
||||
|
||||
## Proposed Extension of CustomResourceDefinition
|
||||
|
||||
### API Types
|
||||
|
||||
The addition of the following external types in `apiextensions.k8s.io/v1beta1` is proposed:
|
||||
|
||||
```go
|
||||
type CustomResourceDefinitionSpec struct {
|
||||
...
|
||||
// SubResources describes the subresources for CustomResources
|
||||
// This field is alpha-level and should only be sent to servers that enable
|
||||
// subresources via the CurstomResourceSubResources feature gate.
|
||||
// +optional
|
||||
SubResources *CustomResourceSubResources `json:"subResources,omitempty"`
|
||||
}
|
||||
|
||||
// CustomResourceSubResources defines the status and scale subresources for CustomResources.
|
||||
type CustomResourceSubResources struct {
|
||||
// Status denotes the status subresource for CustomResources
|
||||
Status *CustomResourceSubResourceStatus `json:"status,omitempty"`
|
||||
// Scale denotes the scale subresource for CustomResources
|
||||
Scale *CustomResourceSubResourceScale `json:"scale,omitempty"`
|
||||
}
|
||||
|
||||
// CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status.
|
||||
type CustomResourceSubResourceStatus struct {
|
||||
}
|
||||
|
||||
// CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale.
|
||||
type CustomResourceSubResourceScale struct {
|
||||
// required, e.g. “.spec.replicas”. Must be under `.spec`.
|
||||
// Only JSON paths without the array notation are allowed.
|
||||
SpecReplicasPath string `json:"specReplicasPath"`
|
||||
// optional, e.g. “.status.replicas”. Must be under `.status`.
|
||||
// Only JSON paths without the array notation are allowed.
|
||||
StatusReplicasPath string `json:"statusReplicasPath,omitempty"`
|
||||
// optional, e.g. “.spec.labelSelector”. Must be under `.spec`.
|
||||
// Only JSON paths without the array notation are allowed.
|
||||
LabelSelectorPath string `json:"labelSelectorPath,omitempty"`
|
||||
// ScaleGroupVersion denotes the GroupVersion of the Scale
|
||||
// object sent as the payload for /scale. It allows transition
|
||||
// to future versions easily.
|
||||
// Today only autoscaling/v1 is allowed.
|
||||
ScaleGroupVersion schema.GroupVersion `json:"groupVersion"`
|
||||
}
|
||||
```
|
||||
|
||||
### Feature Gate
|
||||
|
||||
The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate.
|
||||
If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is dropped on creation and updates of CRDs.
|
||||
|
||||
### Scale type
|
||||
|
||||
The `Scale` object is the payload sent over the wire for `/scale`. The [polymorphic `Scale` type](https://github.com/kubernetes/kubernetes/pull/53743) i.e. `autoscaling/v1.Scale` is used for the `Scale` object.
|
||||
|
||||
Since the GroupVersion of the `Scale` object is specified in `CustomResourceSubResourceScale`, transition to future versions (eg `autoscaling/v2.Scale`) can be done easily.
|
||||
|
||||
Note: If `autoscaling/v1.Scale` is deprecated, then it would be deprecated here as well.
|
||||
|
||||
## Semantics
|
||||
|
||||
### Validation Behavior
|
||||
|
||||
#### Status
|
||||
|
||||
The status endpoint of a CustomResource receives a full CR object. Changes outside of the `.status` subpath are ignored.
|
||||
For validation, the JSON Schema present in the CRD is validated only against the `.status` subpath.
|
||||
|
||||
To validate only against the schema for the `.status` subpath, `oneOf` and `anyOf` constructs are not allowed within the root of the schema, but only under a properties sub-schema (with this restriction, we can project a schema to a sub-path). The following is forbidden in the CRD spec:
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
openAPIV3Schema:
|
||||
oneOf:
|
||||
...
|
||||
```
|
||||
|
||||
**Note**: The restriction for `oneOf` and `anyOf` allows us to write a projection function `ProjectJSONSchema(schema *JSONSchemaProps, path []string) (*JSONSchemaProps, error)` that can be used to apply a given schema for the whole object to only the sub-path `.status` or `.spec`.
|
||||
|
||||
#### Scale
|
||||
|
||||
Moreover, if the scale subresource is enabled:
|
||||
|
||||
On update, we copy the values from the `Scale` object into the specified paths in the CustomResource, if the path is set (`StatusReplicasPath` and `LabelSelectorPath` are optional).
|
||||
If `StatusReplicasPath` or `LabelSelectorPath` is not set, we validate that the value in `Scale` is also not specified and return an error otherwise.
|
||||
|
||||
On `get` and on `update` (after copying the values into the CustomResource as described above), we verify that:
|
||||
|
||||
- The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is a non-negative integer value and is not empty.
|
||||
|
||||
- The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is an integer value if it exists (i.e. this can be empty).
|
||||
|
||||
- The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is a valid label selector if it exists (i.e. this can be empty).
|
||||
|
||||
**Note**: The values at the JSON Paths specified by `SpecReplicasPath`, `LabelSelectorPath` and `StatusReplicasPath` are also validated with the same rules when the whole object or, in case the `/status` subresource is enabled, the `.status` sub-object is updated.
|
||||
|
||||
### Status Behavior
|
||||
|
||||
If the `/status` subresource is enabled, the following behaviors change:
|
||||
|
||||
- The main resource endpoint will ignore all changes in the status subpath.
|
||||
(note: it will **not** reject requests which try to change the status, following the existing semantics of other resources).
|
||||
|
||||
- The `.metadata.generation` field is updated if and only if the value at the `.spec` subpath changes.
|
||||
Additionally, if the spec does not change, `.metadata.generation` is not updated.
|
||||
|
||||
- The `/status` subresource receives a full resource object, but only considers the value at the `.status` subpath for the update.
|
||||
The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539.
|
||||
|
||||
Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer.
|
||||
|
||||
The `/status` endpoint supports both `get` and `update` verbs.
|
||||
|
||||
### Scale Behavior
|
||||
|
||||
The number of CustomResources can be easily scaled up or down depending on the replicas field present in the `.spec` subpath.
|
||||
|
||||
Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified:
|
||||
|
||||
1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`.
|
||||
|
||||
2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`.
|
||||
In other words, the scale and the CustomResource share the resource version used for optimistic concurrency.
|
||||
Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource.
|
||||
|
||||
The `/scale` endpoint supports both `get` and `update` verbs.
|
||||
|
||||
#### Status Replicas Behavior
|
||||
|
||||
As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only.
|
||||
|
||||
#### Selector Behavior
|
||||
|
||||
`CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count.
|
||||
The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty.
|
||||
Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much):
|
||||
|
||||
1. `/status` subresource
|
||||
2. `/scale` subresource
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Scope
|
||||
|
||||
In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`.
|
||||
We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`.
|
||||
Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions.
|
||||
|
||||
**Note**: The types do not make the addition of other subresources impossible in the future.
|
||||
|
||||
We also restrict the JSON path for the status and the spec within the CustomResource.
|
||||
We could make them definable by the user and the proposed types actually allow us to open this up in the future.
|
||||
For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well.
|
||||
Loading…
Reference in New Issue