--- title: Extend the Kubernetes API with CustomResourceDefinitions reviewers: - deads2k - jpbetz - liggitt - roycaihw - sttts content_type: task min-kubernetes-server-version: 1.16 weight: 20 --- This page shows how to install a [custom resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/) into the Kubernetes API by creating a [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1-apiextensions-k8s-io). ## {{% heading "prerequisites" %}} {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} If you are using an older version of Kubernetes that is still supported, switch to the documentation for that version to see advice that is relevant for your cluster. ## Create a CustomResourceDefinition When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server creates a new RESTful resource path for each version you specify. The CRD can be either namespaced or cluster-scoped, as specified in the CRD's `scope` field. As with existing built-in objects, deleting a namespace deletes all custom objects in that namespace. CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces. For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`: ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: # name must match the spec fields below, and be in the form: . name: crontabs.stable.example.com spec: # group name to use for REST API: /apis// group: stable.example.com # list of versions supported by this CustomResourceDefinition versions: - name: v1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer # either Namespaced or Cluster scope: Namespaced names: # plural name to be used in the URL: /apis/// plural: crontabs # singular name to be used as an alias on the CLI and for display singular: crontab # kind is normally the CamelCased singular type. Your resource manifests use this. kind: CronTab # shortNames allow shorter string to match your resource on the CLI shortNames: - ct ``` and create it: ```shell kubectl apply -f resourcedefinition.yaml ``` Then a new namespaced RESTful API endpoint is created at: ``` /apis/stable.example.com/v1/namespaces/*/crontabs/... ``` This endpoint URL can then be used to create and manage custom objects. The `kind` of these objects will be `CronTab` from the spec of the CustomResourceDefinition object you created above. It might take a few seconds for the endpoint to be created. You can watch the `Established` condition of your CustomResourceDefinition to be true or watch the discovery information of the API server for your resource to show up. ## Create custom objects After the CustomResourceDefinition object has been created, you can create custom objects. Custom objects can contain custom fields. These fields can contain arbitrary JSON. In the following example, the `cronSpec` and `image` custom fields are set in a custom object of kind `CronTab`. The kind `CronTab` comes from the spec of the CustomResourceDefinition object you created above. If you save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image ``` and create it: ```shell kubectl apply -f my-crontab.yaml ``` You can then manage your CronTab objects using kubectl. For example: ```shell kubectl get crontab ``` Should print a list like this: ```none NAME AGE my-new-cron-object 6s ``` Resource names are not case-sensitive when using kubectl, and you can use either the singular or plural forms defined in the CRD, as well as any short names. You can also view the raw YAML data: ```shell kubectl get ct -o yaml ``` You should see that it contains the custom `cronSpec` and `image` fields from the YAML you used to create it: ```yaml apiVersion: v1 kind: List items: - apiVersion: stable.example.com/v1 kind: CronTab metadata: creationTimestamp: 2017-05-31T12:56:35Z generation: 1 name: my-new-cron-object namespace: default resourceVersion: "285" uid: 9423255b-4600-11e7-af6a-28d2447dc82b spec: cronSpec: '* * * * */5' image: my-awesome-cron-image metadata: resourceVersion: "" ``` ## Delete a CustomResourceDefinition When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint and delete all custom objects stored in it. ```shell kubectl delete -f resourcedefinition.yaml kubectl get crontabs ``` ```none Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could not find the requested resource (get crontabs.stable.example.com) ``` If you later recreate the same CustomResourceDefinition, it will start out empty. ## Specifying a structural schema CustomResources store structured data in custom fields (alongside the built-in fields `apiVersion`, `kind` and `metadata`, which the API server validates implicitly). With [OpenAPI v3.0 validation](#validation) a schema can be specified, which is validated during creation and updates, compare below for details and limits of such a schema. With `apiextensions.k8s.io/v1` the definition of a structural schema is mandatory for CustomResourceDefinitions. In the beta version of CustomResourceDefinition, the structural schema was optional. A structural schema is an [OpenAPI v3.0 validation schema](#validation) which: 1. specifies a non-empty type (via `type` in OpenAPI) for the root, for each specified field of an object node (via `properties` or `additionalProperties` in OpenAPI) and for each item in an array node (via `items` in OpenAPI), with the exception of: * a node with `x-kubernetes-int-or-string: true` * a node with `x-kubernetes-preserve-unknown-fields: true` 2. for each field in an object and each item in an array which is specified within any of `allOf`, `anyOf`, `oneOf` or `not`, the schema also specifies the field/item outside of those logical junctors (compare example 1 and 2). 3. does not set `description`, `type`, `default`, `additionalProperties`, `nullable` within an `allOf`, `anyOf`, `oneOf` or `not`, with the exception of the two pattern for `x-kubernetes-int-or-string: true` (see below). 4. if `metadata` is specified, then only restrictions on `metadata.name` and `metadata.generateName` are allowed. Non-structural example 1: ```yaml allOf: - properties: foo: ... ``` conflicts with rule 2. The following would be correct: ```yaml properties: foo: ... allOf: - properties: foo: ... ``` Non-structural example 2: ```yaml allOf: - items: properties: foo: ... ``` conflicts with rule 2. The following would be correct: ```yaml items: properties: foo: ... allOf: - items: properties: foo: ... ``` Non-structural example 3: ```yaml properties: foo: pattern: "abc" metadata: type: object properties: name: type: string pattern: "^a" finalizers: type: array items: type: string pattern: "my-finalizer" anyOf: - properties: bar: type: integer minimum: 42 required: ["bar"] description: "foo bar object" ``` is not a structural schema because of the following violations: * the type at the root is missing (rule 1). * the type of `foo` is missing (rule 1). * `bar` inside of `anyOf` is not specified outside (rule 2). * `bar`'s `type` is within `anyOf` (rule 3). * the description is set within `anyOf` (rule 3). * `metadata.finalizers` might not be restricted (rule 4). In contrast, the following, corresponding schema is structural: ```yaml type: object description: "foo bar object" properties: foo: type: string pattern: "abc" bar: type: integer metadata: type: object properties: name: type: string pattern: "^a" anyOf: - properties: bar: minimum: 42 required: ["bar"] ``` Violations of the structural schema rules are reported in the `NonStructural` condition in the CustomResourceDefinition. ### Field pruning CustomResourceDefinitions store validated resource data in the cluster's persistence store, {{< glossary_tooltip term_id="etcd" text="etcd">}}. As with native Kubernetes resources such as {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}}, if you specify a field that the API server does not recognize, the unknown field is _pruned_ (removed) before being persisted. {{< note >}} CRDs converted from `apiextensions.k8s.io/v1beta1` to `apiextensions.k8s.io/v1` might lack structural schemas, and `spec.preserveUnknownFields` might be `true`. For legacy CustomResourceDefinition objects created as `apiextensions.k8s.io/v1beta1` with `spec.preserveUnknownFields` set to `true`, the following is also true: * Pruning is not enabled. * You can store arbitrary data. For compatibility with `apiextensions.k8s.io/v1`, update your custom resource definitions to: 1. Use a structural OpenAPI schema. 2. Set `spec.preserveUnknownFields` to `false`. {{< /note >}} If you save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image someRandomField: 42 ``` and create it: ```shell kubectl create --validate=false -f my-crontab.yaml -o yaml ``` your output is similar to: ```console apiVersion: stable.example.com/v1 kind: CronTab metadata: creationTimestamp: 2017-05-31T12:56:35Z generation: 1 name: my-new-cron-object namespace: default resourceVersion: "285" uid: 9423255b-4600-11e7-af6a-28d2447dc82b spec: cronSpec: '* * * * */5' image: my-awesome-cron-image ``` Notice that the field `someRandomField` was pruned. This example turned off client-side validation to demonstrate the API server's behavior, by adding the `--validate=false` command line option. Because the [OpenAPI validation schemas are also published](#publish-validation-schema-in-openapi-v2) to clients, `kubectl` also checks for unknown fields and rejects those objects well before they would be sent to the API server. #### Controlling pruning By default, all unspecified fields for a custom resource, across all versions, are pruned. It is possible though to opt-out of that for specifc sub-trees of fields by adding `x-kubernetes-preserve-unknown-fields: true` in the [structural OpenAPI v3 validation schema](#specifying-a-structural-schema). For example: ```yaml type: object properties: json: x-kubernetes-preserve-unknown-fields: true ``` The field `json` can store any JSON value, without anything being pruned. You can also partially specify the permitted JSON; for example: ```yaml type: object properties: json: x-kubernetes-preserve-unknown-fields: true type: object description: this is arbitrary JSON ``` With this, only `object` type values are allowed. Pruning is enabled again for each specified property (or `additionalProperties`): ```yaml type: object properties: json: x-kubernetes-preserve-unknown-fields: true type: object properties: spec: type: object properties: foo: type: string bar: type: string ``` With this, the value: ```yaml json: spec: foo: abc bar: def something: x status: something: x ``` is pruned to: ```yaml json: spec: foo: abc bar: def status: something: x ``` This means that the `something` field in the specified `spec` object is pruned, but everything outside is not. ### IntOrString Nodes in a schema with `x-kubernetes-int-or-string: true` are excluded from rule 1, such that the following is structural: ```yaml type: object properties: foo: x-kubernetes-int-or-string: true ``` Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed (exactly those, without variations in order to additional fields): ```yaml x-kubernetes-int-or-string: true anyOf: - type: integer - type: string ... ``` and ```yaml x-kubernetes-int-or-string: true allOf: - anyOf: - type: integer - type: string - ... # zero or more ... ``` With one of those specification, both an integer and a string validate. In [Validation Schema Publishing](#publish-validation-schema-in-openapi-v2), `x-kubernetes-int-or-string: true` is unfolded to one of the two patterns shown above. ### RawExtension RawExtensions (as in `runtime.RawExtension` defined in [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery/blob/03ac7a9ade429d715a1a46ceaa3724c18ebae54f/pkg/runtime/types.go#L94)) holds complete Kubernetes objects, i.e. with `apiVersion` and `kind` fields. It is possible to specify those embedded objects (both completely without constraints or partially specified) by setting `x-kubernetes-embedded-resource: true`. For example: ```yaml type: object properties: foo: x-kubernetes-embedded-resource: true x-kubernetes-preserve-unknown-fields: true ``` Here, the field `foo` holds a complete object, e.g.: ```yaml foo: apiVersion: v1 kind: Pod spec: ... ``` Because `x-kubernetes-preserve-unknown-fields: true` is specified alongside, nothing is pruned. The use of `x-kubernetes-preserve-unknown-fields: true` is optional though. With `x-kubernetes-embedded-resource: true`, the `apiVersion`, `kind` and `metadata` are implicitly specified and validated. ## Serving multiple versions of a CRD See [Custom resource definition versioning](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) for more information about serving multiple versions of your CustomResourceDefinition and migrating your objects from one version to another. ## Advanced topics ### Finalizers *Finalizers* allow controllers to implement asynchronous pre-delete hooks. Custom objects support finalizers just like built-in objects. You can add a finalizer to a custom object like this: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: finalizers: - stable.example.com/finalizer ``` Identifiers of custom finalizers consist of a domain name, a forward slash and the name of the finalizer. Any controller can add a finalizer to any object's list of finalizers. The first delete request on an object with finalizers sets a value for the `metadata.deletionTimestamp` field but does not delete it. Once this value is set, entries in the `finalizers` list can only be removed. While any finalizers remain it is also impossible to force the deletion of an object. When the `metadata.deletionTimestamp` field is set, controllers watching the object execute any finalizers they handle and remove the finalizer from the list after they are done. It is the responsibility of each controller to remove its finalizer from the list. The value of `metadata.deletionGracePeriodSeconds` controls the interval between polling updates. Once the list of finalizers is empty, meaning all finalizers have been executed, the resource is deleted by Kubernetes. ### Validation Custom resources are validated via [OpenAPI v3 schemas](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject) and you can add additional validation using [admission webhooks](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook). Additionally, the following restrictions are applied to the schema: - These fields cannot be set: - `definitions`, - `dependencies`, - `deprecated`, - `discriminator`, - `id`, - `patternProperties`, - `readOnly`, - `writeOnly`, - `xml`, - `$ref`. - The field `uniqueItems` cannot be set to `true`. - The field `additionalProperties` cannot be set to `false`. - The field `additionalProperties` is mutually exclusive with `properties`. The `default` field can be set when the [Defaulting feature](#defaulting) is enabled, which is the case with `apiextensions.k8s.io/v1` CustomResourceDefinitions. Defaulting is in GA since 1.17 (beta since 1.16 with the `CustomResourceDefaulting` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled, which is the case automatically for many clusters for beta features). Refer to the [structural schemas](#specifying-a-structural-schema) section for other restrictions and CustomResourceDefinition features. The schema is defined in the CustomResourceDefinition. In the following example, the CustomResourceDefinition applies the following validations on the custom object: - `spec.cronSpec` must be a string and must be of the form described by the regular expression. - `spec.replicas` must be an integer and must have a minimum value of 1 and a maximum value of 10. Save the CustomResourceDefinition to `resourcedefinition.yaml`: ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: # openAPIV3Schema is the schema for validating custom objects. openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' image: type: string replicas: type: integer minimum: 1 maximum: 10 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct ``` and create it: ```shell kubectl apply -f resourcedefinition.yaml ``` A request to create a custom object of kind CronTab is rejected if there are invalid values in its fields. In the following example, the custom object contains fields with invalid values: - `spec.cronSpec` does not match the regular expression. - `spec.replicas` is greater than 10. If you save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * *" image: my-awesome-cron-image replicas: 15 ``` and attempt to create it: ```shell kubectl apply -f my-crontab.yaml ``` then you get an error: ```console The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}: validation failure list: spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' spec.replicas in body should be less than or equal to 10 ``` If the fields contain valid values, the object creation request is accepted. Save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image replicas: 5 ``` And create it: ```shell kubectl apply -f my-crontab.yaml crontab "my-new-cron-object" created ``` ### Defaulting {{< note >}} To use defaulting, your CustomResourceDefinition must use API version `apiextensions.k8s.io/v1`. {{< /note >}} Defaulting allows to specify default values in the [OpenAPI v3 validation schema](#validation): ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: # openAPIV3Schema is the schema for validating custom objects. openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' default: "5 0 * * *" image: type: string replicas: type: integer minimum: 1 maximum: 10 default: 1 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct ``` With this both `cronSpec` and `replicas` are defaulted: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: image: my-awesome-cron-image ``` leads to ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "5 0 * * *" image: my-awesome-cron-image replicas: 1 ``` Defaulting happens on the object * in the request to the API server using the request version defaults, * when reading from etcd using the storage version defaults, * after mutating admission plugins with non-empty patches using the admission webhook object version defaults. Defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd. Default values must be pruned (with the exception of defaults for `metadata` fields) and must validate against a provided schema. Default values for `metadata` fields of `x-kubernetes-embedded-resources: true` nodes (or parts of a default value covering `metadata`) are not pruned during CustomResourceDefinition creation, but through the pruning step during handling of requests. #### Defaulting and Nullable **New in 1.20:** null values for fields that either don't specify the nullable flag, or give it a `false` value, will be pruned before defaulting happens. If a default is present, it will be applied. When nullable is `true`, null values will be conserved and won't be defaulted. For example, given the OpenAPI schema below: ```yaml type: object properties: spec: type: object properties: foo: type: string nullable: false default: "default" bar: type: string nullable: true baz: type: string ``` creating an object with null values for `foo` and `bar` and `baz` ```yaml spec: foo: null bar: null baz: null ``` leads to ```yaml spec: foo: "default" bar: null ``` with `foo` pruned and defaulted because the field is non-nullable, `bar` maintaining the null value due to `nullable: true`, and `baz` pruned because the field is non-nullable and has no default. ### Publish Validation Schema in OpenAPI v2 CustomResourceDefinition [OpenAPI v3 validation schemas](#validation) which are [structural](#specifying-a-structural-schema) and [enable pruning](#field-pruning) are published as part of the [OpenAPI v2 spec](/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions) from Kubernetes API server. The [kubectl](/docs/reference/kubectl/overview) command-line tool consumes the published schema to perform client-side validation (`kubectl create` and `kubectl apply`), schema explanation (`kubectl explain`) on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation. The OpenAPI v3 validation schema is converted to OpenAPI v2 schema, and show up in `definitions` and `paths` fields in the [OpenAPI v2 spec](/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions). The following modifications are applied during the conversion to keep backwards compatibility with kubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejecting valid OpenAPI schemas that it doesn't understand. The conversion won't modify the validation schema defined in CRD, and therefore won't affect [validation](#validation) in the API server. 1. The following fields are removed as they aren't supported by OpenAPI v2 (in future versions OpenAPI v3 will be used without these restrictions) - The fields `allOf`, `anyOf`, `oneOf` and `not` are removed 2. If `nullable: true` is set, we drop `type`, `nullable`, `items` and `properties` because OpenAPI v2 is not able to express nullable. To avoid kubectl to reject good objects, this is necessary. ### Additional printer columns The kubectl tool relies on server-side output formatting. Your cluster's API server decides which columns are shown by the `kubectl get` command. You can customize these columns for a CustomResourceDefinition. The following example adds the `Spec`, `Replicas`, and `Age` columns. Save the CustomResourceDefinition to `resourcedefinition.yaml`: ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer additionalPrinterColumns: - name: Spec type: string description: The cron spec defining the interval a CronJob is run jsonPath: .spec.cronSpec - name: Replicas type: integer description: The number of jobs launched by the CronJob jsonPath: .spec.replicas - name: Age type: date jsonPath: .metadata.creationTimestamp ``` Create the CustomResourceDefinition: ```shell kubectl apply -f resourcedefinition.yaml ``` Create an instance using the `my-crontab.yaml` from the previous section. Invoke the server-side printing: ```shell kubectl get crontab my-new-cron-object ``` Notice the `NAME`, `SPEC`, `REPLICAS`, and `AGE` columns in the output: ``` NAME SPEC REPLICAS AGE my-new-cron-object * * * * * 1 7s ``` {{< note >}} The `NAME` column is implicit and does not need to be defined in the CustomResourceDefinition. {{< /note >}} #### Priority Each column includes a `priority` field. Currently, the priority differentiates between columns shown in standard view or wide view (using the `-o wide` flag). - Columns with priority `0` are shown in standard view. - Columns with priority greater than `0` are shown only in wide view. #### Type A column's `type` field can be any of the following (compare [OpenAPI v3 data types](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#dataTypes)): - `integer` – non-floating-point numbers - `number` – floating point numbers - `string` – strings - `boolean` – `true` or `false` - `date` – rendered differentially as time since this timestamp. If the value inside a CustomResource does not match the type specified for the column, the value is omitted. Use CustomResource validation to ensure that the value types are correct. #### Format A column's `format` field can be any of the following: - `int32` - `int64` - `float` - `double` - `byte` - `date` - `date-time` - `password` The column's `format` controls the style used when `kubectl` prints the value. ### Subresources Custom resources support `/status` and `/scale` subresources. The status and scale subresources can be optionally enabled by defining them in the CustomResourceDefinition. #### Status subresource When the status subresource is enabled, the `/status` subresource for the custom resource is exposed. - The status and the spec stanzas are represented by the `.status` and `.spec` JSONPaths respectively inside of a custom resource. - `PUT` requests to the `/status` subresource take a custom resource object and ignore changes to anything except the status stanza. - `PUT` requests to the `/status` subresource only validate the status stanza of the custom resource. - `PUT`/`POST`/`PATCH` requests to the custom resource ignore changes to the status stanza. - The `.metadata.generation` value is incremented for all changes, except for changes to `.metadata` or `.status`. - Only the following constructs are allowed at the root of the CRD OpenAPI validation schema: - `description` - `example` - `exclusiveMaximum` - `exclusiveMinimum` - `externalDocs` - `format` - `items` - `maximum` - `maxItems` - `maxLength` - `minimum` - `minItems` - `minLength` - `multipleOf` - `pattern` - `properties` - `required` - `title` - `type` - `uniqueItems` #### Scale subresource When the scale subresource is enabled, the `/scale` subresource for the custom resource is exposed. The `autoscaling/v1.Scale` object is sent as the payload for `/scale`. To enable the scale subresource, the following fields are defined in the CustomResourceDefinition. - `specReplicasPath` defines the JSONPath inside of a custom resource that corresponds to `scale.spec.replicas`. - It is a required value. - Only JSONPaths under `.spec` and with the dot notation are allowed. - If there is no value under the `specReplicasPath` in the custom resource, the `/scale` subresource will return an error on GET. - `statusReplicasPath` defines the JSONPath inside of a custom resource that corresponds to `scale.status.replicas`. - It is a required value. - Only JSONPaths under `.status` and with the dot notation are allowed. - If there is no value under the `statusReplicasPath` in the custom resource, the status replica value in the `/scale` subresource will default to 0. - `labelSelectorPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Status.Selector`. - It is an optional value. - It must be set to work with HPA. - Only JSONPaths under `.status` or `.spec` and with the dot notation are allowed. - If there is no value under the `labelSelectorPath` in the custom resource, the status selector value in the `/scale` subresource will default to the empty string. - The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form. In the following example, both status and scale subresources are enabled. Save the CustomResourceDefinition to `resourcedefinition.yaml`: ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer status: type: object properties: replicas: type: integer labelSelector: type: string # subresources describes the subresources for custom resources. subresources: # status enables the status subresource. status: {} # scale enables the scale subresource. scale: # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas. specReplicasPath: .spec.replicas # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas. statusReplicasPath: .status.replicas # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector. labelSelectorPath: .status.labelSelector scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct ``` And create it: ```shell kubectl apply -f resourcedefinition.yaml ``` After the CustomResourceDefinition object has been created, you can create custom objects. If you save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image replicas: 3 ``` and create it: ```shell kubectl apply -f my-crontab.yaml ``` Then new namespaced RESTful API endpoints are created at: ``` /apis/stable.example.com/v1/namespaces/*/crontabs/status ``` and ``` /apis/stable.example.com/v1/namespaces/*/crontabs/scale ``` A custom resource can be scaled using the `kubectl scale` command. For example, the following command sets `.spec.replicas` of the custom resource created above to 5: ```shell kubectl scale --replicas=5 crontabs/my-new-cron-object crontabs "my-new-cron-object" scaled kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}' 5 ``` You can use a [PodDisruptionBudget](/docs/tasks/run-application/configure-pdb/) to protect custom resources that have the scale subresource enabled. ### Categories Categories is a list of grouped resources the custom resource belongs to (eg. `all`). You can use `kubectl get ` to list the resources belonging to the category. The following example adds `all` in the list of categories in the CustomResourceDefinition and illustrates how to output the custom resource using `kubectl get all`. Save the following CustomResourceDefinition to `resourcedefinition.yaml`: ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct # categories is a list of grouped resources the custom resource belongs to. categories: - all ``` and create it: ```shell kubectl apply -f resourcedefinition.yaml ``` After the CustomResourceDefinition object has been created, you can create custom objects. Save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image ``` and create it: ```shell kubectl apply -f my-crontab.yaml ``` You can specify the category when using `kubectl get`: ``` kubectl get all ``` and it will include the custom resources of kind `CronTab`: ```console NAME AGE crontabs/my-new-cron-object 3s ``` ## {{% heading "whatsnext" %}} * Read about [custom resources](/docs/concepts/extend-kubernetes/api-extension/custom-resources/). * See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1-apiextensions-k8s-io). * Serve [multiple versions](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) of a CustomResourceDefinition.