diff --git a/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md b/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md index 5f035f9b3..d2b894d2d 100644 --- a/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md +++ b/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md @@ -44,7 +44,7 @@ that does not contain a discriminator. |---|---| | non-inlined non-discriminated union | Yes | | non-inlined discriminated union | Yes | -| inlined union with [patchMergeKey](/contributors/devel/api-conventions.md#strategic-merge-patch) only | Yes | +| inlined union with [patchMergeKey](/contributors/devel/sig-architecture/api-conventions.md#strategic-merge-patch) only | Yes | | other inlined union | No | For the inlined union with patchMergeKey, we move the tag to the parent struct's instead of diff --git a/contributors/design-proposals/api-machinery/aggregated-api-servers.md b/contributors/design-proposals/api-machinery/aggregated-api-servers.md index d436c6b96..3c3310b07 100644 --- a/contributors/design-proposals/api-machinery/aggregated-api-servers.md +++ b/contributors/design-proposals/api-machinery/aggregated-api-servers.md @@ -80,7 +80,7 @@ There are two configurations in which it makes sense to run `kube-aggregator`. `api.mycompany.com/v1/grobinators` from different apiservers. This restriction allows us to limit the scope of `kube-aggregator` to a manageable level. * Follow API conventions: APIs exposed by every API server should adhere to [kubernetes API - conventions](../../devel/api-conventions.md). + conventions](/contributors/devel/sig-architecture/api-conventions.md). * Support discovery API: Each API server should support the kubernetes discovery API (list the supported groupVersions at `/apis` and list the supported resources at `/apis//`) diff --git a/contributors/design-proposals/api-machinery/customresource-conversion-webhook.md b/contributors/design-proposals/api-machinery/customresource-conversion-webhook.md index 2b4aeb25e..54991fd6e 100644 --- a/contributors/design-proposals/api-machinery/customresource-conversion-webhook.md +++ b/contributors/design-proposals/api-machinery/customresource-conversion-webhook.md @@ -148,7 +148,7 @@ in *CRD v1* (apiextensions.k8s.io/v1), there will be only version list with no t #### Alternative approaches considered -First a defaulting approach is considered which per-version fields would be defaulted to top level fields. but that breaks backward incompatible change; Quoting from API [guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md#backward-compatibility-gotchas): +First a defaulting approach is considered which per-version fields would be defaulted to top level fields. but that breaks backward incompatible change; Quoting from API [guidelines](/contributors/devel/sig-architecture/api_changes.md#backward-compatibility-gotchas): > A single feature/property cannot be represented using multiple spec fields in the same API version simultaneously diff --git a/contributors/design-proposals/api-machinery/extending-api.md b/contributors/design-proposals/api-machinery/extending-api.md index f5e2de6a6..9a0c9263b 100644 --- a/contributors/design-proposals/api-machinery/extending-api.md +++ b/contributors/design-proposals/api-machinery/extending-api.md @@ -31,7 +31,7 @@ The `Version` object currently only specifies: ## Expectations about third party objects Every object that is added to a third-party Kubernetes object store is expected -to contain Kubernetes compatible [object metadata](../devel/api-conventions.md#metadata). +to contain Kubernetes compatible [object metadata](/contributors/devel/sig-architecture/api-conventions.md#metadata). This requirement enables the Kubernetes API server to provide the following features: * Filtering lists of objects via label queries. diff --git a/contributors/design-proposals/architecture/declarative-application-management.md b/contributors/design-proposals/architecture/declarative-application-management.md index a5fbdf24c..68425d027 100644 --- a/contributors/design-proposals/architecture/declarative-application-management.md +++ b/contributors/design-proposals/architecture/declarative-application-management.md @@ -6,7 +6,7 @@ Most users will deploy a combination of applications they build themselves, also In the case of the latter, users sometimes have the choice of using hosted SaaS products that are entirely managed by the service provider and are therefore opaque, also known as **_blackbox_** *services*. However, they often run open-source components themselves, and must configure, deploy, scale, secure, monitor, update, and otherwise manage the lifecycles of these **_whitebox_** *COTS applications*. -This document proposes a unified method of managing both bespoke and off-the-shelf applications declaratively using the same tools and application operator workflow, while leveraging developer-friendly CLIs and UIs, streamlining common tasks, and avoiding common pitfalls. The approach is based on observations of several dozen configuration projects and hundreds of configured applications within Google and in the Kubernetes ecosystem, as well as quantitative analysis of Borg configurations and work on the Kubernetes [system architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md), [API](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md), and command-line tool ([kubectl](https://github.com/kubernetes/community/wiki/Roadmap:-kubectl)). +This document proposes a unified method of managing both bespoke and off-the-shelf applications declaratively using the same tools and application operator workflow, while leveraging developer-friendly CLIs and UIs, streamlining common tasks, and avoiding common pitfalls. The approach is based on observations of several dozen configuration projects and hundreds of configured applications within Google and in the Kubernetes ecosystem, as well as quantitative analysis of Borg configurations and work on the Kubernetes [system architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md), [API](/contributors/devel/sig-architecture/api-conventions.md), and command-line tool ([kubectl](https://github.com/kubernetes/community/wiki/Roadmap:-kubectl)). The central idea is that a toolbox of composable configuration tools should manipulate configuration data in the form of declarative API resource specifications, which serve as a [declarative data model](https://docs.google.com/document/d/1RmHXdLhNbyOWPW_AtnnowaRfGejw-qlKQIuLKQWlwzs/edit#), not express configuration as code or some other representation that is restrictive, non-standard, and/or difficult to manipulate. diff --git a/contributors/design-proposals/architecture/principles.md b/contributors/design-proposals/architecture/principles.md index ae98d6609..7bb548d21 100644 --- a/contributors/design-proposals/architecture/principles.md +++ b/contributors/design-proposals/architecture/principles.md @@ -4,7 +4,7 @@ Principles to follow when extending Kubernetes. ## API -See also the [API conventions](../../devel/api-conventions.md). +See also the [API conventions](/contributors/devel/sig-architecture/api-conventions.md). * All APIs should be declarative. * API objects should be complementary and composable, not opaque wrappers. diff --git a/contributors/design-proposals/architecture/resource-management.md b/contributors/design-proposals/architecture/resource-management.md index 5b6d66b8d..472c3a565 100644 --- a/contributors/design-proposals/architecture/resource-management.md +++ b/contributors/design-proposals/architecture/resource-management.md @@ -60,7 +60,7 @@ Most resources also contain the [desired state ](https://kubernetes.io/docs/conc A few other subresources (e.g., `/scale`), with their own API types, similarly enable distinct authorization policies for controllers, and also polymorphism, since the same subresource type may be implemented for multiple parent resource types. Where distinct authorization policies are not required, polymorphism may be achieved simply by convention, using patch, akin to duck typing. -Supported data formats include YAML, JSON, and protocol buffers. +Supported data formats include YAML, JSON, and protocol buffers. Example resource: @@ -89,7 +89,7 @@ API groups may be exposed as a unified API surface while being served by distinc Each API server supports a custom [discovery API](https://github.com/kubernetes/client-go/blob/master/discovery/discovery_client.go) to enable clients to discover available API groups, versions, and types, and also [OpenAPI](https://kubernetes.io/blog/2016/12/kubernetes-supports-openapi/), which can be used to extract documentation and validation information about the resource types. -See the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md ) for more details. +See the [Kubernetes API conventions](/contributors/devel/sig-architecture/api-conventions.md ) for more details. ## Resource semantics and lifecycle @@ -97,12 +97,12 @@ Each API resource undergoes [a common sequence of behaviors](https://kubernetes. 1. [Authentication](https://kubernetes.io/docs/admin/authentication/) 2. [Authorization](https://kubernetes.io/docs/admin/authorization/): [Built-in](https://kubernetes.io/docs/admin/authorization/rbac/) and/or [administrator-defined](https://kubernetes.io/docs/admin/authorization/webhook/) identity-based policies -3. [Defaulting](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#defaulting): API-version-specific default values are made explicit and persisted +3. [Defaulting](/contributors/devel/sig-architecture/api-conventions.md#defaulting): API-version-specific default values are made explicit and persisted 4. Conversion: The apiserver converts between the client-requested [API version](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#API-versioning) and the version it uses to store each resource type in etcd 5. [Admission control](https://kubernetes.io/docs/admin/admission-controllers/): [Built-in](https://kubernetes.io/docs/admin/admission-controllers/) and/or [administrator-defined](https://kubernetes.io/docs/admin/extensible-admission-controllers/) resource-type-specific policies -6. [Validation](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#validation): Resource field values are validated. Other than the presence of required fields, the API resource schema is not currently validated, but optional validation may be added in the future +6. [Validation](/contributors/devel/sig-architecture/api-conventions.md#validation): Resource field values are validated. Other than the presence of required fields, the API resource schema is not currently validated, but optional validation may be added in the future 7. Idempotence: Resources are accessed via immutable client-provided, declarative-friendly names -8. [Optimistic concurrency](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#concurrency-control-and-consistency): Writes may specify a precondition that the **resourceVersion** last reported for a resource has not changed +8. [Optimistic concurrency](/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency): Writes may specify a precondition that the **resourceVersion** last reported for a resource has not changed 9. [Audit logging](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/): Records the sequence of changes to each resource by all actors Additional behaviors are supported upon deletion: diff --git a/contributors/design-proposals/auth/apparmor.md b/contributors/design-proposals/auth/apparmor.md index a88154bb1..f01b15489 100644 --- a/contributors/design-proposals/auth/apparmor.md +++ b/contributors/design-proposals/auth/apparmor.md @@ -63,7 +63,7 @@ and is supported on several # Alpha Design This section describes the proposed design for -[alpha-level](../devel/api_changes.md#alpha-beta-and-stable-versions) support, although +[alpha-level](/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions) support, although additional features are described in [future work](#future-work). For AppArmor alpha support (targeted for Kubernetes 1.4) we will enable: diff --git a/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md b/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md index c247eff8e..c1f30f5ca 100644 --- a/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md +++ b/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md @@ -47,7 +47,7 @@ feature's owner(s). The following are suggested conventions: - Features that touch multiple components should reserve the same key in each component to toggle on/off. - Alpha features should be disabled by default. Beta features may - be enabled by default. Refer to docs/devel/api_changes.md#alpha-beta-and-stable-versions + be enabled by default. Refer to [this file](/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions) for more detailed guidance on alpha vs. beta. ## Upgrade support diff --git a/contributors/design-proposals/multicluster/cluster-registry/api-design.md b/contributors/design-proposals/multicluster/cluster-registry/api-design.md index 2133f4993..3c2b748c4 100644 --- a/contributors/design-proposals/multicluster/cluster-registry/api-design.md +++ b/contributors/design-proposals/multicluster/cluster-registry/api-design.md @@ -84,7 +84,7 @@ Optional API operations: support WATCH for this API. Implementations can choose to support or not support this operation. An implementation that does not support the operation should return HTTP error 405, StatusMethodNotAllowed, per the - [relevant Kubernetes API conventions](/contributors/devel/api-conventions.md#error-codes). + [relevant Kubernetes API conventions](/contributors/devel/sig-architecture/api-conventions.md#error-codes). We also intend to support a use case where the server returns a file that can be stored for later use. We expect this to be doable with the standard API @@ -107,7 +107,7 @@ objects that contain a value for the `ClusterName` field. The `Cluster` object's of namespace scoped. The `Cluster` object will have `Spec` and `Status` fields, following the -[Kubernetes API conventions](/contributors/devel/api-conventions.md#spec-and-status). +[Kubernetes API conventions](/contributors/devel/sig-architecture/api-conventions.md#spec-and-status). There was argument in favor of a `State` field instead of `Spec` and `Status` fields, since the `Cluster` in the registry does not necessarily hold a user's intent about the cluster being represented, but instead may hold descriptive diff --git a/contributors/design-proposals/network/external-lb-source-ip-preservation.md b/contributors/design-proposals/network/external-lb-source-ip-preservation.md index 50140a0ee..f6a7d680f 100644 --- a/contributors/design-proposals/network/external-lb-source-ip-preservation.md +++ b/contributors/design-proposals/network/external-lb-source-ip-preservation.md @@ -50,7 +50,7 @@ lot of applications and customer use-cases. # Alpha Design This section describes the proposed design for -[alpha-level](../devel/api_changes.md#alpha-beta-and-stable-versions) support, although +[alpha-level](/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions) support, although additional features are described in [future work](#future-work). ## Overview diff --git a/contributors/design-proposals/node/secret-configmap-downwardapi-file-mode.md b/contributors/design-proposals/node/secret-configmap-downwardapi-file-mode.md index cdfe1e1c6..1d5bd7b7a 100644 --- a/contributors/design-proposals/node/secret-configmap-downwardapi-file-mode.md +++ b/contributors/design-proposals/node/secret-configmap-downwardapi-file-mode.md @@ -49,7 +49,7 @@ This was asked on the mailing list here[2] and here[3], too. Several alternatives have been considered: * Add a mode to the API definition when using secrets: this is backward - compatible as described in (docs/devel/api_changes.md) IIUC and seems like the + compatible as described [here](/contributors/devel/sig-architecture/api_changes.md) IIUC and seems like the way to go. Also @thockin said in the ML that he would consider such an approach. But it might be worth to consider if we want to do the same for configmaps or owners, but there is no need to do it now either. diff --git a/contributors/design-proposals/storage/container-storage-interface.md b/contributors/design-proposals/storage/container-storage-interface.md index 9c4db8b8e..e368b4acc 100644 --- a/contributors/design-proposals/storage/container-storage-interface.md +++ b/contributors/design-proposals/storage/container-storage-interface.md @@ -243,7 +243,7 @@ type VolumeAttachment struct { metav1.TypeMeta `json:",inline"` // Standard object metadata. - // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` diff --git a/contributors/design-proposals/storage/csi-snapshot.md b/contributors/design-proposals/storage/csi-snapshot.md index beb46d58a..19c3c38b6 100644 --- a/contributors/design-proposals/storage/csi-snapshot.md +++ b/contributors/design-proposals/storage/csi-snapshot.md @@ -59,7 +59,7 @@ The API design of VolumeSnapshot and VolumeSnapshotContent is modeled after Pers type VolumeSnapshot struct { metav1.TypeMeta `json:",inline"` // Standard object's metadata. - // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` @@ -144,7 +144,7 @@ Note that if an error occurs before the snapshot is cut, `Error` will be set and type VolumeSnapshotContent struct { metav1.TypeMeta `json:",inline"` // Standard object's metadata. - // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` @@ -234,7 +234,7 @@ A new VolumeSnapshotClass API object will be added instead of reusing the existi type VolumeSnapshotClass struct { metav1.TypeMeta `json:",inline"` // Standard object's metadata. - // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` diff --git a/contributors/devel/README.md b/contributors/devel/README.md index ab1f9c747..31c0bcacf 100644 --- a/contributors/devel/README.md +++ b/contributors/devel/README.md @@ -26,7 +26,7 @@ Guide](http://kubernetes.io/docs/admin/). * **Testing** ([testing.md](sig-testing/testing.md)): How to run unit, integration, and end-to-end tests in your development sandbox. -* **Conformance Testing** ([conformance-tests.md](conformance-tests.md)) +* **Conformance Testing** ([conformance-tests.md](sig-architecture/conformance-tests.md)) What is conformance testing and how to create/manage them. * **Hunting flaky tests** ([flaky-tests.md](sig-testing/flaky-tests.md)): We have a goal of 99.9% flake free tests. diff --git a/contributors/devel/api-conventions.md b/contributors/devel/api-conventions.md index 2e0bd7ad8..0e122c5f1 100644 --- a/contributors/devel/api-conventions.md +++ b/contributors/devel/api-conventions.md @@ -1,1367 +1,3 @@ -API Conventions -=============== - -Updated: 3/7/2017 - -*This document is oriented at users who want a deeper understanding of the -Kubernetes API structure, and developers wanting to extend the Kubernetes API. -An introduction to using resources with kubectl can be found in [the object management overview](https://kubernetes.io/docs/tutorials/object-management-kubectl/object-management/).* - -**Table of Contents** - - - - [Types (Kinds)](#types-kinds) - - [Resources](#resources) - - [Objects](#objects) - - [Metadata](#metadata) - - [Spec and Status](#spec-and-status) - - [Typical status properties](#typical-status-properties) - - [References to related objects](#references-to-related-objects) - - [Lists of named subobjects preferred over maps](#lists-of-named-subobjects-preferred-over-maps) - - [Primitive types](#primitive-types) - - [Constants](#constants) - - [Unions](#unions) - - [Lists and Simple kinds](#lists-and-simple-kinds) - - [Differing Representations](#differing-representations) - - [Verbs on Resources](#verbs-on-resources) - - [PATCH operations](#patch-operations) - - [Strategic Merge Patch](#strategic-merge-patch) - - [Idempotency](#idempotency) - - [Optional vs. Required](#optional-vs-required) - - [Defaulting](#defaulting) - - [Late Initialization](#late-initialization) - - [Concurrency Control and Consistency](#concurrency-control-and-consistency) - - [Serialization Format](#serialization-format) - - [Units](#units) - - [Selecting Fields](#selecting-fields) - - [Object references](#object-references) - - [HTTP Status codes](#http-status-codes) - - [Success codes](#success-codes) - - [Error codes](#error-codes) - - [Response Status Kind](#response-status-kind) - - [Events](#events) - - [Naming conventions](#naming-conventions) - - [Label, selector, and annotation conventions](#label-selector-and-annotation-conventions) - - [WebSockets and SPDY](#websockets-and-spdy) - - [Validation](#validation) - - -The conventions of the [Kubernetes API](https://kubernetes.io/docs/api/) (and related APIs in the -ecosystem) are intended to ease client development and ensure that configuration -mechanisms can be implemented that work across a diverse set of use cases -consistently. - -The general style of the Kubernetes API is RESTful - clients create, update, -delete, or retrieve a description of an object via the standard HTTP verbs -(POST, PUT, DELETE, and GET) - and those APIs preferentially accept and return -JSON. Kubernetes also exposes additional endpoints for non-standard verbs and -allows alternative content types. All of the JSON accepted and returned by the -server has a schema, identified by the "kind" and "apiVersion" fields. Where -relevant HTTP header fields exist, they should mirror the content of JSON -fields, but the information should not be represented only in the HTTP header. - -The following terms are defined: - -* **Kind** the name of a particular object schema (e.g. the "Cat" and "Dog" -kinds would have different attributes and properties) -* **Resource** a representation of a system entity, sent or retrieved as JSON -via HTTP to the server. Resources are exposed via: - * Collections - a list of resources of the same type, which may be queryable - * Elements - an individual resource, addressable via a URL -* **API Group** a set of resources that are exposed together. Along -with the version is exposed in the "apiVersion" field as "GROUP/VERSION", e.g. -"policy.k8s.io/v1". - -Each resource typically accepts and returns data of a single kind. A kind may be -accepted or returned by multiple resources that reflect specific use cases. For -instance, the kind "Pod" is exposed as a "pods" resource that allows end users -to create, update, and delete pods, while a separate "pod status" resource (that -acts on "Pod" kind) allows automated processes to update a subset of the fields -in that resource. - -Resources are bound together in API groups - each group may have one or more -versions that evolve independent of other API groups, and each version within -the group has one or more resources. Group names are typically in domain name -form - the Kubernetes project reserves use of the empty group, all single -word names ("extensions", "apps"), and any group name ending in "*.k8s.io" for -its sole use. When choosing a group name, we recommend selecting a subdomain -your group or organization owns, such as "widget.mycompany.com". - -Resource collections should be all lowercase and plural, whereas kinds are -CamelCase and singular. Group names must be lower case and be valid DNS -subdomains. - - -## Types (Kinds) - -Kinds are grouped into three categories: - -1. **Objects** represent a persistent entity in the system. - - Creating an API object is a record of intent - once created, the system will -work to ensure that resource exists. All API objects have common metadata. - - An object may have multiple resources that clients can use to perform -specific actions that create, update, delete, or get. - - Examples: `Pod`, `ReplicationController`, `Service`, `Namespace`, `Node`. - -2. **Lists** are collections of **resources** of one (usually) or more -(occasionally) kinds. - - The name of a list kind must end with "List". Lists have a limited set of -common metadata. All lists use the required "items" field to contain the array -of objects they return. Any kind that has the "items" field must be a list kind. - - Most objects defined in the system should have an endpoint that returns the -full set of resources, as well as zero or more endpoints that return subsets of -the full list. Some objects may be singletons (the current user, the system -defaults) and may not have lists. - - In addition, all lists that return objects with labels should support label -filtering (see [the labels documentation](https://kubernetes.io/docs/user-guide/labels/)), and most -lists should support filtering by fields. - - Examples: `PodLists`, `ServiceLists`, `NodeLists`. - - TODO: Describe field filtering below or in a separate doc. - -3. **Simple** kinds are used for specific actions on objects and for -non-persistent entities. - - Given their limited scope, they have the same set of limited common metadata -as lists. - - For instance, the "Status" kind is returned when errors occur and is not -persisted in the system. - - Many simple resources are "subresources", which are rooted at API paths of -specific resources. When resources wish to expose alternative actions or views -that are closely coupled to a single resource, they should do so using new -sub-resources. Common subresources include: - - * `/binding`: Used to bind a resource representing a user request (e.g., Pod, -PersistentVolumeClaim) to a cluster infrastructure resource (e.g., Node, -PersistentVolume). - * `/status`: Used to write just the status portion of a resource. For -example, the `/pods` endpoint only allows updates to `metadata` and `spec`, -since those reflect end-user intent. An automated process should be able to -modify status for users to see by sending an updated Pod kind to the server to -the "/pods/<name>/status" endpoint - the alternate endpoint allows -different rules to be applied to the update, and access to be appropriately -restricted. - * `/scale`: Used to read and write the count of a resource in a manner that -is independent of the specific resource schema. - - Two additional subresources, `proxy` and `portforward`, provide access to -cluster resources as described in -[accessing the cluster](https://kubernetes.io/docs/user-guide/accessing-the-cluster/). - -The standard REST verbs (defined below) MUST return singular JSON objects. Some -API endpoints may deviate from the strict REST pattern and return resources that -are not singular JSON objects, such as streams of JSON objects or unstructured -text log data. - -A common set of "meta" API objects are used across all API groups and are -thus considered part of the server group named `meta.k8s.io`. These types may -evolve independent of the API group that uses them and API servers may allow -them to be addressed in their generic form. Examples are `ListOptions`, -`DeleteOptions`, `List`, `Status`, `WatchEvent`, and `Scale`. For historical -reasons these types are part of each existing API group. Generic tools like -quota, garbage collection, autoscalers, and generic clients like kubectl -leverage these types to define consistent behavior across different resource -types, like the interfaces in programming languages. - -The term "kind" is reserved for these "top-level" API types. The term "type" -should be used for distinguishing sub-categories within objects or subobjects. - -### Resources - -All JSON objects returned by an API MUST have the following fields: - -* kind: a string that identifies the schema this object should have -* apiVersion: a string that identifies the version of the schema the object -should have - -These fields are required for proper decoding of the object. They may be -populated by the server by default from the specified URL path, but the client -likely needs to know the values in order to construct the URL path. - -### Objects - -#### Metadata - -Every object kind MUST have the following metadata in a nested object field -called "metadata": - -* namespace: a namespace is a DNS compatible label that objects are subdivided -into. The default namespace is 'default'. See -[the namespace docs](https://kubernetes.io/docs/user-guide/namespaces/) for more. -* name: a string that uniquely identifies this object within the current -namespace (see [the identifiers docs](https://kubernetes.io/docs/user-guide/identifiers/)). -This value is used in the path when retrieving an individual object. -* uid: a unique in time and space value (typically an RFC 4122 generated -identifier, see [the identifiers docs](https://kubernetes.io/docs/user-guide/identifiers/)) -used to distinguish between objects with the same name that have been deleted -and recreated - -Every object SHOULD have the following metadata in a nested object field called -"metadata": - -* resourceVersion: a string that identifies the internal version of this object -that can be used by clients to determine when objects have changed. This value -MUST be treated as opaque by clients and passed unmodified back to the server. -Clients should not assume that the resource version has meaning across -namespaces, different kinds of resources, or different servers. (See -[concurrency control](#concurrency-control-and-consistency), below, for more -details.) -* generation: a sequence number representing a specific generation of the -desired state. Set by the system and monotonically increasing, per-resource. May -be compared, such as for RAW and WAW consistency. -* creationTimestamp: a string representing an RFC 3339 date of the date and time -an object was created -* deletionTimestamp: a string representing an RFC 3339 date of the date and time -after which this resource will be deleted. This field is set by the server when -a graceful deletion is requested by the user, and is not directly settable by a -client. The resource will be deleted (no longer visible from resource lists, and -not reachable by name) after the time in this field except when the object has -a finalizer set. In case the finalizer is set the deletion of the object is -postponed at least until the finalizer is removed. -Once the deletionTimestamp is set, this value may not be unset or be set further -into the future, although it may be shortened or the resource may be deleted -prior to this time. -* labels: a map of string keys and values that can be used to organize and -categorize objects (see [the labels docs](https://kubernetes.io/docs/user-guide/labels/)) -* annotations: a map of string keys and values that can be used by external -tooling to store and retrieve arbitrary metadata about this object (see -[the annotations docs](https://kubernetes.io/docs/user-guide/annotations/)) - -Labels are intended for organizational purposes by end users (select the pods -that match this label query). Annotations enable third-party automation and -tooling to decorate objects with additional metadata for their own use. - -#### Spec and Status - -By convention, the Kubernetes API makes a distinction between the specification -of the desired state of an object (a nested object field called "spec") and the -status of the object at the current time (a nested object field called -"status"). The specification is a complete description of the desired state, -including configuration settings provided by the user, -[default values](#defaulting) expanded by the system, and properties initialized -or otherwise changed after creation by other ecosystem components (e.g., -schedulers, auto-scalers), and is persisted in stable storage with the API -object. If the specification is deleted, the object will be purged from the -system. The status summarizes the current state of the object in the system, and -is usually persisted with the object by an automated processes but may be -generated on the fly. At some cost and perhaps some temporary degradation in -behavior, the status could be reconstructed by observation if it were lost. - -When a new version of an object is POSTed or PUT, the "spec" is updated and -available immediately. Over time the system will work to bring the "status" into -line with the "spec". The system will drive toward the most recent "spec" -regardless of previous versions of that stanza. In other words, if a value is -changed from 2 to 5 in one PUT and then back down to 3 in another PUT the system -is not required to 'touch base' at 5 before changing the "status" to 3. In other -words, the system's behavior is *level-based* rather than *edge-based*. This -enables robust behavior in the presence of missed intermediate state changes. - -The Kubernetes API also serves as the foundation for the declarative -configuration schema for the system. In order to facilitate level-based -operation and expression of declarative configuration, fields in the -specification should have declarative rather than imperative names and -semantics -- they represent the desired state, not actions intended to yield the -desired state. - -The PUT and POST verbs on objects MUST ignore the "status" values, to avoid -accidentally overwriting the status in read-modify-write scenarios. A `/status` -subresource MUST be provided to enable system components to update statuses of -resources they manage. - -Otherwise, PUT expects the whole object to be specified. Therefore, if a field -is omitted it is assumed that the client wants to clear that field's value. The -PUT verb does not accept partial updates. Modification of just part of an object -may be achieved by GETting the resource, modifying part of the spec, labels, or -annotations, and then PUTting it back. See -[concurrency control](#concurrency-control-and-consistency), below, regarding -read-modify-write consistency when using this pattern. Some objects may expose -alternative resource representations that allow mutation of the status, or -performing custom actions on the object. - -All objects that represent a physical resource whose state may vary from the -user's desired intent SHOULD have a "spec" and a "status". Objects whose state -cannot vary from the user's desired intent MAY have only "spec", and MAY rename -"spec" to a more appropriate name. - -Objects that contain both spec and status should not contain additional -top-level fields other than the standard metadata fields. - -Some objects which are not persisted in the system - such as `SubjectAccessReview` -and other webhook style calls - may choose to add spec and status to encapsulate -a "call and response" pattern. The spec is the request (often a request for -information) and the status is the response. For these RPC like objects the only -operation may be POST, but having a consistent schema between submission and -response reduces the complexity of these clients. - - -##### Typical status properties - -**Conditions** represent the latest available observations of an object's -state. They are an extension mechanism intended to be used when the details of -an observation are not a priori known or would not apply to all instances of a -given Kind. For observations that are well known and apply to all instances, a -regular field is preferred. An example of a Condition that probably should -have been a regular field is Pod's "Ready" condition - it is managed by core -controllers, it is well understood, and it applies to all Pods. - -Objects may report multiple conditions, and new types of conditions may be -added in the future or by 3rd party controllers. Therefore, conditions are -represented using a list/slice, where all have similar structure. - -The `FooCondition` type for some resource type `Foo` may include a subset of the -following fields, but must contain at least `type` and `status` fields: - -```go - Type FooConditionType `json:"type" description:"type of Foo condition"` - Status ConditionStatus `json:"status" description:"status of the condition, one of True, False, Unknown"` - - // +optional - Reason *string `json:"reason,omitempty" description:"one-word CamelCase reason for the condition's last transition"` - // +optional - Message *string `json:"message,omitempty" description:"human-readable message indicating details about last transition"` - - // +optional - LastHeartbeatTime *unversioned.Time `json:"lastHeartbeatTime,omitempty" description:"last time we got an update on a given condition"` - // +optional - LastTransitionTime *unversioned.Time `json:"lastTransitionTime,omitempty" description:"last time the condition transit from one status to another"` -``` - -Additional fields may be added in the future. - -Do not use fields that you don't need - simpler is better. - -Use of the `Reason` field is encouraged. - -Use the `LastHeartbeatTime` with great caution - frequent changes to this field -can cause a large fan-out effect for some resources. - -Conditions should be added to explicitly convey properties that users and -components care about rather than requiring those properties to be inferred from -other observations. Once defined, the meaning of a Condition can not be -changed arbitrarily - it becomes part of the API, and has the same backwards- -and forwards-compatibility concerns of any other part of the API. - -Condition status values may be `True`, `False`, or `Unknown`. The absence of a -condition should be interpreted the same as `Unknown`. How controllers handle -`Unknown` depends on the Condition in question. - -Condition types should indicate state in the "abnormal-true" polarity. For -example, if the condition indicates when a policy is invalid, the "is valid" -case is probably the norm, so the condition should be called "Invalid". - -The thinking around conditions has evolved over time, so there are several -non-normative examples in wide use. - -In general, condition values may change back and forth, but some condition -transitions may be monotonic, depending on the resource and condition type. -However, conditions are observations and not, themselves, state machines, nor do -we define comprehensive state machines for objects, nor behaviors associated -with state transitions. The system is level-based rather than edge-triggered, -and should assume an Open World. - -An example of an oscillating condition type is `Ready` (despite it running -afoul of current guidance), which indicates the object was believed to be fully -operational at the time it was last probed. A possible monotonic condition -could be `Failed`. A `True` status for `Failed` would imply failure with no -retry. An object that was still active would generally not have a `Failed` -condition. - -Some resources in the v1 API contain fields called **`phase`**, and associated -`message`, `reason`, and other status fields. The pattern of using `phase` is -deprecated. Newer API types should use conditions instead. Phase was -essentially a state-machine enumeration field, that contradicted [system-design -principles](../design-proposals/architecture/principles.md#control-logic) and -hampered evolution, since [adding new enum values breaks backward -compatibility](api_changes.md). Rather than encouraging clients to infer -implicit properties from phases, we prefer to explicitly expose the individual -conditions that clients need to monitor. Conditions also have the benefit that -it is possible to create some conditions with uniform meaning across all -resource types, while still exposing others that are unique to specific -resource types. See [#7856](http://issues.k8s.io/7856) for more details and -discussion. - -In condition types, and everywhere else they appear in the API, **`Reason`** is -intended to be a one-word, CamelCase representation of the category of cause of -the current status, and **`Message`** is intended to be a human-readable phrase -or sentence, which may contain specific details of the individual occurrence. -`Reason` is intended to be used in concise output, such as one-line -`kubectl get` output, and in summarizing occurrences of causes, whereas -`Message` is intended to be presented to users in detailed status explanations, -such as `kubectl describe` output. - -Historical information status (e.g., last transition time, failure counts) is -only provided with reasonable effort, and is not guaranteed to not be lost. - -Status information that may be large (especially proportional in size to -collections of other resources, such as lists of references to other objects -- -see below) and/or rapidly changing, such as -[resource usage](../design-proposals/scheduling/resources.md#usage-data), should be put into separate -objects, with possibly a reference from the original object. This helps to -ensure that GETs and watch remain reasonably efficient for the majority of -clients, which may not need that data. - -Some resources report the `observedGeneration`, which is the `generation` most -recently observed by the component responsible for acting upon changes to the -desired state of the resource. This can be used, for instance, to ensure that -the reported status reflects the most recent desired status. - -#### References to related objects - -References to loosely coupled sets of objects, such as -[pods](https://kubernetes.io/docs/user-guide/pods/) overseen by a -[replication controller](https://kubernetes.io/docs/user-guide/replication-controller/), are usually -best referred to using a [label selector](https://kubernetes.io/docs/user-guide/labels/). In order to -ensure that GETs of individual objects remain bounded in time and space, these -sets may be queried via separate API queries, but will not be expanded in the -referring object's status. - -References to specific objects, especially specific resource versions and/or -specific fields of those objects, are specified using the `ObjectReference` type -(or other types representing strict subsets of it). Unlike partial URLs, the -ObjectReference type facilitates flexible defaulting of fields from the -referring object or other contextual information. - -References in the status of the referee to the referrer may be permitted, when -the references are one-to-one and do not need to be frequently updated, -particularly in an edge-based manner. - -#### Lists of named subobjects preferred over maps - -Discussed in [#2004](http://issue.k8s.io/2004) and elsewhere. There are no maps -of subobjects in any API objects. Instead, the convention is to use a list of -subobjects containing name fields. - -For example: - -```yaml -ports: - - name: www - containerPort: 80 -``` - -vs. - -```yaml -ports: - www: - containerPort: 80 -``` - -This rule maintains the invariant that all JSON/YAML keys are fields in API -objects. The only exceptions are pure maps in the API (currently, labels, -selectors, annotations, data), as opposed to sets of subobjects. - -#### Primitive types - -* Avoid floating-point values as much as possible, and never use them in spec. - Floating-point values cannot be reliably round-tripped (encoded and - re-decoded) without changing, and have varying precision and representations - across languages and architectures. -* All numbers (e.g., uint32, int64) are converted to float64 by Javascript and - some other languages, so any field which is expected to exceed that either in - magnitude or in precision (specifically integer values > 53 bits) should be - serialized and accepted as strings. -* Do not use unsigned integers, due to inconsistent support across languages and - libraries. Just validate that the integer is non-negative if that's the case. -* Do not use enums. Use aliases for string instead (e.g., `NodeConditionType`). -* Look at similar fields in the API (e.g., ports, durations) and follow the - conventions of existing fields. -* All public integer fields MUST use the Go `(u)int32` or Go `(u)int64` types, - not `(u)int` (which is ambiguous depending on target platform). Internal - types may use `(u)int`. -* Think twice about `bool` fields. Many ideas start as boolean but eventually - trend towards a small set of mutually exclusive options. Plan for future - expansions by describing the policy options explicitly as a string type - alias (e.g. `TerminationMessagePolicy`). - -#### Constants - -Some fields will have a list of allowed values (enumerations). These values will -be strings, and they will be in CamelCase, with an initial uppercase letter. -Examples: `ClusterFirst`, `Pending`, `ClientIP`. - -#### Unions - -Sometimes, at most one of a set of fields can be set. For example, the -[volumes] field of a PodSpec has 17 different volume type-specific fields, such -as `nfs` and `iscsi`. All fields in the set should be -[Optional](#optional-vs-required). - -Sometimes, when a new type is created, the api designer may anticipate that a -union will be needed in the future, even if only one field is allowed initially. -In this case, be sure to make the field [Optional](#optional-vs-required) -optional. In the validation, you may still return an error if the sole field is -unset. Do not set a default value for that field. - -### Lists and Simple kinds - -Every list or simple kind SHOULD have the following metadata in a nested object -field called "metadata": - -* resourceVersion: a string that identifies the common version of the objects -returned by in a list. This value MUST be treated as opaque by clients and -passed unmodified back to the server. A resource version is only valid within a -single namespace on a single kind of resource. - -Every simple kind returned by the server, and any simple kind sent to the server -that must support idempotency or optimistic concurrency should return this -value. Since simple resources are often used as input alternate actions that -modify objects, the resource version of the simple resource should correspond to -the resource version of the object. - - -## Differing Representations - -An API may represent a single entity in different ways for different clients, or -transform an object after certain transitions in the system occur. In these -cases, one request object may have two representations available as different -resources, or different kinds. - -An example is a Service, which represents the intent of the user to group a set -of pods with common behavior on common ports. When Kubernetes detects a pod -matches the service selector, the IP address and port of the pod are added to an -Endpoints resource for that Service. The Endpoints resource exists only if the -Service exists, but exposes only the IPs and ports of the selected pods. The -full service is represented by two distinct resources - under the original -Service resource the user created, as well as in the Endpoints resource. - -As another example, a "pod status" resource may accept a PUT with the "pod" -kind, with different rules about what fields may be changed. - -Future versions of Kubernetes may allow alternative encodings of objects beyond -JSON. - - -## Verbs on Resources - -API resources should use the traditional REST pattern: - -* GET /<resourceNamePlural> - Retrieve a list of type -<resourceName>, e.g. GET /pods returns a list of Pods. -* POST /<resourceNamePlural> - Create a new resource from the JSON object -provided by the client. -* GET /<resourceNamePlural>/<name> - Retrieves a single resource -with the given name, e.g. GET /pods/first returns a Pod named 'first'. Should be -constant time, and the resource should be bounded in size. -* DELETE /<resourceNamePlural>/<name> - Delete the single resource -with the given name. DeleteOptions may specify gracePeriodSeconds, the optional -duration in seconds before the object should be deleted. Individual kinds may -declare fields which provide a default grace period, and different kinds may -have differing kind-wide default grace periods. A user provided grace period -overrides a default grace period, including the zero grace period ("now"). -* PUT /<resourceNamePlural>/<name> - Update or create the resource -with the given name with the JSON object provided by the client. -* PATCH /<resourceNamePlural>/<name> - Selectively modify the -specified fields of the resource. See more information [below](#patch-operations). -* GET /<resourceNamePlural>&watch=true - Receive a stream of JSON -objects corresponding to changes made to any resource of the given kind over -time. - -### PATCH operations - -The API supports three different PATCH operations, determined by their -corresponding Content-Type header: - -* JSON Patch, `Content-Type: application/json-patch+json` - * As defined in [RFC6902](https://tools.ietf.org/html/rfc6902), a JSON Patch is -a sequence of operations that are executed on the resource, e.g. `{"op": "add", -"path": "/a/b/c", "value": [ "foo", "bar" ]}`. For more details on how to use -JSON Patch, see the RFC. -* Merge Patch, `Content-Type: application/merge-patch+json` - * As defined in [RFC7386](https://tools.ietf.org/html/rfc7386), a Merge Patch -is essentially a partial representation of the resource. The submitted JSON is -"merged" with the current resource to create a new one, then the new one is -saved. For more details on how to use Merge Patch, see the RFC. -* Strategic Merge Patch, `Content-Type: application/strategic-merge-patch+json` - * Strategic Merge Patch is a custom implementation of Merge Patch. For a -detailed explanation of how it works and why it needed to be introduced, see -below. - -#### Strategic Merge Patch - -Details of Strategic Merge Patch are covered [here](strategic-merge-patch.md). - -## Idempotency - -All compatible Kubernetes APIs MUST support "name idempotency" and respond with -an HTTP status code 409 when a request is made to POST an object that has the -same name as an existing object in the system. See -[the identifiers docs](https://kubernetes.io/docs/user-guide/identifiers/) for details. - -Names generated by the system may be requested using `metadata.generateName`. -GenerateName indicates that the name should be made unique by the server prior -to persisting it. A non-empty value for the field indicates the name will be -made unique (and the name returned to the client will be different than the name -passed). The value of this field will be combined with a unique suffix on the -server if the Name field has not been provided. The provided value must be valid -within the rules for Name, and may be truncated by the length of the suffix -required to make the value unique on the server. If this field is specified, and -Name is not present, the server will NOT return a 409 if the generated name -exists - instead, it will either return 201 Created or 504 with Reason -`ServerTimeout` indicating a unique name could not be found in the time -allotted, and the client should retry (optionally after the time indicated in -the Retry-After header). - -## Optional vs. Required - -Fields must be either optional or required. - -Optional fields have the following properties: - -- They have the `+optional` comment tag in Go. -- They are a pointer type in the Go definition (e.g. `bool *awesomeFlag`) or -have a built-in `nil` value (e.g. maps and slices). -- The API server should allow POSTing and PUTing a resource with this field -unset. - -In most cases, optional fields should also have the `omitempty` struct tag (the -`omitempty` option specifies that the field should be omitted from the json -encoding if the field has an empty value). However, If you want to have -different logic for an optional field which is not provided vs. provided with -empty values, do not use `omitempty` (e.g. https://github.com/kubernetes/kubernetes/issues/34641). - -Note that for backward compatibility, any field that has the `omitempty` struct -tag will considered to be optional but this may change in future and having -the `+optional` comment tag is highly recommended. - -Required fields have the opposite properties, namely: - -- They do not have an `+optional` comment tag. -- They do not have an `omitempty` struct tag. -- They are not a pointer type in the Go definition (e.g. `bool otherFlag`). -- The API server should not allow POSTing or PUTing a resource with this field -unset. - -Using the `+optional` or the `omitempty` tag causes OpenAPI documentation to -reflect that the field is optional. - -Using a pointer allows distinguishing unset from the zero value for that type. -There are some cases where, in principle, a pointer is not needed for an -optional field since the zero value is forbidden, and thus implies unset. There -are examples of this in the codebase. However: - -- it can be difficult for implementors to anticipate all cases where an empty -value might need to be distinguished from a zero value -- structs are not omitted from encoder output even where omitempty is specified, -which is messy; -- having a pointer consistently imply optional is clearer for users of the Go -language client, and any other clients that use corresponding types - -Therefore, we ask that pointers always be used with optional fields that do not -have a built-in `nil` value. - - -## Defaulting - -Default resource values are API version-specific, and they are applied during -the conversion from API-versioned declarative configuration to internal objects -representing the desired state (`Spec`) of the resource. Subsequent GETs of the -resource will include the default values explicitly. - -Incorporating the default values into the `Spec` ensures that `Spec` depicts the -full desired state so that it is easier for the system to determine how to -achieve the state, and for the user to know what to anticipate. - -API version-specific default values are set by the API server. - -## Late Initialization - -Late initialization is when resource fields are set by a system controller -after an object is created/updated. - -For example, the scheduler sets the `pod.spec.nodeName` field after the pod is -created. - -Late-initializers should only make the following types of modifications: - - Setting previously unset fields - - Adding keys to maps - - Adding values to arrays which have mergeable semantics -(`patchStrategy:"merge"` attribute in the type definition). - -These conventions: - 1. allow a user (with sufficient privilege) to override any system-default - behaviors by setting the fields that would otherwise have been defaulted. - 1. enables updates from users to be merged with changes made during late -initialization, using strategic merge patch, as opposed to clobbering the -change. - 1. allow the component which does the late-initialization to use strategic -merge patch, which facilitates composition and concurrency of such components. - -Although the apiserver Admission Control stage acts prior to object creation, -Admission Control plugins should follow the Late Initialization conventions -too, to allow their implementation to be later moved to a 'controller', or to -client libraries. - -## Concurrency Control and Consistency - -Kubernetes leverages the concept of *resource versions* to achieve optimistic -concurrency. All Kubernetes resources have a "resourceVersion" field as part of -their metadata. This resourceVersion is a string that identifies the internal -version of an object that can be used by clients to determine when objects have -changed. When a record is about to be updated, it's version is checked against a -pre-saved value, and if it doesn't match, the update fails with a StatusConflict -(HTTP status code 409). - -The resourceVersion is changed by the server every time an object is modified. -If resourceVersion is included with the PUT operation the system will verify -that there have not been other successful mutations to the resource during a -read/modify/write cycle, by verifying that the current value of resourceVersion -matches the specified value. - -The resourceVersion is currently backed by [etcd's -modifiedIndex](https://coreos.com/etcd/docs/latest/v2/api.html). -However, it's important to note that the application should *not* rely on the -implementation details of the versioning system maintained by Kubernetes. We may -change the implementation of resourceVersion in the future, such as to change it -to a timestamp or per-object counter. - -The only way for a client to know the expected value of resourceVersion is to -have received it from the server in response to a prior operation, typically a -GET. This value MUST be treated as opaque by clients and passed unmodified back -to the server. Clients should not assume that the resource version has meaning -across namespaces, different kinds of resources, or different servers. -Currently, the value of resourceVersion is set to match etcd's sequencer. You -could think of it as a logical clock the API server can use to order requests. -However, we expect the implementation of resourceVersion to change in the -future, such as in the case we shard the state by kind and/or namespace, or port -to another storage system. - -In the case of a conflict, the correct client action at this point is to GET the -resource again, apply the changes afresh, and try submitting again. This -mechanism can be used to prevent races like the following: - -``` -Client #1 Client #2 -GET Foo GET Foo -Set Foo.Bar = "one" Set Foo.Baz = "two" -PUT Foo PUT Foo -``` - -When these sequences occur in parallel, either the change to Foo.Bar or the -change to Foo.Baz can be lost. - -On the other hand, when specifying the resourceVersion, one of the PUTs will -fail, since whichever write succeeds changes the resourceVersion for Foo. - -resourceVersion may be used as a precondition for other operations (e.g., GET, -DELETE) in the future, such as for read-after-write consistency in the presence -of caching. - -"Watch" operations specify resourceVersion using a query parameter. It is used -to specify the point at which to begin watching the specified resources. This -may be used to ensure that no mutations are missed between a GET of a resource -(or list of resources) and a subsequent Watch, even if the current version of -the resource is more recent. This is currently the main reason that list -operations (GET on a collection) return resourceVersion. - - -## Serialization Format - -APIs may return alternative representations of any resource in response to an -Accept header or under alternative endpoints, but the default serialization for -input and output of API responses MUST be JSON. - -A protobuf encoding is also accepted for built-in resources. As proto is not -self-describing, there is an envelope wrapper which describes the type of -the contents. - -All dates should be serialized as RFC3339 strings. - -## Units - -Units must either be explicit in the field name (e.g., `timeoutSeconds`), or -must be specified as part of the value (e.g., `resource.Quantity`). Which -approach is preferred is TBD, though currently we use the `fooSeconds` -convention for durations. - -Duration fields must be represented as integer fields with units being -part of the field name (e.g. `leaseDurationSeconds`). We don't use Duration -in the API since that would require clients to implement go-compatible parsing. - -## Selecting Fields - -Some APIs may need to identify which field in a JSON object is invalid, or to -reference a value to extract from a separate resource. The current -recommendation is to use standard JavaScript syntax for accessing that field, -assuming the JSON object was transformed into a JavaScript object, without the -leading dot, such as `metadata.name`. - -Examples: - -* Find the field "current" in the object "state" in the second item in the array -"fields": `fields[1].state.current` - -## Object references - -Object references should either be called `fooName` if referring to an object of -kind `Foo` by just the name (within the current namespace, if a namespaced -resource), or should be called `fooRef`, and should contain a subset of the -fields of the `ObjectReference` type. - - -TODO: Plugins, extensions, nested kinds, headers - - -## HTTP Status codes - -The server will respond with HTTP status codes that match the HTTP spec. See the -section below for a breakdown of the types of status codes the server will send. - -The following HTTP status codes may be returned by the API. - -#### Success codes - -* `200 StatusOK` - * Indicates that the request completed successfully. -* `201 StatusCreated` - * Indicates that the request to create kind completed successfully. -* `204 StatusNoContent` - * Indicates that the request completed successfully, and the response contains -no body. - * Returned in response to HTTP OPTIONS requests. - -#### Error codes - -* `307 StatusTemporaryRedirect` - * Indicates that the address for the requested resource has changed. - * Suggested client recovery behavior: - * Follow the redirect. - - -* `400 StatusBadRequest` - * Indicates the requested is invalid. - * Suggested client recovery behavior: - * Do not retry. Fix the request. - - -* `401 StatusUnauthorized` - * Indicates that the server can be reached and understood the request, but -refuses to take any further action, because the client must provide -authorization. If the client has provided authorization, the server is -indicating the provided authorization is unsuitable or invalid. - * Suggested client recovery behavior: - * If the user has not supplied authorization information, prompt them for -the appropriate credentials. If the user has supplied authorization information, -inform them their credentials were rejected and optionally prompt them again. - - -* `403 StatusForbidden` - * Indicates that the server can be reached and understood the request, but -refuses to take any further action, because it is configured to deny access for -some reason to the requested resource by the client. - * Suggested client recovery behavior: - * Do not retry. Fix the request. - - -* `404 StatusNotFound` - * Indicates that the requested resource does not exist. - * Suggested client recovery behavior: - * Do not retry. Fix the request. - - -* `405 StatusMethodNotAllowed` - * Indicates that the action the client attempted to perform on the resource -was not supported by the code. - * Suggested client recovery behavior: - * Do not retry. Fix the request. - - -* `409 StatusConflict` - * Indicates that either the resource the client attempted to create already -exists or the requested update operation cannot be completed due to a conflict. - * Suggested client recovery behavior: - * * If creating a new resource: - * * Either change the identifier and try again, or GET and compare the -fields in the pre-existing object and issue a PUT/update to modify the existing -object. - * * If updating an existing resource: - * See `Conflict` from the `status` response section below on how to -retrieve more information about the nature of the conflict. - * GET and compare the fields in the pre-existing object, merge changes (if -still valid according to preconditions), and retry with the updated request -(including `ResourceVersion`). - - -* `410 StatusGone` - * Indicates that the item is no longer available at the server and no -forwarding address is known. - * Suggested client recovery behavior: - * Do not retry. Fix the request. - - -* `422 StatusUnprocessableEntity` - * Indicates that the requested create or update operation cannot be completed -due to invalid data provided as part of the request. - * Suggested client recovery behavior: - * Do not retry. Fix the request. - - -* `429 StatusTooManyRequests` - * Indicates that the either the client rate limit has been exceeded or the -server has received more requests then it can process. - * Suggested client recovery behavior: - * Read the `Retry-After` HTTP header from the response, and wait at least -that long before retrying. - - -* `500 StatusInternalServerError` - * Indicates that the server can be reached and understood the request, but -either an unexpected internal error occurred and the outcome of the call is -unknown, or the server cannot complete the action in a reasonable time (this may -be due to temporary server load or a transient communication issue with another -server). - * Suggested client recovery behavior: - * Retry with exponential backoff. - - -* `503 StatusServiceUnavailable` - * Indicates that required service is unavailable. - * Suggested client recovery behavior: - * Retry with exponential backoff. - - -* `504 StatusServerTimeout` - * Indicates that the request could not be completed within the given time. -Clients can get this response ONLY when they specified a timeout param in the -request. - * Suggested client recovery behavior: - * Increase the value of the timeout param and retry with exponential -backoff. - -## Response Status Kind - -Kubernetes will always return the `Status` kind from any API endpoint when an -error occurs. Clients SHOULD handle these types of objects when appropriate. - -A `Status` kind will be returned by the API in two cases: - * When an operation is not successful (i.e. when the server would return a non -2xx HTTP status code). - * When a HTTP `DELETE` call is successful. - -The status object is encoded as JSON and provided as the body of the response. -The status object contains fields for humans and machine consumers of the API to -get more detailed information for the cause of the failure. The information in -the status object supplements, but does not override, the HTTP status code's -meaning. When fields in the status object have the same meaning as generally -defined HTTP headers and that header is returned with the response, the header -should be considered as having higher priority. - -**Example:** - -```console -$ curl -v -k -H "Authorization: Bearer WhCDvq4VPpYhrcfmF6ei7V9qlbqTubUc" https://10.240.122.184:443/api/v1/namespaces/default/pods/grafana - -> GET /api/v1/namespaces/default/pods/grafana HTTP/1.1 -> User-Agent: curl/7.26.0 -> Host: 10.240.122.184 -> Accept: */* -> Authorization: Bearer WhCDvq4VPpYhrcfmF6ei7V9qlbqTubUc -> - -< HTTP/1.1 404 Not Found -< Content-Type: application/json -< Date: Wed, 20 May 2015 18:10:42 GMT -< Content-Length: 232 -< -{ - "kind": "Status", - "apiVersion": "v1", - "metadata": {}, - "status": "Failure", - "message": "pods \"grafana\" not found", - "reason": "NotFound", - "details": { - "name": "grafana", - "kind": "pods" - }, - "code": 404 -} -``` - -`status` field contains one of two possible values: -* `Success` -* `Failure` - -`message` may contain human-readable description of the error - -`reason` may contain a machine-readable, one-word, CamelCase description of why -this operation is in the `Failure` status. If this value is empty there is no -information available. The `reason` clarifies an HTTP status code but does not -override it. - -`details` may contain extended data associated with the reason. Each reason may -define its own extended details. This field is optional and the data returned is -not guaranteed to conform to any schema except that defined by the reason type. - -Possible values for the `reason` and `details` fields: -* `BadRequest` - * Indicates that the request itself was invalid, because the request doesn't -make any sense, for example deleting a read-only object. - * This is different than `status reason` `Invalid` above which indicates that -the API call could possibly succeed, but the data was invalid. - * API calls that return BadRequest can never succeed. - * Http status code: `400 StatusBadRequest` - - -* `Unauthorized` - * Indicates that the server can be reached and understood the request, but -refuses to take any further action without the client providing appropriate -authorization. If the client has provided authorization, this error indicates -the provided credentials are insufficient or invalid. - * Details (optional): - * `kind string` - * The kind attribute of the unauthorized resource (on some operations may -differ from the requested resource). - * `name string` - * The identifier of the unauthorized resource. - * HTTP status code: `401 StatusUnauthorized` - - -* `Forbidden` - * Indicates that the server can be reached and understood the request, but -refuses to take any further action, because it is configured to deny access for -some reason to the requested resource by the client. - * Details (optional): - * `kind string` - * The kind attribute of the forbidden resource (on some operations may -differ from the requested resource). - * `name string` - * The identifier of the forbidden resource. - * HTTP status code: `403 StatusForbidden` - - -* `NotFound` - * Indicates that one or more resources required for this operation could not -be found. - * Details (optional): - * `kind string` - * The kind attribute of the missing resource (on some operations may -differ from the requested resource). - * `name string` - * The identifier of the missing resource. - * HTTP status code: `404 StatusNotFound` - - -* `AlreadyExists` - * Indicates that the resource you are creating already exists. - * Details (optional): - * `kind string` - * The kind attribute of the conflicting resource. - * `name string` - * The identifier of the conflicting resource. - * HTTP status code: `409 StatusConflict` - -* `Conflict` - * Indicates that the requested update operation cannot be completed due to a -conflict. The client may need to alter the request. Each resource may define -custom details that indicate the nature of the conflict. - * HTTP status code: `409 StatusConflict` - - -* `Invalid` - * Indicates that the requested create or update operation cannot be completed -due to invalid data provided as part of the request. - * Details (optional): - * `kind string` - * the kind attribute of the invalid resource - * `name string` - * the identifier of the invalid resource - * `causes` - * One or more `StatusCause` entries indicating the data in the provided -resource that was invalid. The `reason`, `message`, and `field` attributes will -be set. - * HTTP status code: `422 StatusUnprocessableEntity` - - -* `Timeout` - * Indicates that the request could not be completed within the given time. -Clients may receive this response if the server has decided to rate limit the -client, or if the server is overloaded and cannot process the request at this -time. - * Http status code: `429 TooManyRequests` - * The server should set the `Retry-After` HTTP header and return -`retryAfterSeconds` in the details field of the object. A value of `0` is the -default. - - -* `ServerTimeout` - * Indicates that the server can be reached and understood the request, but -cannot complete the action in a reasonable time. This maybe due to temporary -server load or a transient communication issue with another server. - * Details (optional): - * `kind string` - * The kind attribute of the resource being acted on. - * `name string` - * The operation that is being attempted. - * The server should set the `Retry-After` HTTP header and return -`retryAfterSeconds` in the details field of the object. A value of `0` is the -default. - * Http status code: `504 StatusServerTimeout` - - -* `MethodNotAllowed` - * Indicates that the action the client attempted to perform on the resource -was not supported by the code. - * For instance, attempting to delete a resource that can only be created. - * API calls that return MethodNotAllowed can never succeed. - * Http status code: `405 StatusMethodNotAllowed` - - -* `InternalError` - * Indicates that an internal error occurred, it is unexpected and the outcome -of the call is unknown. - * Details (optional): - * `causes` - * The original error. - * Http status code: `500 StatusInternalServerError` `code` may contain the suggested HTTP return code for this status. - - -## Events - -Events are complementary to status information, since they can provide some -historical information about status and occurrences in addition to current or -previous status. Generate events for situations users or administrators should -be alerted about. - -Choose a unique, specific, short, CamelCase reason for each event category. For -example, `FreeDiskSpaceInvalid` is a good event reason because it is likely to -refer to just one situation, but `Started` is not a good reason because it -doesn't sufficiently indicate what started, even when combined with other event -fields. - -`Error creating foo` or `Error creating foo %s` would be appropriate for an -event message, with the latter being preferable, since it is more informational. - -Accumulate repeated events in the client, especially for frequent events, to -reduce data volume, load on the system, and noise exposed to users. - -## Naming conventions - -* Go field names must be CamelCase. JSON field names must be camelCase. Other -than capitalization of the initial letter, the two should almost always match. -No underscores nor dashes in either. -* Field and resource names should be declarative, not imperative (DoSomething, -SomethingDoer, DoneBy, DoneAt). -* Use `Node` where referring to -the node resource in the context of the cluster. Use `Host` where referring to -properties of the individual physical/virtual system, such as `hostname`, -`hostPath`, `hostNetwork`, etc. -* `FooController` is a deprecated kind naming convention. Name the kind after -the thing being controlled instead (e.g., `Job` rather than `JobController`). -* The name of a field that specifies the time at which `something` occurs should -be called `somethingTime`. Do not use `stamp` (e.g., `creationTimestamp`). -* We use the `fooSeconds` convention for durations, as discussed in the [units -subsection](#units). - * `fooPeriodSeconds` is preferred for periodic intervals and other waiting -periods (e.g., over `fooIntervalSeconds`). - * `fooTimeoutSeconds` is preferred for inactivity/unresponsiveness deadlines. - * `fooDeadlineSeconds` is preferred for activity completion deadlines. -* Do not use abbreviations in the API, except where they are extremely commonly -used, such as "id", "args", or "stdin". -* Acronyms should similarly only be used when extremely commonly known. All -letters in the acronym should have the same case, using the appropriate case for -the situation. For example, at the beginning of a field name, the acronym should -be all lowercase, such as "httpGet". Where used as a constant, all letters -should be uppercase, such as "TCP" or "UDP". -* The name of a field referring to another resource of kind `Foo` by name should -be called `fooName`. The name of a field referring to another resource of kind -`Foo` by ObjectReference (or subset thereof) should be called `fooRef`. -* More generally, include the units and/or type in the field name if they could -be ambiguous and they are not specified by the value or value type. -* The name of a field expressing a boolean property called 'fooable' should be -called `Fooable`, not `IsFooable`. - -### Namespace Names -* The name of a namespace must be a -[DNS_LABEL](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/identifiers.md). -* The `kube-` prefix is reserved for Kubernetes system namespaces, e.g. `kube-system` and `kube-public`. -* See -[the namespace docs](https://kubernetes.io/docs/user-guide/namespaces/) for more information. - -## Label, selector, and annotation conventions - -Labels are the domain of users. They are intended to facilitate organization and -management of API resources using attributes that are meaningful to users, as -opposed to meaningful to the system. Think of them as user-created mp3 or email -inbox labels, as opposed to the directory structure used by a program to store -its data. The former enables the user to apply an arbitrary ontology, whereas -the latter is implementation-centric and inflexible. Users will use labels to -select resources to operate on, display label values in CLI/UI columns, etc. -Users should always retain full power and flexibility over the label schemas -they apply to labels in their namespaces. - -However, we should support conveniences for common cases by default. For -example, what we now do in ReplicationController is automatically set the RC's -selector and labels to the labels in the pod template by default, if they are -not already set. That ensures that the selector will match the template, and -that the RC can be managed using the same labels as the pods it creates. Note -that once we generalize selectors, it won't necessarily be possible to -unambiguously generate labels that match an arbitrary selector. - -If the user wants to apply additional labels to the pods that it doesn't select -upon, such as to facilitate adoption of pods or in the expectation that some -label values will change, they can set the selector to a subset of the pod -labels. Similarly, the RC's labels could be initialized to a subset of the pod -template's labels, or could include additional/different labels. - -For disciplined users managing resources within their own namespaces, it's not -that hard to consistently apply schemas that ensure uniqueness. One just needs -to ensure that at least one value of some label key in common differs compared -to all other comparable resources. We could/should provide a verification tool -to check that. However, development of conventions similar to the examples in -[Labels](https://kubernetes.io/docs/user-guide/labels/) make uniqueness straightforward. Furthermore, -relatively narrowly used namespaces (e.g., per environment, per application) can -be used to reduce the set of resources that could potentially cause overlap. - -In cases where users could be running misc. examples with inconsistent schemas, -or where tooling or components need to programmatically generate new objects to -be selected, there needs to be a straightforward way to generate unique label -sets. A simple way to ensure uniqueness of the set is to ensure uniqueness of a -single label value, such as by using a resource name, uid, resource hash, or -generation number. - -Problems with uids and hashes, however, include that they have no semantic -meaning to the user, are not memorable nor readily recognizable, and are not -predictable. Lack of predictability obstructs use cases such as creation of a -replication controller from a pod, such as people want to do when exploring the -system, bootstrapping a self-hosted cluster, or deletion and re-creation of a -new RC that adopts the pods of the previous one, such as to rename it. -Generation numbers are more predictable and much clearer, assuming there is a -logical sequence. Fortunately, for deployments that's the case. For jobs, use of -creation timestamps is common internally. Users should always be able to turn -off auto-generation, in order to permit some of the scenarios described above. -Note that auto-generated labels will also become one more field that needs to be -stripped out when cloning a resource, within a namespace, in a new namespace, in -a new cluster, etc., and will need to be ignored around when updating a resource -via patch or read-modify-write sequence. - -Inclusion of a system prefix in a label key is fairly hostile to UX. A prefix is -only necessary in the case that the user cannot choose the label key, in order -to avoid collisions with user-defined labels. However, I firmly believe that the -user should always be allowed to select the label keys to use on their -resources, so it should always be possible to override default label keys. - -Therefore, resources supporting auto-generation of unique labels should have a -`uniqueLabelKey` field, so that the user could specify the key if they wanted -to, but if unspecified, it could be set by default, such as to the resource -type, like job, deployment, or replicationController. The value would need to be -at least spatially unique, and perhaps temporally unique in the case of job. - -Annotations have very different intended usage from labels. They are -primarily generated and consumed by tooling and system extensions, or are used -by end-users to engage non-standard behavior of components. For example, an -annotation might be used to indicate that an instance of a resource expects -additional handling by non-kubernetes controllers. Annotations may carry -arbitrary payloads, including JSON documents. Like labels, annotation keys can -be prefixed with a governing domain (e.g. `example.com/key-name`). Unprefixed -keys (e.g. `key-name`) are reserved for end-users. Third-party components must -use prefixed keys. Key prefixes under the "kubernetes.io" and "k8s.io" domains -are reserved for use by the kubernetes project and must not be used by -third-parties. - -In early versions of Kubernetes, some in-development features represented new -API fields as annotations, generally with the form `something.alpha.kubernetes.io/name` or -`something.beta.kubernetes.io/name` (depending on our confidence in it). This -pattern is deprecated. Some such annotations may still exist, but no new -annotations may be defined. New API fields are now developed as regular fields. - -Other advice regarding use of labels, annotations, taints, and other generic map keys by -Kubernetes components and tools: - - Key names should be all lowercase, with words separated by dashes instead of camelCase - - For instance, prefer `foo.kubernetes.io/foo-bar` over `foo.kubernetes.io/fooBar`, prefer - `desired-replicas` over `DesiredReplicas` - - Unprefixed keys are reserved for end-users. All other labels and annotations must be prefixed. - - Key prefixes under "kubernetes.io" and "k8s.io" are reserved for the Kubernetes - project. - - Such keys are effectively part of the kubernetes API and may be subject - to deprecation and compatibility policies. - - Key names, including prefixes, should be precise enough that a user could - plausibly understand where it came from and what it is for. - - Key prefixes should carry as much context as possible. - - For instance, prefer `subsystem.kubernetes.io/parameter` over `kubernetes.io/subsystem-parameter` - - Use annotations to store API extensions that the controller responsible for -the resource doesn't need to know about, experimental fields that aren't -intended to be generally used API fields, etc. Beware that annotations aren't -automatically handled by the API conversion machinery. - -## WebSockets and SPDY - -Some of the API operations exposed by Kubernetes involve transfer of binary -streams between the client and a container, including attach, exec, portforward, -and logging. The API therefore exposes certain operations over upgradeable HTTP -connections ([described in RFC 2817](https://tools.ietf.org/html/rfc2817)) via -the WebSocket and SPDY protocols. These actions are exposed as subresources with -their associated verbs (exec, log, attach, and portforward) and are requested -via a GET (to support JavaScript in a browser) and POST (semantically accurate). - -There are two primary protocols in use today: - -1. Streamed channels - - When dealing with multiple independent binary streams of data such as the -remote execution of a shell command (writing to STDIN, reading from STDOUT and -STDERR) or forwarding multiple ports the streams can be multiplexed onto a -single TCP connection. Kubernetes supports a SPDY based framing protocol that -leverages SPDY channels and a WebSocket framing protocol that multiplexes -multiple channels onto the same stream by prefixing each binary chunk with a -byte indicating its channel. The WebSocket protocol supports an optional -subprotocol that handles base64-encoded bytes from the client and returns -base64-encoded bytes from the server and character based channel prefixes ('0', -'1', '2') for ease of use from JavaScript in a browser. - -2. Streaming response - - The default log output for a channel of streaming data is an HTTP Chunked -Transfer-Encoding, which can return an arbitrary stream of binary data from the -server. Browser-based JavaScript is limited in its ability to access the raw -data from a chunked response, especially when very large amounts of logs are -returned, and in future API calls it may be desirable to transfer large files. -The streaming API endpoints support an optional WebSocket upgrade that provides -a unidirectional channel from the server to the client and chunks data as binary -WebSocket frames. An optional WebSocket subprotocol is exposed that base64 -encodes the stream before returning it to the client. - -Clients should use the SPDY protocols if their clients have native support, or -WebSockets as a fallback. Note that WebSockets is susceptible to Head-of-Line -blocking and so clients must read and process each message sequentially. In -the future, an HTTP/2 implementation will be exposed that deprecates SPDY. - - -## Validation - -API objects are validated upon receipt by the apiserver. Validation errors are -flagged and returned to the caller in a `Failure` status with `reason` set to -`Invalid`. In order to facilitate consistent error messages, we ask that -validation logic adheres to the following guidelines whenever possible (though -exceptional cases will exist). - -* Be as precise as possible. -* Telling users what they CAN do is more useful than telling them what they -CANNOT do. -* When asserting a requirement in the positive, use "must". Examples: "must be -greater than 0", "must match regex '[a-z]+'". Words like "should" imply that -the assertion is optional, and must be avoided. -* When asserting a formatting requirement in the negative, use "must not". -Example: "must not contain '..'". Words like "should not" imply that the -assertion is optional, and must be avoided. -* When asserting a behavioral requirement in the negative, use "may not". -Examples: "may not be specified when otherField is empty", "only `name` may be -specified". -* When referencing a literal string value, indicate the literal in -single-quotes. Example: "must not contain '..'". -* When referencing another field name, indicate the name in back-quotes. -Example: "must be greater than `request`". -* When specifying inequalities, use words rather than symbols. Examples: "must -be less than 256", "must be greater than or equal to 0". Do not use words -like "larger than", "bigger than", "more than", "higher than", etc. -* When specifying numeric ranges, use inclusive ranges when possible. +This file has moved to https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md. +This file is a placeholder to preserve links. Please remove by April 24, 2019 or the release of kubernetes 1.13, whichever comes first. \ No newline at end of file diff --git a/contributors/devel/api_changes.md b/contributors/devel/api_changes.md index bdbec9633..cab9130f0 100644 --- a/contributors/devel/api_changes.md +++ b/contributors/devel/api_changes.md @@ -1,1007 +1,3 @@ -*This document is oriented at developers who want to change existing APIs. -A set of API conventions, which applies to new APIs and to changes, can be -found at [API Conventions](api-conventions.md). +This file has moved to https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md. -**Table of Contents** - -- [So you want to change the API?](#so-you-want-to-change-the-api) - - [Operational overview](#operational-overview) - - [On compatibility](#on-compatibility) - - [Backward compatibility gotchas](#backward-compatibility-gotchas) - - [Incompatible API changes](#incompatible-api-changes) - - [Changing versioned APIs](#changing-versioned-apis) - - [Edit types.go](#edit-typesgo) - - [Edit defaults.go](#edit-defaultsgo) - - [Edit conversion.go](#edit-conversiongo) - - [Changing the internal structures](#changing-the-internal-structures) - - [Edit types.go](#edit-typesgo-1) - - [Edit validation.go](#edit-validationgo) - - [Edit version conversions](#edit-version-conversions) - - [Generate protobuf objects](#generate-protobuf-objects) - - [Edit json (un)marshaling code](#edit-json-unmarshaling-code) - - [Making a new API Version](#making-a-new-api-version) - - [Making a new API Group](#making-a-new-api-group) - - [Update the fuzzer](#update-the-fuzzer) - - [Update the semantic comparisons](#update-the-semantic-comparisons) - - [Implement your change](#implement-your-change) - - [Write end-to-end tests](#write-end-to-end-tests) - - [Examples and docs](#examples-and-docs) - - [Alpha, Beta, and Stable Versions](#alpha-beta-and-stable-versions) - - [Adding Unstable Features to Stable Versions](#adding-unstable-features-to-stable-versions) - - -# So you want to change the API? - -Before attempting a change to the API, you should familiarize yourself with a -number of existing API types and with the [API conventions](api-conventions.md). -If creating a new API type/resource, we also recommend that you first send a PR -containing just a proposal for the new API types. - -The Kubernetes API has two major components - the internal structures and -the versioned APIs. The versioned APIs are intended to be stable, while the -internal structures are implemented to best reflect the needs of the Kubernetes -code itself. - -What this means for API changes is that you have to be somewhat thoughtful in -how you approach changes, and that you have to touch a number of pieces to make -a complete change. This document aims to guide you through the process, though -not all API changes will need all of these steps. - -## Operational overview - -It is important to have a high level understanding of the API system used in -Kubernetes in order to navigate the rest of this document. - -As mentioned above, the internal representation of an API object is decoupled -from any one API version. This provides a lot of freedom to evolve the code, -but it requires robust infrastructure to convert between representations. There -are multiple steps in processing an API operation - even something as simple as -a GET involves a great deal of machinery. - -The conversion process is logically a "star" with the internal form at the -center. Every versioned API can be converted to the internal form (and -vice-versa), but versioned APIs do not convert to other versioned APIs directly. -This sounds like a heavy process, but in reality we do not intend to keep more -than a small number of versions alive at once. While all of the Kubernetes code -operates on the internal structures, they are always converted to a versioned -form before being written to storage (disk or etcd) or being sent over a wire. -Clients should consume and operate on the versioned APIs exclusively. - -To demonstrate the general process, here is a (hypothetical) example: - - 1. A user POSTs a `Pod` object to `/api/v7beta1/...` - 2. The JSON is unmarshalled into a `v7beta1.Pod` structure - 3. Default values are applied to the `v7beta1.Pod` - 4. The `v7beta1.Pod` is converted to an `api.Pod` structure - 5. The `api.Pod` is validated, and any errors are returned to the user - 6. The `api.Pod` is converted to a `v6.Pod` (because v6 is the latest stable -version) - 7. The `v6.Pod` is marshalled into JSON and written to etcd - -Now that we have the `Pod` object stored, a user can GET that object in any -supported api version. For example: - - 1. A user GETs the `Pod` from `/api/v5/...` - 2. The JSON is read from etcd and unmarshalled into a `v6.Pod` structure - 3. Default values are applied to the `v6.Pod` - 4. The `v6.Pod` is converted to an `api.Pod` structure - 5. The `api.Pod` is converted to a `v5.Pod` structure - 6. The `v5.Pod` is marshalled into JSON and sent to the user - -The implication of this process is that API changes must be done carefully and -backward-compatibly. - -## On compatibility - -Before talking about how to make API changes, it is worthwhile to clarify what -we mean by API compatibility. Kubernetes considers forwards and backwards -compatibility of its APIs a top priority. Compatibility is *hard*, especially -handling issues around rollback-safety. This is something every API change -must consider. - -An API change is considered compatible if it: - - * adds new functionality that is not required for correct behavior (e.g., -does not add a new required field) - * does not change existing semantics, including: - * the semantic meaning of default values *and behavior* - * interpretation of existing API types, fields, and values - * which fields are required and which are not - * mutable fields do not become immutable - * valid values do not become invalid - * explicitly invalid values do not become valid - -Put another way: - -1. Any API call (e.g. a structure POSTed to a REST endpoint) that succeeded -before your change must succeed after your change. -2. Any API call that does not use your change must behave the same as it did -before your change. -3. Any API call that uses your change must not cause problems (e.g. crash or -degrade behavior) when issued against an API servers that do not include your -change. -4. It must be possible to round-trip your change (convert to different API -versions and back) with no loss of information. -5. Existing clients need not be aware of your change in order for them to -continue to function as they did previously, even when your change is in use. -6. It must be possible to rollback to a previous version of API server that -does not include your change and have no impact on API objects which do not use -your change. API objects that use your change will be impacted in case of a -rollback. - -If your change does not meet these criteria, it is not considered compatible, -and may break older clients, or result in newer clients causing undefined -behavior. Such changes are generally disallowed, though exceptions have been -made in extreme cases (e.g. security or obvious bugs). - -Let's consider some examples. - -In a hypothetical API (assume we're at version v6), the `Frobber` struct looks -something like this: - -```go -// API v6. -type Frobber struct { - Height int `json:"height"` - Param string `json:"param"` -} -``` - -You want to add a new `Width` field. It is generally allowed to add new fields -without changing the API version, so you can simply change it to: - -```go -// Still API v6. -type Frobber struct { - Height int `json:"height"` - Width int `json:"width"` - Param string `json:"param"` -} -``` - -The onus is on you to define a sane default value for `Width` such that rules -#1 and #2 above are true - API calls and stored objects that used to work must -continue to work. - -For your next change you want to allow multiple `Param` values. You can not -simply remove `Param string` and add `Params []string` (without creating a -whole new API version) - that fails rules #1, #2, #3, and #6. Nor can you -simply add `Params []string` and use it instead - that fails #2 and #6. - -You must instead define a new field and the relationship between that field and -the existing field(s). Start by adding the new plural field: - -```go -// Still API v6. -type Frobber struct { - Height int `json:"height"` - Width int `json:"width"` - Param string `json:"param"` // the first param - Params []string `json:"params"` // all of the params -} -``` - -This new field must be inclusive of the singular field. In order to satisfy -the compatibility rules you must handle all the cases of version skew, multiple -clients, and rollbacks. This can be handled by defaulting or admission control -logic linking the fields together with context from the API operation to get as -close as possible to the user's intentions. - -Upon any mutating API operation: - * If only the singular field is specified (e.g. an older client), API logic - must populate plural[0] from the singular value, and de-dup the plural - field. - * If only the plural field is specified (e.g. a newer client), API logic must - populate the singular value from plural[0]. - * If both the singular and plural fields are specified, API logic must - validate that the singular value matches plural[0]. - * Any other case is an error and must be rejected. - -For this purpose "is specified" means the following: - * On a create or patch operation: the field is present in the user-provided input - * On an update operation: the field is present and has changed from the - current value - -Older clients that only know the singular field will continue to succeed and -produce the same results as before the change. Newer clients can use your -change without impacting older clients. The API server can be rolled back and -only objects that use your change will be impacted. - -Part of the reason for versioning APIs and for using internal types that are -distinct from any one version is to handle growth like this. The internal -representation can be implemented as: - -```go -// Internal, soon to be v7beta1. -type Frobber struct { - Height int - Width int - Params []string -} -``` - -The code that converts to/from versioned APIs can decode this into the -compatible structure. Eventually, a new API version, e.g. v7beta1, -will be forked and it can drop the singular field entirely. - -We've seen how to satisfy rules #1, #2, and #3. Rule #4 means that you can not -extend one versioned API without also extending the others. For example, an -API call might POST an object in API v7beta1 format, which uses the cleaner -`Params` field, but the API server might store that object in trusty old v6 -form (since v7beta1 is "beta"). When the user reads the object back in the -v7beta1 API it would be unacceptable to have lost all but `Params[0]`. This -means that, even though it is ugly, a compatible change must be made to the v6 -API, as above. - -For some changes, this can be challenging to do correctly. It may require multiple -representations of the same information in the same API resource, which need to -be kept in sync should either be changed. - -For example, let's say you decide to rename a field within the same API -version. In this case, you add units to `height` and `width`. You implement -this by adding new fields: - -```go -type Frobber struct { - Height *int `json:"height"` - Width *int `json:"width"` - HeightInInches *int `json:"heightInInches"` - WidthInInches *int `json:"widthInInches"` -} -``` - -You convert all of the fields to pointers in order to distinguish between unset -and set to 0, and then set each corresponding field from the other in the -defaulting logic (e.g. `heightInInches` from `height`, and vice versa). That -works fine when the user creates a sends a hand-written configuration -- -clients can write either field and read either field. - -But what about creation or update from the output of a GET, or update via PATCH -(see [In-place updates](https://kubernetes.io/docs/user-guide/managing-deployments/#in-place-updates-of-resources))? -In these cases, the two fields will conflict, because only one field would be -updated in the case of an old client that was only aware of the old field -(e.g. `height`). - -Suppose the client creates: - -```json -{ - "height": 10, - "width": 5 -} -``` - -and GETs: - -```json -{ - "height": 10, - "heightInInches": 10, - "width": 5, - "widthInInches": 5 -} -``` - -then PUTs back: - -```json -{ - "height": 13, - "heightInInches": 10, - "width": 5, - "widthInInches": 5 -} -``` - -As per the compatibility rules, the update must not fail, because it would have -worked before the change. - -## Backward compatibility gotchas - -* A single feature/property cannot be represented using multiple spec fields - simultaneously within an API version. Only one representation can be - populated at a time, and the client needs to be able to specify which field - they expect to use (typically via API version), on both mutation and read. As - above, older clients must continue to function properly. - -* A new representation, even in a new API version, that is more expressive than an - old one breaks backward compatibility, since clients that only understood the - old representation would not be aware of the new representation nor its - semantics. Examples of proposals that have run into this challenge include - [generalized label selectors](http://issues.k8s.io/341) and [pod-level security context](http://prs.k8s.io/12823). - -* Enumerated values cause similar challenges. Adding a new value to an enumerated set - is *not* a compatible change. Clients which assume they know how to handle all possible - values of a given field will not be able to handle the new values. However, removing a - value from an enumerated set *can* be a compatible change, if handled properly (treat the - removed value as deprecated but allowed). For enumeration-like fields that expect to add - new values in the future, such as `reason` fields, please document that expectation clearly - in the API field descriptions. Clients should treat such sets of values as potentially - open-ended. - -* For [Unions](api-conventions.md#unions), sets of fields where at most one should - be set, it is acceptable to add a new option to the union if the [appropriate - conventions](api-conventions.md#objects) were followed in the original object. - Removing an option requires following the [deprecation process](https://kubernetes.io/docs/reference/deprecation-policy/). - -* Changing any validation rules always has the potential of breaking some client, since it changes the - assumptions about part of the API, similar to adding new enum values. Validation rules on spec fields can - neither be relaxed nor strengthened. Strengthening cannot be permitted because any requests that previously - worked must continue to work. Weakening validation has the potential to break other consumers and generators - of the API resource. Status fields whose writers are under our control (e.g., written by non-pluggable - controllers), may potentially tighten validation, since that would cause a subset of previously valid - values to be observable by clients. - -* Do not add a new API version of an existing resource and make it the preferred version in the same - release, and do not make it the storage version. The latter is necessary so that a rollback of the - apiserver doesn't render resources in etcd undecodable after rollback. - -* Any field with a default value in one API version must have a *non-nil* default - value in all API versions. This can be split into 2 cases: - * Adding a new API version with a default value for an existing non-defaulted - field: it is required to add a default value semantically equivalent to - being unset in all previous API versions, to preserve the semantic meaning - of the value being unset. - * Adding a new field with a default value: the default values must be - semantically equivalent in all currently supported API versions. - -## Incompatible API changes - -There are times when incompatible changes might be OK, but mostly we want -changes that meet the above definitions. If you think you need to break -compatibility, you should talk to the Kubernetes API reviewers first. - -Breaking compatibility of a beta or stable API version, such as v1, is -unacceptable. Compatibility for experimental or alpha APIs is not strictly -required, but breaking compatibility should not be done lightly, as it disrupts -all users of the feature. Alpha and beta API versions may be deprecated and -eventually removed wholesale, as described in the [deprecation policy](https://kubernetes.io/docs/reference/deprecation-policy/). - -If your change is going to be backward incompatible or might be a breaking -change for API consumers, please send an announcement to -`kubernetes-dev@googlegroups.com` before the change gets in. If you are unsure, -ask. Also make sure that the change gets documented in the release notes for the -next release by labeling the PR with the "release-note-action-required" github label. - -If you found that your change accidentally broke clients, it should be reverted. - -In short, the expected API evolution is as follows: - -* `newapigroup/v1alpha1` -> ... -> `newapigroup/v1alphaN` -> -* `newapigroup/v1beta1` -> ... -> `newapigroup/v1betaN` -> -* `newapigroup/v1` -> -* `newapigroup/v2alpha1` -> ... - -While in alpha we expect to move forward with it, but may break it. - -Once in beta we will preserve forward compatibility, but may introduce new -versions and delete old ones. - -v1 must be backward-compatible for an extended length of time. - -## Changing versioned APIs - -For most changes, you will probably find it easiest to change the versioned -APIs first. This forces you to think about how to make your change in a -compatible way. Rather than doing each step in every version, it's usually -easier to do each versioned API one at a time, or to do all of one version -before starting "all the rest". - -### Edit types.go - -The struct definitions for each API are in -`staging/src/k8s.io/api///types.go`. Edit those files to reflect -the change you want to make. Note that all types and non-inline fields in -versioned APIs must be preceded by descriptive comments - these are used to -generate documentation. Comments for types should not contain the type name; API -documentation is generated from these comments and end-users should not be -exposed to golang type names. - -For types that need the generated -[DeepCopyObject](https://github.com/kubernetes/kubernetes/commit/8dd0989b395b29b872e1f5e06934721863e4a210#diff-6318847735efb6fae447e7dbf198c8b2R3767) -methods, usually only required by the top-level types like `Pod`, add this line -to the comment -([example](https://github.com/kubernetes/kubernetes/commit/39d95b9b065fffebe5b6f233d978fe1723722085#diff-ab819c2e7a94a3521aecf6b477f9b2a7R30)): - -```golang - // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object -``` - -Optional fields should have the `,omitempty` json tag; fields are interpreted as -being required otherwise. - -### Edit defaults.go - -If your change includes new fields for which you will need default values, you -need to add cases to `pkg/apis///defaults.go`. - -**Note:** When adding default values to new fields, you *must* also add default -values in all API versions, instead of leaving new fields unset (e.g. `nil`) in -old API versions. This is required because defaulting happens whenever a -serialized version is read (see [#66135]). When possible, pick meaningful values -as sentinels for unset values. - -In the past the core v1 API -was special. Its `defaults.go` used to live at `pkg/api/v1/defaults.go`. -If you see code referencing that path, you can be sure its outdated. Now the core v1 api lives at -`pkg/apis/core/v1/defaults.go` which follows the above convention. - -Of course, since you have added code, you have to add a test: -`pkg/apis///defaults_test.go`. - -Do use pointers to scalars when you need to distinguish between an unset value -and an automatic zero value. For example, -`PodSpec.TerminationGracePeriodSeconds` is defined as `*int64` the go type -definition. A zero value means 0 seconds, and a nil value asks the system to -pick a default. - -Don't forget to run the tests! - -[#66135]: https://github.com/kubernetes/kubernetes/issues/66135 - -### Edit conversion.go - -Given that you have not yet changed the internal structs, this might feel -premature, and that's because it is. You don't yet have anything to convert to -or from. We will revisit this in the "internal" section. If you're doing this -all in a different order (i.e. you started with the internal structs), then you -should jump to that topic below. In the very rare case that you are making an -incompatible change you might or might not want to do this now, but you will -have to do more later. The files you want are -`pkg/apis///conversion.go` and -`pkg/apis///conversion_test.go`. - -Note that the conversion machinery doesn't generically handle conversion of -values, such as various kinds of field references and API constants. [The client -library](https://github.com/kubernetes/client-go/blob/v4.0.0-beta.0/rest/request.go#L352) -has custom conversion code for field references. You also need to add a call to -`AddFieldLabelConversionFunc` of your scheme with a mapping function that -understands supported translations, like this -[line](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/api/v1/conversion.go#L165). - -## Changing the internal structures - -Now it is time to change the internal structs so your versioned changes can be -used. - -### Edit types.go - -Similar to the versioned APIs, the definitions for the internal structs are in -`pkg/apis//types.go`. Edit those files to reflect the change you want to -make. Keep in mind that the internal structs must be able to express *all* of -the versioned APIs. - -Similar to the versioned APIs, you need to add the `+k8s:deepcopy-gen` tag to -types that need generated DeepCopyObject methods. - -## Edit validation.go - -Most changes made to the internal structs need some form of input validation. -Validation is currently done on internal objects in -`pkg/apis//validation/validation.go`. This validation is the one of the -first opportunities we have to make a great user experience - good error -messages and thorough validation help ensure that users are giving you what you -expect and, when they don't, that they know why and how to fix it. Think hard -about the contents of `string` fields, the bounds of `int` fields and the -optionality of fields. - -Of course, code needs tests - `pkg/apis//validation/validation_test.go`. - -## Edit version conversions - -At this point you have both the versioned API changes and the internal -structure changes done. If there are any notable differences - field names, -types, structural change in particular - you must add some logic to convert -versioned APIs to and from the internal representation. If you see errors from -the `serialization_test`, it may indicate the need for explicit conversions. - -Performance of conversions very heavily influence performance of apiserver. -Thus, we are auto-generating conversion functions that are much more efficient -than the generic ones (which are based on reflections and thus are highly -inefficient). - -The conversion code resides with each versioned API. There are two files: - - - `pkg/apis///conversion.go` containing manually written - conversion functions - - `pkg/apis///zz_generated.conversion.go` containing - auto-generated conversion functions - -Since auto-generated conversion functions are using manually written ones, -those manually written should be named with a defined convention, i.e. a -function converting type `X` in pkg `a` to type `Y` in pkg `b`, should be named: -`convert_a_X_To_b_Y`. - -Also note that you can (and for efficiency reasons should) use auto-generated -conversion functions when writing your conversion functions. - -Adding manually written conversion also requires you to add tests to -`pkg/apis///conversion_test.go`. - -Once all the necessary manually written conversions are added, you need to -regenerate auto-generated ones. To regenerate them run: - -```sh -make clean && make generated_files -``` - -`make clean` is important, otherwise the generated files might be stale, because -the build system uses custom cache. - -`make all` will invoke `make generated_files` as well. - -The `make generated_files` will also regenerate the `zz_generated.deepcopy.go`, -`zz_generated.defaults.go`, and `api/openapi-spec/swagger.json`. - -If regeneration is somehow not possible due to compile errors, the easiest -workaround is to remove the files causing errors and rerun the command. - -## Generate Code - -Apart from the `defaulter-gen`, `deepcopy-gen`, `conversion-gen` and -`openapi-gen`, there are a few other generators: - - `go-to-protobuf` - - `client-gen` - - `lister-gen` - - `informer-gen` - - `codecgen` (for fast json serialization with ugorji codec) - -Many of the generators are based on -[`gengo`](https://github.com/kubernetes/gengo) and share common -flags. The `--verify-only` flag will check the existing files on disk -and fail if they are not what would have been generated. - -The generators that create go code have a `--go-header-file` flag -which should be a file that contains the header that should be -included. This header is the copyright that should be present at the -top of the generated file and should be checked with the -[`repo-infra/verify/verify-boilerplane.sh`](https://git.k8s.io/repo-infra/verify/verify-boilerplate.sh) -script at a later stage of the build. - -To invoke these generators, you can run `make update`, which runs a bunch of -[scripts](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/hack/update-all.sh#L63-L78). -Please continue to read the next a few sections, because some generators have -prerequisites, also because they introduce how to invoke the generators -individually if you find `make update` takes too long to run. - -### Generate protobuf objects - -For any core API object, we also need to generate the Protobuf IDL and marshallers. -That generation is invoked with - -```sh -hack/update-generated-protobuf.sh -``` - -The vast majority of objects will not need any consideration when converting -to protobuf, but be aware that if you depend on a Golang type in the standard -library there may be additional work required, although in practice we typically -use our own equivalents for JSON serialization. The `pkg/api/serialization_test.go` -will verify that your protobuf serialization preserves all fields - be sure to -run it several times to ensure there are no incompletely calculated fields. - -### Generate Clientset - -`client-gen` is a tool to generate clientsets for top-level API objects. - -`client-gen` requires the `// +genclient` annotation on each -exported type in both the internal `pkg/apis//types.go` as well as each -specifically versioned `staging/src/k8s.io/api///types.go`. - -If the apiserver hosts your API under a different group name than the `` -in the filesystem, (usually this is because the `` in the filesystem -omits the "k8s.io" suffix, e.g., admission vs. admission.k8s.io), you can -instruct the `client-gen` to use the correct group name by adding the `// -+groupName=` annotation in the `doc.go` in both the internal -`pkg/apis//doc.go` as well as in each specifically versioned -`staging/src/k8s.io/api///types.go`. - -Once you added the annotations, generate the client with - -```sh -hack/update-codegen.sh -``` - -Note that you can use the optional `// +groupGoName=` to specify a CamelCase -custom Golang identifier to de-conflict e.g. `policy.authorization.k8s.io` and -`policy.k8s.io`. These two would both map to `Policy()` in clientsets. - -client-gen is flexible. See [this document](generating-clientset.md) if you need -client-gen for non-kubernetes API. - -### Generate Listers - -`lister-gen` is a tool to generate listers for a client. It reuses the -`//+genclient` and the `// +groupName=` annotations, so you do not need to -specify extra annotations. - -Your previous run of `hack/update-codegen.sh` has invoked `lister-gen`. - -### Generate Informers - -`informer-gen` generates the very useful Informers which watch API -resources for changes. It reuses the `//+genclient` and the -`//+groupName=` annotations, so you do not need to specify extra annotations. - -Your previous run of `hack/update-codegen.sh` has invoked `informer-gen`. - -### Edit json (un)marshaling code - -We are auto-generating code for marshaling and unmarshaling json representation -of api objects - this is to improve the overall system performance. - -The auto-generated code resides with each versioned API: - - - `staging/src/k8s.io/api///generated.proto` - - `staging/src/k8s.io/api///generated.pb.go` - -To regenerate them run: - -```sh -hack/update-generated-protobuf.sh -``` - -## Making a new API Version - -This section is under construction, as we make the tooling completely generic. - -If you are adding a new API version to an existing group, you can copy the -structure of the existing `pkg/apis//` and -`staging/src/k8s.io/api//` directories. - -Due to the fast changing nature of the project, the following content is probably out-dated: -* You can control if the version is enabled by default by update -[pkg/master/master.go](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/master/master.go#L381). -* You must add the new version to - [pkg/apis/group_name/install/install.go](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/apps/install/install.go). -* You must add the new version to - [hack/lib/init.sh#KUBE_AVAILABLE_GROUP_VERSIONS](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/hack/lib/init.sh#L53). -* You must add the new version to - [hack/update-generated-protobuf-dockerized.sh](https://github.com/kubernetes/kubernetes/blob/v1.8.2/hack/update-generated-protobuf-dockerized.sh#L44) - to generate protobuf IDL and marshallers. -* You must add the new version to - [cmd/kube-apiserver/app#apiVersionPriorities](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/cmd/kube-apiserver/app/aggregator.go#L172) -* You must setup storage for the new version in - [pkg/registry/group_name/rest](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/registry/authentication/rest/storage_authentication.go) - -You need to regenerate the generated code as instructed in the sections above. - -## Making a new API Group - -You'll have to make a new directory under `pkg/apis/` and -`staging/src/k8s.io/api`; copy the directory structure of an existing API group, -e.g. `pkg/apis/authentication` and `staging/src/k8s.io/api/authentication`; -replace "authentication" with your group name and replace versions with your -versions; replace the API kinds in -[versioned](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/staging/src/k8s.io/api/authentication/v1/register.go#L47) -and -[internal](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/authentication/register.go#L47) -register.go, and -[install.go](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/authentication/install/install.go#L43) -with your kinds. - -You'll have to add your API group/version to a few places in the code base, as -noted in [Making a new API Version](#making-a-new-api-version) section. - -You need to regenerate the generated code as instructed in the sections above. - -## Update the fuzzer - -Part of our testing regimen for APIs is to "fuzz" (fill with random values) API -objects and then convert them to and from the different API versions. This is -a great way of exposing places where you lost information or made bad -assumptions. If you have added any fields which need very careful formatting -(the test does not run validation) or if you have made assumptions such as -"this slice will always have at least 1 element", you may get an error or even -a panic from the `serialization_test`. If so, look at the diff it produces (or -the backtrace in case of a panic) and figure out what you forgot. Encode that -into the fuzzer's custom fuzz functions. Hint: if you added defaults for a -field, that field will need to have a custom fuzz function that ensures that the -field is fuzzed to a non-empty value. - -The fuzzer can be found in `pkg/api/testing/fuzzer.go`. - -## Update the semantic comparisons - -VERY VERY rarely is this needed, but when it hits, it hurts. In some rare cases -we end up with objects (e.g. resource quantities) that have morally equivalent -values with different bitwise representations (e.g. value 10 with a base-2 -formatter is the same as value 0 with a base-10 formatter). The only way Go -knows how to do deep-equality is through field-by-field bitwise comparisons. -This is a problem for us. - -The first thing you should do is try not to do that. If you really can't avoid -this, I'd like to introduce you to our `apiequality.Semantic.DeepEqual` routine. -It supports custom overrides for specific types - you can find that in -`pkg/api/helper/helpers.go`. - -There's one other time when you might have to touch this: `unexported fields`. -You see, while Go's `reflect` package is allowed to touch `unexported fields`, -us mere mortals are not - this includes `apiequality.Semantic.DeepEqual`. -Fortunately, most of our API objects are "dumb structs" all the way down - all -fields are exported (start with a capital letter) and there are no unexported -fields. But sometimes you want to include an object in our API that does have -unexported fields somewhere in it (for example, `time.Time` has unexported fields). -If this hits you, you may have to touch the `apiequality.Semantic.DeepEqual` -customization functions. - -## Implement your change - -Now you have the API all changed - go implement whatever it is that you're -doing! - -## Write end-to-end tests - -Check out the [E2E docs](/contributors/devel/sig-testing/e2e-tests.md) for detailed information about how to -write end-to-end tests for your feature. - -## Examples and docs - -At last, your change is done, all unit tests pass, e2e passes, you're done, -right? Actually, no. You just changed the API. If you are touching an existing -facet of the API, you have to try *really* hard to make sure that *all* the -examples and docs are updated. There's no easy way to do this, due in part to -JSON and YAML silently dropping unknown fields. You're clever - you'll figure it -out. Put `grep` or `ack` to good use. - -If you added functionality, you should consider documenting it and/or writing -an example to illustrate your change. - -Make sure you update the swagger and OpenAPI spec by running: - -```sh -hack/update-swagger-spec.sh -hack/update-openapi-spec.sh -``` - -The API spec changes should be in a commit separate from your other changes. - -## Alpha, Beta, and Stable Versions - -New feature development proceeds through a series of stages of increasing -maturity: - -- Development level - - Object Versioning: no convention - - Availability: not committed to main kubernetes repo, and thus not available -in official releases - - Audience: other developers closely collaborating on a feature or -proof-of-concept - - Upgradeability, Reliability, Completeness, and Support: no requirements or -guarantees -- Alpha level - - Object Versioning: API version name contains `alpha` (e.g. `v1alpha1`) - - Availability: committed to main kubernetes repo; appears in an official -release; feature is disabled by default, but may be enabled by flag - - Audience: developers and expert users interested in giving early feedback on -features - - Completeness: some API operations, CLI commands, or UI support may not be -implemented; the API need not have had an *API review* (an intensive and -targeted review of the API, on top of a normal code review) - - Upgradeability: the object schema and semantics may change in a later -software release, without any provision for preserving objects in an existing -cluster; removing the upgradability concern allows developers to make rapid -progress; in particular, API versions can increment faster than the minor -release cadence and the developer need not maintain multiple versions; -developers should still increment the API version when object schema or -semantics change in an [incompatible way](#on-compatibility) - - Cluster Reliability: because the feature is relatively new, and may lack -complete end-to-end tests, enabling the feature via a flag might expose bugs -with destabilize the cluster (e.g. a bug in a control loop might rapidly create -excessive numbers of object, exhausting API storage). - - Support: there is *no commitment* from the project to complete the feature; -the feature may be dropped entirely in a later software release - - Recommended Use Cases: only in short-lived testing clusters, due to -complexity of upgradeability and lack of long-term support and lack of -upgradability. -- Beta level: - - Object Versioning: API version name contains `beta` (e.g. `v2beta3`) - - Availability: in official Kubernetes releases, and enabled by default - - Audience: users interested in providing feedback on features - - Completeness: all API operations, CLI commands, and UI support should be -implemented; end-to-end tests complete; the API has had a thorough API review -and is thought to be complete, though use during beta may frequently turn up API -issues not thought of during review - - Upgradeability: the object schema and semantics may change in a later -software release; when this happens, an upgrade path will be documented; in some -cases, objects will be automatically converted to the new version; in other -cases, a manual upgrade may be necessary; a manual upgrade may require downtime -for anything relying on the new feature, and may require manual conversion of -objects to the new version; when manual conversion is necessary, the project -will provide documentation on the process - - Cluster Reliability: since the feature has e2e tests, enabling the feature -via a flag should not create new bugs in unrelated features; because the feature -is new, it may have minor bugs - - Support: the project commits to complete the feature, in some form, in a -subsequent Stable version; typically this will happen within 3 months, but -sometimes longer; releases should simultaneously support two consecutive -versions (e.g. `v1beta1` and `v1beta2`; or `v1beta2` and `v1`) for at least one -minor release cycle (typically 3 months) so that users have enough time to -upgrade and migrate objects - - Recommended Use Cases: in short-lived testing clusters; in production -clusters as part of a short-lived evaluation of the feature in order to provide -feedback -- Stable level: - - Object Versioning: API version `vX` where `X` is an integer (e.g. `v1`) - - Availability: in official Kubernetes releases, and enabled by default - - Audience: all users - - Completeness: must have conformance tests, approved by SIG Architecture, -in the appropriate conformance profile (e.g., non-portable and/or optional -features may not be in the default profile) - - Upgradeability: only [strictly compatible](#on-compatibility) changes -allowed in subsequent software releases - - Cluster Reliability: high - - Support: API version will continue to be present for many subsequent -software releases; - - Recommended Use Cases: any - -### Adding Unstable Features to Stable Versions - -When adding a feature to an object which is already Stable, the new fields and -new behaviors need to meet the Stable level requirements. If these cannot be -met, then the new field cannot be added to the object. - -For example, consider the following object: - -```go -// API v6. -type Frobber struct { - // height ... - Height *int32 `json:"height" - // param ... - Param string `json:"param" -} -``` - -A developer is considering adding a new `Width` parameter, like this: - -```go -// API v6. -type Frobber struct { - // height ... - Height *int32 `json:"height" - // param ... - Param string `json:"param" - // width ... - Width *int32 `json:"width,omitempty" -} -``` - -However, the new feature is not stable enough to be used in a stable version -(`v6`). Some reasons for this might include: - -- the final representation is undecided (e.g. should it be called `Width` or `Breadth`?) -- the implementation is not stable enough for general use (e.g. the `Area()` routine sometimes overflows.) - -The developer cannot add the new field unconditionally until stability is met. However, -sometimes stability cannot be met until some users try the new feature, and some -users are only able or willing to accept a released version of Kubernetes. In -that case, the developer has a few options, both of which require staging work -over several releases. - -#### Alpha field in existing API version - -Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons: - -* They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819) -* They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in [backward compatibility gotchas](#backward-compatibility-gotchas)) - -The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default: - -1. Add a feature gate to the API server to control enablement of the new field (and associated function): - - In [staging/src/k8s.io/apiserver/pkg/features/kube_features.go](https://git.k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/features/kube_features.go): - - ```go - // owner: @you - // alpha: v1.11 - // - // Add multiple dimensions to frobbers. - Frobber2D utilfeature.Feature = "Frobber2D" - - var defaultKubernetesFeatureGates = map[utilfeature.Feature]utilfeature.FeatureSpec{ - ... - Frobber2D: {Default: false, PreRelease: utilfeature.Alpha}, - } - ``` - -2. Add the field to the API type: - - * ensure the field is [optional](api-conventions.md#optional-vs-required) - * add the `omitempty` struct tag - * add the `// +optional` comment tag - * ensure the field is entirely absent from API responses when empty (optional fields should be pointers, anyway) - * include details about the alpha-level in the field description - - ```go - // API v6. - type Frobber struct { - // height ... - Height int32 `json:"height"` - // param ... - Param string `json:"param"` - // width indicates how wide the object is. - // This field is alpha-level and is only honored by servers that enable the Frobber2D feature. - // +optional - Width *int32 `json:"width,omitempty"` - } - ``` - -3. Before persisting the object to storage, clear disabled alpha fields on create, -and on update if the existing object does not already have a value in the field. -This prevents new usage of the feature while it is disabled, while ensuring existing data is preserved. -The recommended place to do this is in the REST storage strategy's PrepareForCreate/PrepareForUpdate methods: - - ```go - func (frobberStrategy) PrepareForCreate(ctx genericapirequest.Context, obj runtime.Object) { - frobber := obj.(*api.Frobber) - - if !utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) { - frobber.Width = nil - } - } - - func (frobberStrategy) PrepareForUpdate(ctx genericapirequest.Context, obj, old runtime.Object) { - newFrobber := obj.(*api.Frobber) - oldFrobber := old.(*api.Frobber) - - if !utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) && oldFrobber.Width == nil { - newFrobber.Width = nil - } - } - ``` - -4. In validation, validate the field if present: - - ```go - func ValidateFrobber(f *api.Frobber, fldPath *field.Path) field.ErrorList { - ... - if f.Width != nil { - ... validation of width field ... - } - ... - } - ``` - -In future Kubernetes versions: - -* if the feature progresses to beta or stable status, the feature gate can be removed or be enabled by default. -* if the schema of the alpha field must change in an incompatible way, a new field name must be used. -* if the feature is abandoned, or the field name is changed, the field should be removed from the go struct, with a tombstone comment ensuring the field name and protobuf tag are not reused: - - ```go - // API v6. - type Frobber struct { - // height ... - Height int32 `json:"height" protobuf:"varint,1,opt,name=height"` - // param ... - Param string `json:"param" protobuf:"bytes,2,opt,name=param"` - - // +k8s:deprecated=width,protobuf=3 - } - ``` - -#### New alpha API version - -Another option is to introduce a new type with an new `alpha` or `beta` version -designator, like this: - -```go -// API v7alpha1 -type Frobber struct { - // height ... - Height *int32 `json:"height"` - // param ... - Param string `json:"param"` - // width ... - Width *int32 `json:"width,omitempty"` -} -``` - -The latter requires that all objects in the same API group as `Frobber` to be -replicated in the new version, `v7alpha1`. This also requires user to use a new -client which uses the other version. Therefore, this is not a preferred option. - -A related issue is how a cluster manager can roll back from a new version -with a new feature, that is already being used by users. See -https://github.com/kubernetes/kubernetes/issues/4855. +This file is a placeholder to preserve links. Please remove by April 24, 2019 or the release of kubernetes 1.13, whichever comes first. diff --git a/contributors/devel/component-config-conventions.md b/contributors/devel/component-config-conventions.md index d3fe10003..5754fb980 100644 --- a/contributors/devel/component-config-conventions.md +++ b/contributors/devel/component-config-conventions.md @@ -1,221 +1,3 @@ -# Component Configuration Conventions +This file has moved to https://git.k8s.io/community/contributors/devel/sig-architecture/component-config-conventions.md. -# Objective - -This document concerns the configuration of Kubernetes system components (as -opposed to the configuration of user workloads running on Kubernetes). -Component configuration is a major operational burden for operators of -Kubernetes clusters. To date, much literature has been written on and much -effort expended to improve component configuration. Despite this, the state of -component configuration remains dissonant. This document attempts to aggregate -that literature and propose a set of guidelines that component owners can -follow to improve consistency across the project. - -# Background - -Currently, component configuration is primarily driven through command line -flags. Command line driven configuration poses certain problems which are -discussed below. Attempts to improve component configuration as a whole have -been slow to make progress and have petered out (ref componentconfig api group, -configmap driven config issues). Some component owners have made use case -specific improvements on a per-need basis. Various comments in issues recommend -subsets of best design practice but no coherent, complete story exists. - -## Pain Points of Current Configuration - -Flag based configuration has poor qualities such as: - -1. Flags exist in a flat namespace, hampering the ability to organize them and expose them in helpful documentation. --help becomes useless as a reference as the number of knobs grows. It's impossible to distinguish useful knobs from cruft. -1. Flags can't easily have different values for different instances of a class. To adjust the resync period in the informers of O(n) controllers requires O(n) different flags in a global namespace. -1. Changing a process's command line necessitates a binary restart. This negatively impacts availability. -1. Flags are unsuitable for passing confidential configuration. The command line of a process is available to unprivileged process running in the host pid namespace. -1. Flags are a public API but are unversioned and unversionable. -1. Many arguments against using global variables apply to flags. - -Configuration in general has poor qualities such as: - -1. Configuration changes have the same forward/backward compatibility requirements as releases but rollout/rollback of configuration largely untested. Examples of configuration changes that might break a cluster: kubelet CNI plugin, etcd storage version. -1. Configuration options often exist only to test a specific feature where the default is reasonable for all real use cases. Examples: many sync periods. -1. Configuration options often exist to defer a "hard" design decision and to pay forward the "TODO(someone-else): think critically". -1. Configuration options are often used to workaround deficiencies of the API. For example `--register-with-labels` and `--register-with-taints` could be solved with a node initializer, if initializers existed. -1. Configuration options often exist to take testing shortcuts. There is a mentality that because a feature is opt-in, it can be released as a flag without robust testing. -1. Configuration accumulates new knobs, knobs accumulate new behaviors, knobs are forgotten and bitrot reducing code quality over time. -1. Number of configuration options is inversely proportional to test coverage. The size of the configuration state space grows >O(2^n) with the number of configuration bits. A handful of states in that space are ever tested. -1. Configuration options hamper troubleshooting efforts. On github, users frequently file tickets from environments that are neither consistent nor reproducible. - -## Types Of Configuration - -Configuration can only come from three sources: - -1. Command line flags. -1. API types serialized and stored on disk. -1. API types serialized and stored in the kubernetes API. - -Configuration options can be partitioned along certain lines. To name a few -important partitions: - -1. Bootstrap: This is configuration that is required before the component can contact the API. Examples include the kubeconfig and the filepath to the kubeconfig. -1. Dynamic vs Static: Dynamic config is config that is expected to change as part of normal operations such as a scheduler configuration or a node entering maintenance mode. Static config is config that is unlikely to change over subsequent deployments and even releases of a component. -1. Shared vs Per-Instance: Per-Instance configuration is configuration whose value is unique to the instance that the node runs on (e.g. Kubelet's `--hostname-override`). -1. Feature Gates: Feature gates are configuration options that enable a feature that has been deemed unsafe to enable by default. -1. Request context dependent: Request context dependent config is config that should probably be scoped to an attribute of the request (such as the user). We do a pretty good job of keeping these out of config and in policy objects (e.g. Quota, RBAC) but we could do more (e.g. rate limits). -1. Environment information: This is configuration that is available through downwards and OS APIs, e.g. node name, pod name, number of cpus, IP address. - -# Requirements - -Desired qualities of a configuration solution: - -1. Secure: We need to control who can change configuration. We need to control who can read sensitive configuration. -1. Manageable: We need to control which instances of a component uses which configuration, especially when those instances differ in version. -1. Reliable: Configuration pushes should just work. If they fail, they should fail early in the rollout, rollback config if possible, and alert noisily. -1. Recoverable: We need to be able to update (e.g. rollback) configuration when a component is down. -1. Monitorable: Both humans and computers need to monitor configuration; humans through json interfaces like /configz, computers through interfaces like prometheus /streamz. Confidential configuration needs to be accounted for, but can also be useful to monitor in an unredacted or partially redacted (i.e. hashed) form. -1. Verifiable: We need to be able to verify that a configuration is good. We need to verify the integrity of the received configuration and we need to validate that the encoded configuration state is sensible. -1. Auditable: We need to be able to trace the origin of a configuration change. -1. Accountable: We need to correlate a configuration push with its impact to the system. We need to be able to do this at the time of the push and later when analyzing logs. -1. Available: We should avoid high frequency configuration updates that require service disruption. We need to take into account system component SLA. -1. Scalable: We need to support distributing configuration to O(10,000) components at our current supported scalability limits. -1. Consistent: There should exist conventions that hold across components. -1. Composable: We should favor composition of configuration sources over layering/templating/inheritance. -1. Normalized: Redundant specification of configuration data should be avoided. -1. Testable: We need to be able to test the system under many different configurations. We also need to test configuration changes, both dynamic changes and those that require process restarts. -1. Maintainable: We need to push back on ever increasing cyclomatic complexity in our codebase. Each if statement and function argument added to support a configuration option negatively impacts the maintainability of our code. -1. Evolvable: We need to be able to extend our configuration API like we extend our other user facing APIs. We need to hold our configuration API to the same SLA and deprecation policy of public facing APIs. (e.g. [dynamic admission control](https://github.com/kubernetes/community/pull/611) and [hooks](https://github.com/kubernetes/kubernetes/issues/3585)) - -These don't need to be implemented immediately but are good to keep in mind. At -some point these should be ranked by priority and implemented. - -# Two Part Solution: - -## Part 1: Don't Make It Configuration - -The most effective way to reduce the operational burden of configuration is to -minimize the amount of configuration. When adding a configuration option, ask -whether alternatives might be a better fit. - -1. Policy objects: Create first class Kubernetes objects to encompass how the system should behave. These are especially useful for request context dependent configuration. We do this already in places such as RBAC and ResourceQuota but we could do more such as rate limiting. We should never hardcode groups or usermaps in configuration. -1. API features: Use (or implement) functionality of the API (e.g. think through and implement initializers instead of --register-with-label). Allowing for extension in the right places is a better way to give users control. -1. Feature discovery: Write components that introspect the existing API to decide whether to enable a feature or not. E.g. controller-manager should start an app controller if the app API is available, kubelet should enable zram if zram is set in the node spec. -1. Downwards API: Use the APIs that the OS and pod environment expose directly before opting to pass in new configuration options. -1. const's: If you don't know whether tweaking a value will be useful, make the value const. Only give it a configuration option once there becomes a need to tweak the value at runtime. -1. Autotuning: Build systems that incorporate feedback and do the best thing under the given circumstances. This makes the system more robust. (e.g. prefer congestion control, load shedding, backoff rather than explicit limiting). -1. Avoid feature flags: Turn on features when they are tested and ready for production. Don't use feature flags as a fallback for poorly tested code. -1. Configuration profiles: Instead of allowing individual configuration options to be modified, try to encompass a broader desire as a configuration profile. For example: instead of enabling individual alpha features, have an EnableAlpha option that enables all. Instead of allowing individual controller knobs to be modified, have a TestMode option that sets a broad number of parameters to be suitable for tests. - -## Part 2: Component Configuration - -### Versioning Configuration - -We create configuration API groups per component that live in the source tree of -the component. Each component has its own API group for configuration. -Components will use the same API machinery that we use for other API groups. -Configuration API serialization doesn't have the same performance requirements -as other APIs so much of the codegen can be avoided (e.g. ugorji, generated -conversions) and we can instead fallback to the reflection based implementations -where they exist. - -Configuration API groups for component config should be named according to the -scheme `.config.k8s.io`. The `.config.k8s.io` suffix serves to -disambiguate types of config API groups from served APIs. - -### Retrieving Configuration - -The primary mechanism for retrieving static configuration should be -deserialization from files. For the majority of components (with the possible -exception of the kubelet, see -[here](https://github.com/kubernetes/kubernetes/pull/29459)), these files will -be source from the configmap API and managed by the kubelet. Reliability of -this mechanism is predicated on kubelet checkpointing of pod dependencies. - - -### Structuring Configuration - -Group related options into distinct objects and subobjects. Instead of writing: - - -```yaml -kind: KubeProxyConfiguration -apiVersion: kubeproxy.config.k8s.io/v1beta3 -ipTablesSyncPeriod: 2 -ipTablesConntrackHashSize: 2 -ipTablesConntrackTableSize: 2 -``` - -Write: - -```yaml -kind: KubeProxyConfiguration -apiVersion: kubeproxy.config.k8s.io/v1beta3 -ipTables: - syncPeriod: 2 - conntrack: - hashSize: 2 - tableSize: 2 -``` - -We should avoid passing around full configuration options to deeply constructed -modules. For example, instead of calling NewSomethingController in the -controller-manager with the full controller-manager config, group relevant -config into a subobject and only pass the subobject. We should expose the -smallest possible necessary configuration to the SomethingController. - - -### Handling Different Types Of Configuration - -Above in "Type Of Configuration" we introduce a few ways to partition -configuration options. Environment information, request context depending -configuration, feature gates, and static configuration should be avoided if at -all possible using a configuration alternative. We should maintain separate -objects along these partitions and consider retrieving these configurations -from separate source (i.e. files). For example: kubeconfig (which falls into -the bootstrapping category) should not be part of the main config option (nor -should the filepath to the kubeconfig), per-instance config should be stored -separately from shared config. This allows for composition and obviates the -need for layering/templating solutions. - - -### In-Process Representation Of Configuration - -We should separate structs for flags, serializable config, and runtime config. - -1. Structs for flags should have enough information for the process startup to retrieve its full configuration. Examples include: path the kubeconfig, path to configuration file, namespace and name of configmap to use for configuration. -1. Structs for serializable configuration: This struct contains the full set of options in a serializable form (e.g. to represent an ip address instead of `net.IP`, use `string`). This is the struct that is versioned and serialized to disk using API machinery. -1. Structs for runtime: This struct holds data in the most appropriate format for execution. This field can hold non-serializable types (e.g. have a `kubeClient` field instead of a `kubeConfig` field, store ip addresses as `net.IP`). - -The flag struct is transformed into the configuration struct which is -transformed into the runtime struct. - - -### Migrating Away From Flags - -Migrating to component configuration can happen incrementally (per component). -By versioning each component's API group separately, we can allow each API -group to advance to beta and GA independently. APIs should be approved by -component owners and reviewers familiar with the component configuration -conventions. We can incentivize operators to migrate away from flags by making -new configuration options only available through the component configuration -APIs. - -# Caveats - -Proposed are not laws but guidelines and as such we've favored completeness -over consistency. There will thus be need for exceptions. - -1. Components (especially those that are not self hosted such as the kubelet) will require custom rollout strategies of new config. -1. Pod checkpointing by kubelet would allow this strategy to be simpler to make reliable. - - -# Miscellaneous Consideration - -1. **This document takes intentionally a very zealous stance against configuration.** Often configuration alternatives are not possible in Kubernetes as they are in proprietary software because Kubernetes has to run in diverse environments, with diverse users, managed by diverse operators. -1. More frequent releases of kubernetes would make "skipping the config knob" more enticing because fixing a bad guess at a const wouldn't take O(4 months) best case to rollout. Factoring in our support for old versions, it takes closer to a year. -1. Self-hosting resolves much of the distribution issue (except for maybe the Kubelet) but reliability is predicated on to-be-implemented features such as kubelet checkpointing of pod dependencies and sound operational practices such as incremental rollout of new configuration using Deployments/DaemonSets. -1. Validating config is hard. Fatal logs lead to crash loops and error logs are ignored. Both options are suboptimal. -1. Configuration needs to be updatable when components are down. -1. Naming style guide: - 1. No negatives, e.g. prefer --enable-foo over --disable-foo - 1. Use the active voice -1. We should actually enforce deprecation. Can we have a test that fails when a comment exists beyond its deadline to be removed? See [#44248](https://github.com/kubernetes/kubernetes/issues/44248) -1. Use different implementations of the same interface rather than if statements to toggle features. This makes deprecation and deletion easy, improving maintainability. -1. How does the proposed solution meet the requirements? Which desired qualities are missed? -1. Configuration changes should trigger predictable and reproducible actions. From a given system state and a given component configuration, we should be able to simulate the actions that the system will take. +This file is a placeholder to preserve links. Please remove by April 24, 2019 or the release of kubernetes 1.13, whichever comes first. \ No newline at end of file diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md index 46ca318df..c380b6c37 100644 --- a/contributors/devel/conformance-tests.md +++ b/contributors/devel/conformance-tests.md @@ -1,216 +1,3 @@ -# Conformance Testing in Kubernetes +This file has moved to https://git.k8s.io/community/contributors/devel/sig-architecture/conformance-tests.md. -The Kubernetes Conformance test suite is a subset of e2e tests that SIG -Architecture has approved to define the core set of interoperable features that -all conformant Kubernetes clusters must support. The tests verify that the -expected behavior works as a user might encounter it in the wild. - -The process to add new conformance tests is intended to decouple the development -of useful tests from their promotion to conformance: -- Contributors write and submit e2e tests, to be approved by owning SIGs -- Tests are proven to meet the [conformance test requirements] by review - and by accumulation of data on flakiness and reliability -- A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance) - -NB: This should be viewed as a living document in a few key areas: -- The desired set of conformant behaviors is not adequately expressed by the - current set of e2e tests, as such this document is currently intended to - guide us in the addition of new e2e tests than can fill this gap -- This document currently focuses solely on the requirements for GA, - non-optional features or APIs. The list of requirements will be refined over - time to the point where it as concrete and complete as possible. -- There are currently conformance tests that violate some of the requirements - (e.g., require privileged access), we will be categorizing these tests and - deciding what to do once we have a better understanding of the situation -- Once we resolve the above issues, we plan on identifying the appropriate areas - to relax requirements to allow for the concept of conformance Profiles that - cover optional or additional behaviors - -## Conformance Test Requirements - -Conformance tests currently test only GA, non-optional features or APIs. More -specifically, a test is eligible for promotion to conformance if: - -- it tests only GA, non-optional features or APIs (e.g., no alpha or beta - endpoints, no feature flags required, no deprecated features) -- it works for all providers (e.g., no `SkipIfProviderIs`/`SkipUnlessProviderIs` - calls) -- it is non-privileged (e.g., does not require root on nodes, access to raw - network interfaces, or cluster admin permissions) -- it works without access to the public internet (short of whatever is required - to pre-pull images for conformance tests) -- it works without non-standard filesystem permissions granted to pods -- it does not rely on any binaries that would not be required for the linux - kernel or kubelet to run (e.g., can't rely on git) -- any container images used within the test support all architectures for which - kubernetes releases are built -- it passes against the appropriate versions of kubernetes as spelled out in - the [conformance test version skew policy] -- it is stable and runs consistently (e.g., no flakes) - -Examples of features which are not currently eligible for conformance tests: - -- node/platform-reliant features, eg: multiple disk mounts, GPUs, high density, - etc. -- optional features, eg: policy enforcement -- cloud-provider-specific features, eg: GCE monitoring, S3 Bucketing, etc. -- anything that requires a non-default admission plugin - -Examples of tests which are not eligible for promotion to conformance: -- anything that checks specific Events are generated, as we make no guarantees - about the contents of events, nor their delivery -- anything that checks optional Condition fields, such as Reason or Message, as - these may change over time (however it is reasonable to verify these fields - exist or are non-empty) - -Examples of areas we may want to relax these requirements once we have a -sufficient corpus of tests that define out of the box functionality in all -reasonable production worthy environments: -- tests may need to create or set objects or fields that are alpha or beta that - bypass policies that are not yet GA, but which may reasonably be enabled on a - conformant cluster (e.g., pod security policy, non-GA scheduler annotations) - -## Conformance Test Version Skew Policy - -As each new release of Kubernetes provides new functionality, the subset of -tests necessary to demonstrate conformance grows with each release. Conformance -is thus considered versioned, with the same backwards compatibility guarantees -as laid out in the [kubernetes versioning policy] - -To quote: - -> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and -> should work with v1.2, v1.3, and v1.4 clients. - -Conformance tests for a given version should be run off of the release branch -that corresponds to that version. Thus `v1.2` conformance tests would be run -from the head of the `release-1.2` branch. - -For example, suppose we're in the midst of developing kubernetes v1.3. Clusters -with the following versions must pass conformance tests built from the -following branches: - -| cluster version | master | release-1.3 | release-1.2 | release-1.1 | -| --------------- | ----- | ----------- | ----------- | ----------- | -| v1.3.0-alpha | yes | yes | yes | no | -| v1.2.x | no | no | yes | yes | -| v1.1.x | no | no | no | yes | - -## Running Conformance Tests - -Conformance tests are designed to be run even when there is no cloud provider -configured. Conformance tests must be able to be run against clusters that have -not been created with `hack/e2e.go`, just provide a kubeconfig with the -appropriate endpoint and credentials. - -These commands are intended to be run within a kubernetes directory, either -cloned from source, or extracted from release artifacts such as -`kubernetes.tar.gz`. They assume you have a valid golang installation. - -```sh -# ensure kubetest is installed -go get -u k8s.io/test-infra/kubetest - -# build test binaries, ginkgo, and kubectl first: -make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" - -# setup for conformance tests -export KUBECONFIG=/path/to/kubeconfig -export KUBERNETES_CONFORMANCE_TEST=y - -# Option A: run all conformance tests serially -kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" - -# Option B: run parallel conformance tests first, then serial conformance tests serially -GINKGO_PARALLEL=y kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" -kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" -``` - -## Kubernetes Conformance Document - -For each Kubernetes release, a Conformance Document will be generated that lists -all of the tests that comprise the conformance test suite, along with the formal -specification of each test. For an example, see the [v1.9 conformance doc]. -This document will help people understand what features are being tested without -having to look through the testcase's code directly. - - -## Promoting Tests to Conformance - -To promote a test to the conformance test suite, open a PR as follows: -- is titled "Promote xxx e2e test to Conformance" -- includes information and metadata in the description as follows: - - "/area conformance" on a newline - - "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews - @kubernetes/cncf-conformance-wg" on a new line, where sig-foo is whichever - sig owns this test - - any necessary information in the description to verify that the test meets - [conformance test requirements], such as links to reports or dashboards that - prove lack of flakiness -- contains no other modifications to test source code other than the following: - - modifies the testcase to use the `framework.ConformanceIt()` function rather - than the `framework.It()` function - - adds a comment immediately before the `ConformanceIt()` call that includes - all of the required [conformance test comment metadata] -- add the PR to SIG Architecture's [Conformance Test Review board] - - -### Conformance Test Comment Metadata - -Each conformance test must include the following piece of metadata -within its associated comment: - -- `Release`: indicates the Kubernetes release that the test was added to the - conformance test suite. If the test was modified in subsequent releases - then those releases should be included as well (comma separated) -- `Testname`: a human readable short name of the test -- `Description`: a detailed description of the test. This field must describe - the required behaviour of the Kubernetes components being tested using - [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field - is meant to be a "specification" of the tested Kubernetes features, as - such, it must be detailed enough so that readers can fully understand - the aspects of Kubernetes that are being tested without having to read - the test's code directly. Additionally, this test should provide a clear - distinction between the parts of the test that are there for the purpose - of validating Kubernetes rather than simply infrastructure logic that - is necessary to setup, or clean up, the test. - -### Sample Conformance Test - -The following snippet of code shows a sample conformance test's metadata: - -``` -/* - Release : v1.9 - Testname: Kubelet: log output - Description: By default the stdout and stderr from the process being - executed in a pod MUST be sent to the pod's logs. -*/ -framework.ConformanceIt("it should print the output to logs", func() { - ... -}) -``` - -The corresponding portion of the Kubernetes Conformance Documentfor this test -would then look like this: - -> ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) -> -> Release : v1.9 -> -> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. - -### Reporting Conformance Test Results - -Conformance test results, by provider and releases, can be viewed in the -[testgrid conformance dashboard]. If you wish to contribute test results -for your provider, please see the [testgrid conformance README] - -[kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew -[Conformance Test Review board]: https://github.com/kubernetes-sigs/architecture-tracking/projects/1 -[conformance test requirements]: #conformance-test-requirements -[conformance test metadata]: #conformance-test-metadata -[conformance test version skew policy]: #conformance-test-version-skew-policy -[testgrid conformance dashboard]: https://testgrid.k8s.io/conformance-all -[testgrid conformance README]: https://github.com/kubernetes/test-infra/blob/master/testgrid/conformance/README.md -[v1.9 conformance doc]: https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md +This file is a placeholder to preserve links. Please remove by April 24, 2019 or the release of kubernetes 1.13, whichever comes first. \ No newline at end of file diff --git a/contributors/devel/development.md b/contributors/devel/development.md index 1d093485c..8c305b148 100644 --- a/contributors/devel/development.md +++ b/contributors/devel/development.md @@ -167,7 +167,7 @@ Kubernetes uses [`godep`](https://github.com/tools/godep) to manage dependencies. Developers who need to manage dependencies in the `vendor/` tree should read -the docs on [using godep to manage dependencies](godep.md). +the docs on [using godep to manage dependencies](sig-architecture/godep.md). ## Build with Bazel/Gazel diff --git a/contributors/devel/godep.md b/contributors/devel/godep.md index 4b10a7d5c..8ade43730 100644 --- a/contributors/devel/godep.md +++ b/contributors/devel/godep.md @@ -1,251 +1,3 @@ -# Using godep to manage dependencies +This file has moved to https://git.k8s.io/community/contributors/devel/sig-architecture/godep.md. -This document is intended to show a way for managing `vendor/` tree dependencies -in Kubernetes. If you do not need to manage vendored dependencies, you probably -do not need to read this. - -## Background - -As a tool, `godep` leaves much to be desired. It builds on `go get`, and adds -the ability to pin dependencies to exact git version. The `go get` tool itself -doesn't have any concept of versions, and tends to blow up if it finds a git -repo synced to anything but `master`, but that is exactly the state that -`godep` leaves repos. This is a recipe for frustration when people try to use -the tools. - -This doc will focus on predictability and reproducibility. - -## Justifications for an update - -Before you update a dependency, take a moment to consider why it should be -updated. Valid reasons include: - 1. We need new functionality that is in a later version. - 2. New or improved APIs in the dependency significantly improve Kubernetes code. - 3. Bugs were fixed that impact Kubernetes. - 4. Security issues were fixed even if they don't impact Kubernetes yet. - 5. Performance, scale, or efficiency was meaningfully improved. - 6. We need dependency A and there is a transitive dependency B. - 7. Kubernetes has an older level of a dependency that is precluding being able -to work with other projects in the ecosystem. - -## Theory of operation - -The `go` toolchain assumes a global workspace that hosts all of your Go code. - -The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`. -This reads the `Godeps.json` file, downloads all of the dependencies from the -internet, and syncs them to the specified revisions. You can then make -changes - sync to different revisions or edit Kubernetes code to use new -dependencies (and satisfy them with `go get`). When ready, you tell `godep` to -"save" everything, which it does by walking the Kubernetes code, finding all -required dependencies, copying them from `$GOPATH` into the `vendor/` directory, -and rewriting `Godeps.json`. - -This does not work well, when combined with a global Go workspace. Instead, we -will set up a private workspace for this process. - -The Kubernetes build process uses this same technique, and offers a tool called -`run-in-gopath.sh` which sets up and switches to a local, private workspace, -including setting up `$GOPATH` and `$PATH`. If you wrap commands with this -tool, they will use the private workspace, which will not conflict with other -projects and is easily cleaned up and recreated. - -To see this in action, you can run an interactive shell in this environment: - -```sh -# Run a shell, but don't run your own shell initializations. -hack/run-in-gopath.sh bash --norc --noprofile -``` - -## Restoring deps - -To extract and download dependencies into `$GOPATH` we provide a script: -`hack/godep-restore.sh`. If you run this tool, it will restore into your own -`$GOPATH`. If you wrap it in `run-in-gopath.sh` it will restore into your -`_output/` directory. - -```sh -hack/run-in-gopath.sh hack/godep-restore.sh -``` - -This script will try to optimize what it needs to download, and if it seems the -dependencies are all present already, it will return very quickly. - -If there's every any doubt about the correctness of your dependencies, you can -simply `make clean` or `rm -rf _output`, and run it again. - -Now you should have a clean copy of all of the Kubernetes dependencies. - -Downloading dependencies might take a while, so if you want to see progress -information use the `-v` flag: - -```sh -hack/run-in-gopath.sh hack/godep-restore.sh -v -``` - -## Making changes - -The most common things people need to do with deps are add and update them. -These are similar but different. - -### Adding a dep - -For the sake of examples, consider that we have discovered a wonderful Go -library at `example.com/go/frob`. The first thing you need to do is get that -code into your workspace: - -```sh -hack/run-in-gopath.sh go get -d example.com/go/frob -``` - -This will fetch, but not compile (omit the `-d` if you want to compile it now), -the library into your private `$GOPATH`. It will pull whatever the default -revision of that library is, typically the `master` branch for git repositories. -If this is not the revision you need, you can change it, for example to -`v1.0.0`: - -```sh -hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0' -``` - -Now that the code is present, you can start to use it in Kubernetes code. -Because it is in your private workspace's `$GOPATH`, it might not be part of -your own `$GOPATH`, so tools like `goimports` might not find it. This is an -unfortunate side-effect of this process. You can either add the whole private -workspace to your own `$GOPATH` or you can `go get` the library into your own -`$GOPATH` until it is properly vendored into Kubernetes. - -Another possible complication is a dep that uses `gopdep` itself. In that case, -you need to restore its dependencies, too: - -```sh -hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore' -``` - -If the transitive deps collide with Kubernetes deps, you may have to manually -resolve things. This is where the ability to run a shell in this environment -comes in handy: - -```sh -hack/run-in-gopath.sh bash --norc --noprofile -``` - -### Updating a dep - -Sometimes we already have a dep, but the version of it is wrong. Because of the -way that `godep` and `go get` interact (badly) it's generally easiest to hit it -with a big hammer: - -```sh -hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob' -hack/run-in-gopath.sh go get -d example.com/go/frob -hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0' -``` - -This will remove the code, re-fetch it, and sync to your desired version. - -### Removing a dep - -This happens almost for free. If you edit Kubernetes code and remove the last -use of a given dependency, you only need to restore and save the deps, and the -`godep` tool will figure out that you don't need that dep any more: - -## Saving deps - -Now that you have made your changes - adding, updating, or removing the use of a -dep - you need to rebuild the dependency database and make changes to the -`vendor/` directory. - -```sh -hack/run-in-gopath.sh hack/godep-save.sh -``` - -This will run through all of the primary targets for the Kubernetes project, -calculate which deps are needed, and rebuild the database. It will also -regenerate other metadata files which the project needs, such as BUILD files and -the LICENSE database. - -Commit the changes before updating deps in staging repos. - -## Saving deps in staging repos - -Kubernetes stores some code in a directory called `staging` which is handled -specially, and is not covered by the above. If you modified any code under -staging, or if you changed a dependency of code under staging (even -transitively), you'll also need to update deps there: - -```sh -./hack/update-staging-godeps.sh -``` - -Then commit the changes generated by the above script. - -## Commit messages - -Terse messages like "Update foo.org/bar to 0.42" are problematic -for maintainability. Please include in your commit message the -detailed reason why the dependencies were modified. - -Too commonly dependency changes have a ripple effect where something -else breaks unexpectedly. The first instinct during issue triage -is to revert a change. If the change was made to fix some other -issue and that issue was not documented, then a revert simply -continues the ripple by fixing one issue and reintroducing another -which then needs refixed. This can needlessly span multiple days -as CI results bubble in and subsequent patches fix and refix and -rerefix issues. This may be avoided if the original modifications -recorded artifacts of the change rationale. - -## Sanity checking - -After all of this is done, `git status` should show you what files have been -modified and added/removed. Make sure to sanity-check them with `git diff`, and -to `git add` and `git rm` them, as needed. It is commonly advised to make one -`git commit` which includes just the dependencies and Godeps files, and -another `git commit` that includes changes to Kubernetes code to use (or stop -using) the new/updated/removed dependency. These commits can go into a single -pull request. - -Before sending your PR, it's a good idea to sanity check that your -Godeps.json file and the contents of `vendor/ `are ok: - -```sh -hack/run-in-gopath.sh hack/verify-godeps.sh -``` - -All this script will do is a restore, followed by a save, and then look for -changes. If you followed the above instructions, it should be clean. If it is -not, you get to figure out why. - -## Manual updates - -It is sometimes expedient to manually fix the `Godeps.json` file to -minimize the changes. However, without great care this can lead to failures -with the verifier scripts. The kubernetes codebase does "interesting things" -with symlinks between `vendor/` and `staging/` to allow multiple Go import -paths to coexist in the same git repo. - -The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull -request. - -## Reviewing and approving dependency changes - -Particular attention to detail should be exercised when reviewing and approving -PRs that add/remove/update dependencies. Importing a new dependency should bring -a certain degree of value as there is a maintenance overhead for maintaining -dependencies into the future. - -When importing a new dependency, be sure to keep an eye out for the following: -- Is the dependency maintained? -- Does the dependency bring value to the project? Could this be done without - adding a new dependency? -- Is the target dependency the original source, or a fork? -- Is there already a dependency in the project that does something similar? -- Does the dependency have a license that is compatible with the Kubernetes - project? - -All new dependency licenses should be reviewed by either Tim Hockin (@thockin) -or the Steering Committee (@kubernetes/steering-committee) to ensure that they -are compatible with the Kubernetes project license. It is also important to note -and flag if a license has changed when updating a dependency, so that these can -also be reviewed. +This file is a placeholder to preserve links. Please remove by April 24, 2019 or the release of kubernetes 1.13, whichever comes first. \ No newline at end of file diff --git a/contributors/devel/sig-architecture/api-conventions.md b/contributors/devel/sig-architecture/api-conventions.md new file mode 100644 index 000000000..2e0bd7ad8 --- /dev/null +++ b/contributors/devel/sig-architecture/api-conventions.md @@ -0,0 +1,1367 @@ +API Conventions +=============== + +Updated: 3/7/2017 + +*This document is oriented at users who want a deeper understanding of the +Kubernetes API structure, and developers wanting to extend the Kubernetes API. +An introduction to using resources with kubectl can be found in [the object management overview](https://kubernetes.io/docs/tutorials/object-management-kubectl/object-management/).* + +**Table of Contents** + + + - [Types (Kinds)](#types-kinds) + - [Resources](#resources) + - [Objects](#objects) + - [Metadata](#metadata) + - [Spec and Status](#spec-and-status) + - [Typical status properties](#typical-status-properties) + - [References to related objects](#references-to-related-objects) + - [Lists of named subobjects preferred over maps](#lists-of-named-subobjects-preferred-over-maps) + - [Primitive types](#primitive-types) + - [Constants](#constants) + - [Unions](#unions) + - [Lists and Simple kinds](#lists-and-simple-kinds) + - [Differing Representations](#differing-representations) + - [Verbs on Resources](#verbs-on-resources) + - [PATCH operations](#patch-operations) + - [Strategic Merge Patch](#strategic-merge-patch) + - [Idempotency](#idempotency) + - [Optional vs. Required](#optional-vs-required) + - [Defaulting](#defaulting) + - [Late Initialization](#late-initialization) + - [Concurrency Control and Consistency](#concurrency-control-and-consistency) + - [Serialization Format](#serialization-format) + - [Units](#units) + - [Selecting Fields](#selecting-fields) + - [Object references](#object-references) + - [HTTP Status codes](#http-status-codes) + - [Success codes](#success-codes) + - [Error codes](#error-codes) + - [Response Status Kind](#response-status-kind) + - [Events](#events) + - [Naming conventions](#naming-conventions) + - [Label, selector, and annotation conventions](#label-selector-and-annotation-conventions) + - [WebSockets and SPDY](#websockets-and-spdy) + - [Validation](#validation) + + +The conventions of the [Kubernetes API](https://kubernetes.io/docs/api/) (and related APIs in the +ecosystem) are intended to ease client development and ensure that configuration +mechanisms can be implemented that work across a diverse set of use cases +consistently. + +The general style of the Kubernetes API is RESTful - clients create, update, +delete, or retrieve a description of an object via the standard HTTP verbs +(POST, PUT, DELETE, and GET) - and those APIs preferentially accept and return +JSON. Kubernetes also exposes additional endpoints for non-standard verbs and +allows alternative content types. All of the JSON accepted and returned by the +server has a schema, identified by the "kind" and "apiVersion" fields. Where +relevant HTTP header fields exist, they should mirror the content of JSON +fields, but the information should not be represented only in the HTTP header. + +The following terms are defined: + +* **Kind** the name of a particular object schema (e.g. the "Cat" and "Dog" +kinds would have different attributes and properties) +* **Resource** a representation of a system entity, sent or retrieved as JSON +via HTTP to the server. Resources are exposed via: + * Collections - a list of resources of the same type, which may be queryable + * Elements - an individual resource, addressable via a URL +* **API Group** a set of resources that are exposed together. Along +with the version is exposed in the "apiVersion" field as "GROUP/VERSION", e.g. +"policy.k8s.io/v1". + +Each resource typically accepts and returns data of a single kind. A kind may be +accepted or returned by multiple resources that reflect specific use cases. For +instance, the kind "Pod" is exposed as a "pods" resource that allows end users +to create, update, and delete pods, while a separate "pod status" resource (that +acts on "Pod" kind) allows automated processes to update a subset of the fields +in that resource. + +Resources are bound together in API groups - each group may have one or more +versions that evolve independent of other API groups, and each version within +the group has one or more resources. Group names are typically in domain name +form - the Kubernetes project reserves use of the empty group, all single +word names ("extensions", "apps"), and any group name ending in "*.k8s.io" for +its sole use. When choosing a group name, we recommend selecting a subdomain +your group or organization owns, such as "widget.mycompany.com". + +Resource collections should be all lowercase and plural, whereas kinds are +CamelCase and singular. Group names must be lower case and be valid DNS +subdomains. + + +## Types (Kinds) + +Kinds are grouped into three categories: + +1. **Objects** represent a persistent entity in the system. + + Creating an API object is a record of intent - once created, the system will +work to ensure that resource exists. All API objects have common metadata. + + An object may have multiple resources that clients can use to perform +specific actions that create, update, delete, or get. + + Examples: `Pod`, `ReplicationController`, `Service`, `Namespace`, `Node`. + +2. **Lists** are collections of **resources** of one (usually) or more +(occasionally) kinds. + + The name of a list kind must end with "List". Lists have a limited set of +common metadata. All lists use the required "items" field to contain the array +of objects they return. Any kind that has the "items" field must be a list kind. + + Most objects defined in the system should have an endpoint that returns the +full set of resources, as well as zero or more endpoints that return subsets of +the full list. Some objects may be singletons (the current user, the system +defaults) and may not have lists. + + In addition, all lists that return objects with labels should support label +filtering (see [the labels documentation](https://kubernetes.io/docs/user-guide/labels/)), and most +lists should support filtering by fields. + + Examples: `PodLists`, `ServiceLists`, `NodeLists`. + + TODO: Describe field filtering below or in a separate doc. + +3. **Simple** kinds are used for specific actions on objects and for +non-persistent entities. + + Given their limited scope, they have the same set of limited common metadata +as lists. + + For instance, the "Status" kind is returned when errors occur and is not +persisted in the system. + + Many simple resources are "subresources", which are rooted at API paths of +specific resources. When resources wish to expose alternative actions or views +that are closely coupled to a single resource, they should do so using new +sub-resources. Common subresources include: + + * `/binding`: Used to bind a resource representing a user request (e.g., Pod, +PersistentVolumeClaim) to a cluster infrastructure resource (e.g., Node, +PersistentVolume). + * `/status`: Used to write just the status portion of a resource. For +example, the `/pods` endpoint only allows updates to `metadata` and `spec`, +since those reflect end-user intent. An automated process should be able to +modify status for users to see by sending an updated Pod kind to the server to +the "/pods/<name>/status" endpoint - the alternate endpoint allows +different rules to be applied to the update, and access to be appropriately +restricted. + * `/scale`: Used to read and write the count of a resource in a manner that +is independent of the specific resource schema. + + Two additional subresources, `proxy` and `portforward`, provide access to +cluster resources as described in +[accessing the cluster](https://kubernetes.io/docs/user-guide/accessing-the-cluster/). + +The standard REST verbs (defined below) MUST return singular JSON objects. Some +API endpoints may deviate from the strict REST pattern and return resources that +are not singular JSON objects, such as streams of JSON objects or unstructured +text log data. + +A common set of "meta" API objects are used across all API groups and are +thus considered part of the server group named `meta.k8s.io`. These types may +evolve independent of the API group that uses them and API servers may allow +them to be addressed in their generic form. Examples are `ListOptions`, +`DeleteOptions`, `List`, `Status`, `WatchEvent`, and `Scale`. For historical +reasons these types are part of each existing API group. Generic tools like +quota, garbage collection, autoscalers, and generic clients like kubectl +leverage these types to define consistent behavior across different resource +types, like the interfaces in programming languages. + +The term "kind" is reserved for these "top-level" API types. The term "type" +should be used for distinguishing sub-categories within objects or subobjects. + +### Resources + +All JSON objects returned by an API MUST have the following fields: + +* kind: a string that identifies the schema this object should have +* apiVersion: a string that identifies the version of the schema the object +should have + +These fields are required for proper decoding of the object. They may be +populated by the server by default from the specified URL path, but the client +likely needs to know the values in order to construct the URL path. + +### Objects + +#### Metadata + +Every object kind MUST have the following metadata in a nested object field +called "metadata": + +* namespace: a namespace is a DNS compatible label that objects are subdivided +into. The default namespace is 'default'. See +[the namespace docs](https://kubernetes.io/docs/user-guide/namespaces/) for more. +* name: a string that uniquely identifies this object within the current +namespace (see [the identifiers docs](https://kubernetes.io/docs/user-guide/identifiers/)). +This value is used in the path when retrieving an individual object. +* uid: a unique in time and space value (typically an RFC 4122 generated +identifier, see [the identifiers docs](https://kubernetes.io/docs/user-guide/identifiers/)) +used to distinguish between objects with the same name that have been deleted +and recreated + +Every object SHOULD have the following metadata in a nested object field called +"metadata": + +* resourceVersion: a string that identifies the internal version of this object +that can be used by clients to determine when objects have changed. This value +MUST be treated as opaque by clients and passed unmodified back to the server. +Clients should not assume that the resource version has meaning across +namespaces, different kinds of resources, or different servers. (See +[concurrency control](#concurrency-control-and-consistency), below, for more +details.) +* generation: a sequence number representing a specific generation of the +desired state. Set by the system and monotonically increasing, per-resource. May +be compared, such as for RAW and WAW consistency. +* creationTimestamp: a string representing an RFC 3339 date of the date and time +an object was created +* deletionTimestamp: a string representing an RFC 3339 date of the date and time +after which this resource will be deleted. This field is set by the server when +a graceful deletion is requested by the user, and is not directly settable by a +client. The resource will be deleted (no longer visible from resource lists, and +not reachable by name) after the time in this field except when the object has +a finalizer set. In case the finalizer is set the deletion of the object is +postponed at least until the finalizer is removed. +Once the deletionTimestamp is set, this value may not be unset or be set further +into the future, although it may be shortened or the resource may be deleted +prior to this time. +* labels: a map of string keys and values that can be used to organize and +categorize objects (see [the labels docs](https://kubernetes.io/docs/user-guide/labels/)) +* annotations: a map of string keys and values that can be used by external +tooling to store and retrieve arbitrary metadata about this object (see +[the annotations docs](https://kubernetes.io/docs/user-guide/annotations/)) + +Labels are intended for organizational purposes by end users (select the pods +that match this label query). Annotations enable third-party automation and +tooling to decorate objects with additional metadata for their own use. + +#### Spec and Status + +By convention, the Kubernetes API makes a distinction between the specification +of the desired state of an object (a nested object field called "spec") and the +status of the object at the current time (a nested object field called +"status"). The specification is a complete description of the desired state, +including configuration settings provided by the user, +[default values](#defaulting) expanded by the system, and properties initialized +or otherwise changed after creation by other ecosystem components (e.g., +schedulers, auto-scalers), and is persisted in stable storage with the API +object. If the specification is deleted, the object will be purged from the +system. The status summarizes the current state of the object in the system, and +is usually persisted with the object by an automated processes but may be +generated on the fly. At some cost and perhaps some temporary degradation in +behavior, the status could be reconstructed by observation if it were lost. + +When a new version of an object is POSTed or PUT, the "spec" is updated and +available immediately. Over time the system will work to bring the "status" into +line with the "spec". The system will drive toward the most recent "spec" +regardless of previous versions of that stanza. In other words, if a value is +changed from 2 to 5 in one PUT and then back down to 3 in another PUT the system +is not required to 'touch base' at 5 before changing the "status" to 3. In other +words, the system's behavior is *level-based* rather than *edge-based*. This +enables robust behavior in the presence of missed intermediate state changes. + +The Kubernetes API also serves as the foundation for the declarative +configuration schema for the system. In order to facilitate level-based +operation and expression of declarative configuration, fields in the +specification should have declarative rather than imperative names and +semantics -- they represent the desired state, not actions intended to yield the +desired state. + +The PUT and POST verbs on objects MUST ignore the "status" values, to avoid +accidentally overwriting the status in read-modify-write scenarios. A `/status` +subresource MUST be provided to enable system components to update statuses of +resources they manage. + +Otherwise, PUT expects the whole object to be specified. Therefore, if a field +is omitted it is assumed that the client wants to clear that field's value. The +PUT verb does not accept partial updates. Modification of just part of an object +may be achieved by GETting the resource, modifying part of the spec, labels, or +annotations, and then PUTting it back. See +[concurrency control](#concurrency-control-and-consistency), below, regarding +read-modify-write consistency when using this pattern. Some objects may expose +alternative resource representations that allow mutation of the status, or +performing custom actions on the object. + +All objects that represent a physical resource whose state may vary from the +user's desired intent SHOULD have a "spec" and a "status". Objects whose state +cannot vary from the user's desired intent MAY have only "spec", and MAY rename +"spec" to a more appropriate name. + +Objects that contain both spec and status should not contain additional +top-level fields other than the standard metadata fields. + +Some objects which are not persisted in the system - such as `SubjectAccessReview` +and other webhook style calls - may choose to add spec and status to encapsulate +a "call and response" pattern. The spec is the request (often a request for +information) and the status is the response. For these RPC like objects the only +operation may be POST, but having a consistent schema between submission and +response reduces the complexity of these clients. + + +##### Typical status properties + +**Conditions** represent the latest available observations of an object's +state. They are an extension mechanism intended to be used when the details of +an observation are not a priori known or would not apply to all instances of a +given Kind. For observations that are well known and apply to all instances, a +regular field is preferred. An example of a Condition that probably should +have been a regular field is Pod's "Ready" condition - it is managed by core +controllers, it is well understood, and it applies to all Pods. + +Objects may report multiple conditions, and new types of conditions may be +added in the future or by 3rd party controllers. Therefore, conditions are +represented using a list/slice, where all have similar structure. + +The `FooCondition` type for some resource type `Foo` may include a subset of the +following fields, but must contain at least `type` and `status` fields: + +```go + Type FooConditionType `json:"type" description:"type of Foo condition"` + Status ConditionStatus `json:"status" description:"status of the condition, one of True, False, Unknown"` + + // +optional + Reason *string `json:"reason,omitempty" description:"one-word CamelCase reason for the condition's last transition"` + // +optional + Message *string `json:"message,omitempty" description:"human-readable message indicating details about last transition"` + + // +optional + LastHeartbeatTime *unversioned.Time `json:"lastHeartbeatTime,omitempty" description:"last time we got an update on a given condition"` + // +optional + LastTransitionTime *unversioned.Time `json:"lastTransitionTime,omitempty" description:"last time the condition transit from one status to another"` +``` + +Additional fields may be added in the future. + +Do not use fields that you don't need - simpler is better. + +Use of the `Reason` field is encouraged. + +Use the `LastHeartbeatTime` with great caution - frequent changes to this field +can cause a large fan-out effect for some resources. + +Conditions should be added to explicitly convey properties that users and +components care about rather than requiring those properties to be inferred from +other observations. Once defined, the meaning of a Condition can not be +changed arbitrarily - it becomes part of the API, and has the same backwards- +and forwards-compatibility concerns of any other part of the API. + +Condition status values may be `True`, `False`, or `Unknown`. The absence of a +condition should be interpreted the same as `Unknown`. How controllers handle +`Unknown` depends on the Condition in question. + +Condition types should indicate state in the "abnormal-true" polarity. For +example, if the condition indicates when a policy is invalid, the "is valid" +case is probably the norm, so the condition should be called "Invalid". + +The thinking around conditions has evolved over time, so there are several +non-normative examples in wide use. + +In general, condition values may change back and forth, but some condition +transitions may be monotonic, depending on the resource and condition type. +However, conditions are observations and not, themselves, state machines, nor do +we define comprehensive state machines for objects, nor behaviors associated +with state transitions. The system is level-based rather than edge-triggered, +and should assume an Open World. + +An example of an oscillating condition type is `Ready` (despite it running +afoul of current guidance), which indicates the object was believed to be fully +operational at the time it was last probed. A possible monotonic condition +could be `Failed`. A `True` status for `Failed` would imply failure with no +retry. An object that was still active would generally not have a `Failed` +condition. + +Some resources in the v1 API contain fields called **`phase`**, and associated +`message`, `reason`, and other status fields. The pattern of using `phase` is +deprecated. Newer API types should use conditions instead. Phase was +essentially a state-machine enumeration field, that contradicted [system-design +principles](../design-proposals/architecture/principles.md#control-logic) and +hampered evolution, since [adding new enum values breaks backward +compatibility](api_changes.md). Rather than encouraging clients to infer +implicit properties from phases, we prefer to explicitly expose the individual +conditions that clients need to monitor. Conditions also have the benefit that +it is possible to create some conditions with uniform meaning across all +resource types, while still exposing others that are unique to specific +resource types. See [#7856](http://issues.k8s.io/7856) for more details and +discussion. + +In condition types, and everywhere else they appear in the API, **`Reason`** is +intended to be a one-word, CamelCase representation of the category of cause of +the current status, and **`Message`** is intended to be a human-readable phrase +or sentence, which may contain specific details of the individual occurrence. +`Reason` is intended to be used in concise output, such as one-line +`kubectl get` output, and in summarizing occurrences of causes, whereas +`Message` is intended to be presented to users in detailed status explanations, +such as `kubectl describe` output. + +Historical information status (e.g., last transition time, failure counts) is +only provided with reasonable effort, and is not guaranteed to not be lost. + +Status information that may be large (especially proportional in size to +collections of other resources, such as lists of references to other objects -- +see below) and/or rapidly changing, such as +[resource usage](../design-proposals/scheduling/resources.md#usage-data), should be put into separate +objects, with possibly a reference from the original object. This helps to +ensure that GETs and watch remain reasonably efficient for the majority of +clients, which may not need that data. + +Some resources report the `observedGeneration`, which is the `generation` most +recently observed by the component responsible for acting upon changes to the +desired state of the resource. This can be used, for instance, to ensure that +the reported status reflects the most recent desired status. + +#### References to related objects + +References to loosely coupled sets of objects, such as +[pods](https://kubernetes.io/docs/user-guide/pods/) overseen by a +[replication controller](https://kubernetes.io/docs/user-guide/replication-controller/), are usually +best referred to using a [label selector](https://kubernetes.io/docs/user-guide/labels/). In order to +ensure that GETs of individual objects remain bounded in time and space, these +sets may be queried via separate API queries, but will not be expanded in the +referring object's status. + +References to specific objects, especially specific resource versions and/or +specific fields of those objects, are specified using the `ObjectReference` type +(or other types representing strict subsets of it). Unlike partial URLs, the +ObjectReference type facilitates flexible defaulting of fields from the +referring object or other contextual information. + +References in the status of the referee to the referrer may be permitted, when +the references are one-to-one and do not need to be frequently updated, +particularly in an edge-based manner. + +#### Lists of named subobjects preferred over maps + +Discussed in [#2004](http://issue.k8s.io/2004) and elsewhere. There are no maps +of subobjects in any API objects. Instead, the convention is to use a list of +subobjects containing name fields. + +For example: + +```yaml +ports: + - name: www + containerPort: 80 +``` + +vs. + +```yaml +ports: + www: + containerPort: 80 +``` + +This rule maintains the invariant that all JSON/YAML keys are fields in API +objects. The only exceptions are pure maps in the API (currently, labels, +selectors, annotations, data), as opposed to sets of subobjects. + +#### Primitive types + +* Avoid floating-point values as much as possible, and never use them in spec. + Floating-point values cannot be reliably round-tripped (encoded and + re-decoded) without changing, and have varying precision and representations + across languages and architectures. +* All numbers (e.g., uint32, int64) are converted to float64 by Javascript and + some other languages, so any field which is expected to exceed that either in + magnitude or in precision (specifically integer values > 53 bits) should be + serialized and accepted as strings. +* Do not use unsigned integers, due to inconsistent support across languages and + libraries. Just validate that the integer is non-negative if that's the case. +* Do not use enums. Use aliases for string instead (e.g., `NodeConditionType`). +* Look at similar fields in the API (e.g., ports, durations) and follow the + conventions of existing fields. +* All public integer fields MUST use the Go `(u)int32` or Go `(u)int64` types, + not `(u)int` (which is ambiguous depending on target platform). Internal + types may use `(u)int`. +* Think twice about `bool` fields. Many ideas start as boolean but eventually + trend towards a small set of mutually exclusive options. Plan for future + expansions by describing the policy options explicitly as a string type + alias (e.g. `TerminationMessagePolicy`). + +#### Constants + +Some fields will have a list of allowed values (enumerations). These values will +be strings, and they will be in CamelCase, with an initial uppercase letter. +Examples: `ClusterFirst`, `Pending`, `ClientIP`. + +#### Unions + +Sometimes, at most one of a set of fields can be set. For example, the +[volumes] field of a PodSpec has 17 different volume type-specific fields, such +as `nfs` and `iscsi`. All fields in the set should be +[Optional](#optional-vs-required). + +Sometimes, when a new type is created, the api designer may anticipate that a +union will be needed in the future, even if only one field is allowed initially. +In this case, be sure to make the field [Optional](#optional-vs-required) +optional. In the validation, you may still return an error if the sole field is +unset. Do not set a default value for that field. + +### Lists and Simple kinds + +Every list or simple kind SHOULD have the following metadata in a nested object +field called "metadata": + +* resourceVersion: a string that identifies the common version of the objects +returned by in a list. This value MUST be treated as opaque by clients and +passed unmodified back to the server. A resource version is only valid within a +single namespace on a single kind of resource. + +Every simple kind returned by the server, and any simple kind sent to the server +that must support idempotency or optimistic concurrency should return this +value. Since simple resources are often used as input alternate actions that +modify objects, the resource version of the simple resource should correspond to +the resource version of the object. + + +## Differing Representations + +An API may represent a single entity in different ways for different clients, or +transform an object after certain transitions in the system occur. In these +cases, one request object may have two representations available as different +resources, or different kinds. + +An example is a Service, which represents the intent of the user to group a set +of pods with common behavior on common ports. When Kubernetes detects a pod +matches the service selector, the IP address and port of the pod are added to an +Endpoints resource for that Service. The Endpoints resource exists only if the +Service exists, but exposes only the IPs and ports of the selected pods. The +full service is represented by two distinct resources - under the original +Service resource the user created, as well as in the Endpoints resource. + +As another example, a "pod status" resource may accept a PUT with the "pod" +kind, with different rules about what fields may be changed. + +Future versions of Kubernetes may allow alternative encodings of objects beyond +JSON. + + +## Verbs on Resources + +API resources should use the traditional REST pattern: + +* GET /<resourceNamePlural> - Retrieve a list of type +<resourceName>, e.g. GET /pods returns a list of Pods. +* POST /<resourceNamePlural> - Create a new resource from the JSON object +provided by the client. +* GET /<resourceNamePlural>/<name> - Retrieves a single resource +with the given name, e.g. GET /pods/first returns a Pod named 'first'. Should be +constant time, and the resource should be bounded in size. +* DELETE /<resourceNamePlural>/<name> - Delete the single resource +with the given name. DeleteOptions may specify gracePeriodSeconds, the optional +duration in seconds before the object should be deleted. Individual kinds may +declare fields which provide a default grace period, and different kinds may +have differing kind-wide default grace periods. A user provided grace period +overrides a default grace period, including the zero grace period ("now"). +* PUT /<resourceNamePlural>/<name> - Update or create the resource +with the given name with the JSON object provided by the client. +* PATCH /<resourceNamePlural>/<name> - Selectively modify the +specified fields of the resource. See more information [below](#patch-operations). +* GET /<resourceNamePlural>&watch=true - Receive a stream of JSON +objects corresponding to changes made to any resource of the given kind over +time. + +### PATCH operations + +The API supports three different PATCH operations, determined by their +corresponding Content-Type header: + +* JSON Patch, `Content-Type: application/json-patch+json` + * As defined in [RFC6902](https://tools.ietf.org/html/rfc6902), a JSON Patch is +a sequence of operations that are executed on the resource, e.g. `{"op": "add", +"path": "/a/b/c", "value": [ "foo", "bar" ]}`. For more details on how to use +JSON Patch, see the RFC. +* Merge Patch, `Content-Type: application/merge-patch+json` + * As defined in [RFC7386](https://tools.ietf.org/html/rfc7386), a Merge Patch +is essentially a partial representation of the resource. The submitted JSON is +"merged" with the current resource to create a new one, then the new one is +saved. For more details on how to use Merge Patch, see the RFC. +* Strategic Merge Patch, `Content-Type: application/strategic-merge-patch+json` + * Strategic Merge Patch is a custom implementation of Merge Patch. For a +detailed explanation of how it works and why it needed to be introduced, see +below. + +#### Strategic Merge Patch + +Details of Strategic Merge Patch are covered [here](strategic-merge-patch.md). + +## Idempotency + +All compatible Kubernetes APIs MUST support "name idempotency" and respond with +an HTTP status code 409 when a request is made to POST an object that has the +same name as an existing object in the system. See +[the identifiers docs](https://kubernetes.io/docs/user-guide/identifiers/) for details. + +Names generated by the system may be requested using `metadata.generateName`. +GenerateName indicates that the name should be made unique by the server prior +to persisting it. A non-empty value for the field indicates the name will be +made unique (and the name returned to the client will be different than the name +passed). The value of this field will be combined with a unique suffix on the +server if the Name field has not been provided. The provided value must be valid +within the rules for Name, and may be truncated by the length of the suffix +required to make the value unique on the server. If this field is specified, and +Name is not present, the server will NOT return a 409 if the generated name +exists - instead, it will either return 201 Created or 504 with Reason +`ServerTimeout` indicating a unique name could not be found in the time +allotted, and the client should retry (optionally after the time indicated in +the Retry-After header). + +## Optional vs. Required + +Fields must be either optional or required. + +Optional fields have the following properties: + +- They have the `+optional` comment tag in Go. +- They are a pointer type in the Go definition (e.g. `bool *awesomeFlag`) or +have a built-in `nil` value (e.g. maps and slices). +- The API server should allow POSTing and PUTing a resource with this field +unset. + +In most cases, optional fields should also have the `omitempty` struct tag (the +`omitempty` option specifies that the field should be omitted from the json +encoding if the field has an empty value). However, If you want to have +different logic for an optional field which is not provided vs. provided with +empty values, do not use `omitempty` (e.g. https://github.com/kubernetes/kubernetes/issues/34641). + +Note that for backward compatibility, any field that has the `omitempty` struct +tag will considered to be optional but this may change in future and having +the `+optional` comment tag is highly recommended. + +Required fields have the opposite properties, namely: + +- They do not have an `+optional` comment tag. +- They do not have an `omitempty` struct tag. +- They are not a pointer type in the Go definition (e.g. `bool otherFlag`). +- The API server should not allow POSTing or PUTing a resource with this field +unset. + +Using the `+optional` or the `omitempty` tag causes OpenAPI documentation to +reflect that the field is optional. + +Using a pointer allows distinguishing unset from the zero value for that type. +There are some cases where, in principle, a pointer is not needed for an +optional field since the zero value is forbidden, and thus implies unset. There +are examples of this in the codebase. However: + +- it can be difficult for implementors to anticipate all cases where an empty +value might need to be distinguished from a zero value +- structs are not omitted from encoder output even where omitempty is specified, +which is messy; +- having a pointer consistently imply optional is clearer for users of the Go +language client, and any other clients that use corresponding types + +Therefore, we ask that pointers always be used with optional fields that do not +have a built-in `nil` value. + + +## Defaulting + +Default resource values are API version-specific, and they are applied during +the conversion from API-versioned declarative configuration to internal objects +representing the desired state (`Spec`) of the resource. Subsequent GETs of the +resource will include the default values explicitly. + +Incorporating the default values into the `Spec` ensures that `Spec` depicts the +full desired state so that it is easier for the system to determine how to +achieve the state, and for the user to know what to anticipate. + +API version-specific default values are set by the API server. + +## Late Initialization + +Late initialization is when resource fields are set by a system controller +after an object is created/updated. + +For example, the scheduler sets the `pod.spec.nodeName` field after the pod is +created. + +Late-initializers should only make the following types of modifications: + - Setting previously unset fields + - Adding keys to maps + - Adding values to arrays which have mergeable semantics +(`patchStrategy:"merge"` attribute in the type definition). + +These conventions: + 1. allow a user (with sufficient privilege) to override any system-default + behaviors by setting the fields that would otherwise have been defaulted. + 1. enables updates from users to be merged with changes made during late +initialization, using strategic merge patch, as opposed to clobbering the +change. + 1. allow the component which does the late-initialization to use strategic +merge patch, which facilitates composition and concurrency of such components. + +Although the apiserver Admission Control stage acts prior to object creation, +Admission Control plugins should follow the Late Initialization conventions +too, to allow their implementation to be later moved to a 'controller', or to +client libraries. + +## Concurrency Control and Consistency + +Kubernetes leverages the concept of *resource versions* to achieve optimistic +concurrency. All Kubernetes resources have a "resourceVersion" field as part of +their metadata. This resourceVersion is a string that identifies the internal +version of an object that can be used by clients to determine when objects have +changed. When a record is about to be updated, it's version is checked against a +pre-saved value, and if it doesn't match, the update fails with a StatusConflict +(HTTP status code 409). + +The resourceVersion is changed by the server every time an object is modified. +If resourceVersion is included with the PUT operation the system will verify +that there have not been other successful mutations to the resource during a +read/modify/write cycle, by verifying that the current value of resourceVersion +matches the specified value. + +The resourceVersion is currently backed by [etcd's +modifiedIndex](https://coreos.com/etcd/docs/latest/v2/api.html). +However, it's important to note that the application should *not* rely on the +implementation details of the versioning system maintained by Kubernetes. We may +change the implementation of resourceVersion in the future, such as to change it +to a timestamp or per-object counter. + +The only way for a client to know the expected value of resourceVersion is to +have received it from the server in response to a prior operation, typically a +GET. This value MUST be treated as opaque by clients and passed unmodified back +to the server. Clients should not assume that the resource version has meaning +across namespaces, different kinds of resources, or different servers. +Currently, the value of resourceVersion is set to match etcd's sequencer. You +could think of it as a logical clock the API server can use to order requests. +However, we expect the implementation of resourceVersion to change in the +future, such as in the case we shard the state by kind and/or namespace, or port +to another storage system. + +In the case of a conflict, the correct client action at this point is to GET the +resource again, apply the changes afresh, and try submitting again. This +mechanism can be used to prevent races like the following: + +``` +Client #1 Client #2 +GET Foo GET Foo +Set Foo.Bar = "one" Set Foo.Baz = "two" +PUT Foo PUT Foo +``` + +When these sequences occur in parallel, either the change to Foo.Bar or the +change to Foo.Baz can be lost. + +On the other hand, when specifying the resourceVersion, one of the PUTs will +fail, since whichever write succeeds changes the resourceVersion for Foo. + +resourceVersion may be used as a precondition for other operations (e.g., GET, +DELETE) in the future, such as for read-after-write consistency in the presence +of caching. + +"Watch" operations specify resourceVersion using a query parameter. It is used +to specify the point at which to begin watching the specified resources. This +may be used to ensure that no mutations are missed between a GET of a resource +(or list of resources) and a subsequent Watch, even if the current version of +the resource is more recent. This is currently the main reason that list +operations (GET on a collection) return resourceVersion. + + +## Serialization Format + +APIs may return alternative representations of any resource in response to an +Accept header or under alternative endpoints, but the default serialization for +input and output of API responses MUST be JSON. + +A protobuf encoding is also accepted for built-in resources. As proto is not +self-describing, there is an envelope wrapper which describes the type of +the contents. + +All dates should be serialized as RFC3339 strings. + +## Units + +Units must either be explicit in the field name (e.g., `timeoutSeconds`), or +must be specified as part of the value (e.g., `resource.Quantity`). Which +approach is preferred is TBD, though currently we use the `fooSeconds` +convention for durations. + +Duration fields must be represented as integer fields with units being +part of the field name (e.g. `leaseDurationSeconds`). We don't use Duration +in the API since that would require clients to implement go-compatible parsing. + +## Selecting Fields + +Some APIs may need to identify which field in a JSON object is invalid, or to +reference a value to extract from a separate resource. The current +recommendation is to use standard JavaScript syntax for accessing that field, +assuming the JSON object was transformed into a JavaScript object, without the +leading dot, such as `metadata.name`. + +Examples: + +* Find the field "current" in the object "state" in the second item in the array +"fields": `fields[1].state.current` + +## Object references + +Object references should either be called `fooName` if referring to an object of +kind `Foo` by just the name (within the current namespace, if a namespaced +resource), or should be called `fooRef`, and should contain a subset of the +fields of the `ObjectReference` type. + + +TODO: Plugins, extensions, nested kinds, headers + + +## HTTP Status codes + +The server will respond with HTTP status codes that match the HTTP spec. See the +section below for a breakdown of the types of status codes the server will send. + +The following HTTP status codes may be returned by the API. + +#### Success codes + +* `200 StatusOK` + * Indicates that the request completed successfully. +* `201 StatusCreated` + * Indicates that the request to create kind completed successfully. +* `204 StatusNoContent` + * Indicates that the request completed successfully, and the response contains +no body. + * Returned in response to HTTP OPTIONS requests. + +#### Error codes + +* `307 StatusTemporaryRedirect` + * Indicates that the address for the requested resource has changed. + * Suggested client recovery behavior: + * Follow the redirect. + + +* `400 StatusBadRequest` + * Indicates the requested is invalid. + * Suggested client recovery behavior: + * Do not retry. Fix the request. + + +* `401 StatusUnauthorized` + * Indicates that the server can be reached and understood the request, but +refuses to take any further action, because the client must provide +authorization. If the client has provided authorization, the server is +indicating the provided authorization is unsuitable or invalid. + * Suggested client recovery behavior: + * If the user has not supplied authorization information, prompt them for +the appropriate credentials. If the user has supplied authorization information, +inform them their credentials were rejected and optionally prompt them again. + + +* `403 StatusForbidden` + * Indicates that the server can be reached and understood the request, but +refuses to take any further action, because it is configured to deny access for +some reason to the requested resource by the client. + * Suggested client recovery behavior: + * Do not retry. Fix the request. + + +* `404 StatusNotFound` + * Indicates that the requested resource does not exist. + * Suggested client recovery behavior: + * Do not retry. Fix the request. + + +* `405 StatusMethodNotAllowed` + * Indicates that the action the client attempted to perform on the resource +was not supported by the code. + * Suggested client recovery behavior: + * Do not retry. Fix the request. + + +* `409 StatusConflict` + * Indicates that either the resource the client attempted to create already +exists or the requested update operation cannot be completed due to a conflict. + * Suggested client recovery behavior: + * * If creating a new resource: + * * Either change the identifier and try again, or GET and compare the +fields in the pre-existing object and issue a PUT/update to modify the existing +object. + * * If updating an existing resource: + * See `Conflict` from the `status` response section below on how to +retrieve more information about the nature of the conflict. + * GET and compare the fields in the pre-existing object, merge changes (if +still valid according to preconditions), and retry with the updated request +(including `ResourceVersion`). + + +* `410 StatusGone` + * Indicates that the item is no longer available at the server and no +forwarding address is known. + * Suggested client recovery behavior: + * Do not retry. Fix the request. + + +* `422 StatusUnprocessableEntity` + * Indicates that the requested create or update operation cannot be completed +due to invalid data provided as part of the request. + * Suggested client recovery behavior: + * Do not retry. Fix the request. + + +* `429 StatusTooManyRequests` + * Indicates that the either the client rate limit has been exceeded or the +server has received more requests then it can process. + * Suggested client recovery behavior: + * Read the `Retry-After` HTTP header from the response, and wait at least +that long before retrying. + + +* `500 StatusInternalServerError` + * Indicates that the server can be reached and understood the request, but +either an unexpected internal error occurred and the outcome of the call is +unknown, or the server cannot complete the action in a reasonable time (this may +be due to temporary server load or a transient communication issue with another +server). + * Suggested client recovery behavior: + * Retry with exponential backoff. + + +* `503 StatusServiceUnavailable` + * Indicates that required service is unavailable. + * Suggested client recovery behavior: + * Retry with exponential backoff. + + +* `504 StatusServerTimeout` + * Indicates that the request could not be completed within the given time. +Clients can get this response ONLY when they specified a timeout param in the +request. + * Suggested client recovery behavior: + * Increase the value of the timeout param and retry with exponential +backoff. + +## Response Status Kind + +Kubernetes will always return the `Status` kind from any API endpoint when an +error occurs. Clients SHOULD handle these types of objects when appropriate. + +A `Status` kind will be returned by the API in two cases: + * When an operation is not successful (i.e. when the server would return a non +2xx HTTP status code). + * When a HTTP `DELETE` call is successful. + +The status object is encoded as JSON and provided as the body of the response. +The status object contains fields for humans and machine consumers of the API to +get more detailed information for the cause of the failure. The information in +the status object supplements, but does not override, the HTTP status code's +meaning. When fields in the status object have the same meaning as generally +defined HTTP headers and that header is returned with the response, the header +should be considered as having higher priority. + +**Example:** + +```console +$ curl -v -k -H "Authorization: Bearer WhCDvq4VPpYhrcfmF6ei7V9qlbqTubUc" https://10.240.122.184:443/api/v1/namespaces/default/pods/grafana + +> GET /api/v1/namespaces/default/pods/grafana HTTP/1.1 +> User-Agent: curl/7.26.0 +> Host: 10.240.122.184 +> Accept: */* +> Authorization: Bearer WhCDvq4VPpYhrcfmF6ei7V9qlbqTubUc +> + +< HTTP/1.1 404 Not Found +< Content-Type: application/json +< Date: Wed, 20 May 2015 18:10:42 GMT +< Content-Length: 232 +< +{ + "kind": "Status", + "apiVersion": "v1", + "metadata": {}, + "status": "Failure", + "message": "pods \"grafana\" not found", + "reason": "NotFound", + "details": { + "name": "grafana", + "kind": "pods" + }, + "code": 404 +} +``` + +`status` field contains one of two possible values: +* `Success` +* `Failure` + +`message` may contain human-readable description of the error + +`reason` may contain a machine-readable, one-word, CamelCase description of why +this operation is in the `Failure` status. If this value is empty there is no +information available. The `reason` clarifies an HTTP status code but does not +override it. + +`details` may contain extended data associated with the reason. Each reason may +define its own extended details. This field is optional and the data returned is +not guaranteed to conform to any schema except that defined by the reason type. + +Possible values for the `reason` and `details` fields: +* `BadRequest` + * Indicates that the request itself was invalid, because the request doesn't +make any sense, for example deleting a read-only object. + * This is different than `status reason` `Invalid` above which indicates that +the API call could possibly succeed, but the data was invalid. + * API calls that return BadRequest can never succeed. + * Http status code: `400 StatusBadRequest` + + +* `Unauthorized` + * Indicates that the server can be reached and understood the request, but +refuses to take any further action without the client providing appropriate +authorization. If the client has provided authorization, this error indicates +the provided credentials are insufficient or invalid. + * Details (optional): + * `kind string` + * The kind attribute of the unauthorized resource (on some operations may +differ from the requested resource). + * `name string` + * The identifier of the unauthorized resource. + * HTTP status code: `401 StatusUnauthorized` + + +* `Forbidden` + * Indicates that the server can be reached and understood the request, but +refuses to take any further action, because it is configured to deny access for +some reason to the requested resource by the client. + * Details (optional): + * `kind string` + * The kind attribute of the forbidden resource (on some operations may +differ from the requested resource). + * `name string` + * The identifier of the forbidden resource. + * HTTP status code: `403 StatusForbidden` + + +* `NotFound` + * Indicates that one or more resources required for this operation could not +be found. + * Details (optional): + * `kind string` + * The kind attribute of the missing resource (on some operations may +differ from the requested resource). + * `name string` + * The identifier of the missing resource. + * HTTP status code: `404 StatusNotFound` + + +* `AlreadyExists` + * Indicates that the resource you are creating already exists. + * Details (optional): + * `kind string` + * The kind attribute of the conflicting resource. + * `name string` + * The identifier of the conflicting resource. + * HTTP status code: `409 StatusConflict` + +* `Conflict` + * Indicates that the requested update operation cannot be completed due to a +conflict. The client may need to alter the request. Each resource may define +custom details that indicate the nature of the conflict. + * HTTP status code: `409 StatusConflict` + + +* `Invalid` + * Indicates that the requested create or update operation cannot be completed +due to invalid data provided as part of the request. + * Details (optional): + * `kind string` + * the kind attribute of the invalid resource + * `name string` + * the identifier of the invalid resource + * `causes` + * One or more `StatusCause` entries indicating the data in the provided +resource that was invalid. The `reason`, `message`, and `field` attributes will +be set. + * HTTP status code: `422 StatusUnprocessableEntity` + + +* `Timeout` + * Indicates that the request could not be completed within the given time. +Clients may receive this response if the server has decided to rate limit the +client, or if the server is overloaded and cannot process the request at this +time. + * Http status code: `429 TooManyRequests` + * The server should set the `Retry-After` HTTP header and return +`retryAfterSeconds` in the details field of the object. A value of `0` is the +default. + + +* `ServerTimeout` + * Indicates that the server can be reached and understood the request, but +cannot complete the action in a reasonable time. This maybe due to temporary +server load or a transient communication issue with another server. + * Details (optional): + * `kind string` + * The kind attribute of the resource being acted on. + * `name string` + * The operation that is being attempted. + * The server should set the `Retry-After` HTTP header and return +`retryAfterSeconds` in the details field of the object. A value of `0` is the +default. + * Http status code: `504 StatusServerTimeout` + + +* `MethodNotAllowed` + * Indicates that the action the client attempted to perform on the resource +was not supported by the code. + * For instance, attempting to delete a resource that can only be created. + * API calls that return MethodNotAllowed can never succeed. + * Http status code: `405 StatusMethodNotAllowed` + + +* `InternalError` + * Indicates that an internal error occurred, it is unexpected and the outcome +of the call is unknown. + * Details (optional): + * `causes` + * The original error. + * Http status code: `500 StatusInternalServerError` `code` may contain the suggested HTTP return code for this status. + + +## Events + +Events are complementary to status information, since they can provide some +historical information about status and occurrences in addition to current or +previous status. Generate events for situations users or administrators should +be alerted about. + +Choose a unique, specific, short, CamelCase reason for each event category. For +example, `FreeDiskSpaceInvalid` is a good event reason because it is likely to +refer to just one situation, but `Started` is not a good reason because it +doesn't sufficiently indicate what started, even when combined with other event +fields. + +`Error creating foo` or `Error creating foo %s` would be appropriate for an +event message, with the latter being preferable, since it is more informational. + +Accumulate repeated events in the client, especially for frequent events, to +reduce data volume, load on the system, and noise exposed to users. + +## Naming conventions + +* Go field names must be CamelCase. JSON field names must be camelCase. Other +than capitalization of the initial letter, the two should almost always match. +No underscores nor dashes in either. +* Field and resource names should be declarative, not imperative (DoSomething, +SomethingDoer, DoneBy, DoneAt). +* Use `Node` where referring to +the node resource in the context of the cluster. Use `Host` where referring to +properties of the individual physical/virtual system, such as `hostname`, +`hostPath`, `hostNetwork`, etc. +* `FooController` is a deprecated kind naming convention. Name the kind after +the thing being controlled instead (e.g., `Job` rather than `JobController`). +* The name of a field that specifies the time at which `something` occurs should +be called `somethingTime`. Do not use `stamp` (e.g., `creationTimestamp`). +* We use the `fooSeconds` convention for durations, as discussed in the [units +subsection](#units). + * `fooPeriodSeconds` is preferred for periodic intervals and other waiting +periods (e.g., over `fooIntervalSeconds`). + * `fooTimeoutSeconds` is preferred for inactivity/unresponsiveness deadlines. + * `fooDeadlineSeconds` is preferred for activity completion deadlines. +* Do not use abbreviations in the API, except where they are extremely commonly +used, such as "id", "args", or "stdin". +* Acronyms should similarly only be used when extremely commonly known. All +letters in the acronym should have the same case, using the appropriate case for +the situation. For example, at the beginning of a field name, the acronym should +be all lowercase, such as "httpGet". Where used as a constant, all letters +should be uppercase, such as "TCP" or "UDP". +* The name of a field referring to another resource of kind `Foo` by name should +be called `fooName`. The name of a field referring to another resource of kind +`Foo` by ObjectReference (or subset thereof) should be called `fooRef`. +* More generally, include the units and/or type in the field name if they could +be ambiguous and they are not specified by the value or value type. +* The name of a field expressing a boolean property called 'fooable' should be +called `Fooable`, not `IsFooable`. + +### Namespace Names +* The name of a namespace must be a +[DNS_LABEL](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/identifiers.md). +* The `kube-` prefix is reserved for Kubernetes system namespaces, e.g. `kube-system` and `kube-public`. +* See +[the namespace docs](https://kubernetes.io/docs/user-guide/namespaces/) for more information. + +## Label, selector, and annotation conventions + +Labels are the domain of users. They are intended to facilitate organization and +management of API resources using attributes that are meaningful to users, as +opposed to meaningful to the system. Think of them as user-created mp3 or email +inbox labels, as opposed to the directory structure used by a program to store +its data. The former enables the user to apply an arbitrary ontology, whereas +the latter is implementation-centric and inflexible. Users will use labels to +select resources to operate on, display label values in CLI/UI columns, etc. +Users should always retain full power and flexibility over the label schemas +they apply to labels in their namespaces. + +However, we should support conveniences for common cases by default. For +example, what we now do in ReplicationController is automatically set the RC's +selector and labels to the labels in the pod template by default, if they are +not already set. That ensures that the selector will match the template, and +that the RC can be managed using the same labels as the pods it creates. Note +that once we generalize selectors, it won't necessarily be possible to +unambiguously generate labels that match an arbitrary selector. + +If the user wants to apply additional labels to the pods that it doesn't select +upon, such as to facilitate adoption of pods or in the expectation that some +label values will change, they can set the selector to a subset of the pod +labels. Similarly, the RC's labels could be initialized to a subset of the pod +template's labels, or could include additional/different labels. + +For disciplined users managing resources within their own namespaces, it's not +that hard to consistently apply schemas that ensure uniqueness. One just needs +to ensure that at least one value of some label key in common differs compared +to all other comparable resources. We could/should provide a verification tool +to check that. However, development of conventions similar to the examples in +[Labels](https://kubernetes.io/docs/user-guide/labels/) make uniqueness straightforward. Furthermore, +relatively narrowly used namespaces (e.g., per environment, per application) can +be used to reduce the set of resources that could potentially cause overlap. + +In cases where users could be running misc. examples with inconsistent schemas, +or where tooling or components need to programmatically generate new objects to +be selected, there needs to be a straightforward way to generate unique label +sets. A simple way to ensure uniqueness of the set is to ensure uniqueness of a +single label value, such as by using a resource name, uid, resource hash, or +generation number. + +Problems with uids and hashes, however, include that they have no semantic +meaning to the user, are not memorable nor readily recognizable, and are not +predictable. Lack of predictability obstructs use cases such as creation of a +replication controller from a pod, such as people want to do when exploring the +system, bootstrapping a self-hosted cluster, or deletion and re-creation of a +new RC that adopts the pods of the previous one, such as to rename it. +Generation numbers are more predictable and much clearer, assuming there is a +logical sequence. Fortunately, for deployments that's the case. For jobs, use of +creation timestamps is common internally. Users should always be able to turn +off auto-generation, in order to permit some of the scenarios described above. +Note that auto-generated labels will also become one more field that needs to be +stripped out when cloning a resource, within a namespace, in a new namespace, in +a new cluster, etc., and will need to be ignored around when updating a resource +via patch or read-modify-write sequence. + +Inclusion of a system prefix in a label key is fairly hostile to UX. A prefix is +only necessary in the case that the user cannot choose the label key, in order +to avoid collisions with user-defined labels. However, I firmly believe that the +user should always be allowed to select the label keys to use on their +resources, so it should always be possible to override default label keys. + +Therefore, resources supporting auto-generation of unique labels should have a +`uniqueLabelKey` field, so that the user could specify the key if they wanted +to, but if unspecified, it could be set by default, such as to the resource +type, like job, deployment, or replicationController. The value would need to be +at least spatially unique, and perhaps temporally unique in the case of job. + +Annotations have very different intended usage from labels. They are +primarily generated and consumed by tooling and system extensions, or are used +by end-users to engage non-standard behavior of components. For example, an +annotation might be used to indicate that an instance of a resource expects +additional handling by non-kubernetes controllers. Annotations may carry +arbitrary payloads, including JSON documents. Like labels, annotation keys can +be prefixed with a governing domain (e.g. `example.com/key-name`). Unprefixed +keys (e.g. `key-name`) are reserved for end-users. Third-party components must +use prefixed keys. Key prefixes under the "kubernetes.io" and "k8s.io" domains +are reserved for use by the kubernetes project and must not be used by +third-parties. + +In early versions of Kubernetes, some in-development features represented new +API fields as annotations, generally with the form `something.alpha.kubernetes.io/name` or +`something.beta.kubernetes.io/name` (depending on our confidence in it). This +pattern is deprecated. Some such annotations may still exist, but no new +annotations may be defined. New API fields are now developed as regular fields. + +Other advice regarding use of labels, annotations, taints, and other generic map keys by +Kubernetes components and tools: + - Key names should be all lowercase, with words separated by dashes instead of camelCase + - For instance, prefer `foo.kubernetes.io/foo-bar` over `foo.kubernetes.io/fooBar`, prefer + `desired-replicas` over `DesiredReplicas` + - Unprefixed keys are reserved for end-users. All other labels and annotations must be prefixed. + - Key prefixes under "kubernetes.io" and "k8s.io" are reserved for the Kubernetes + project. + - Such keys are effectively part of the kubernetes API and may be subject + to deprecation and compatibility policies. + - Key names, including prefixes, should be precise enough that a user could + plausibly understand where it came from and what it is for. + - Key prefixes should carry as much context as possible. + - For instance, prefer `subsystem.kubernetes.io/parameter` over `kubernetes.io/subsystem-parameter` + - Use annotations to store API extensions that the controller responsible for +the resource doesn't need to know about, experimental fields that aren't +intended to be generally used API fields, etc. Beware that annotations aren't +automatically handled by the API conversion machinery. + +## WebSockets and SPDY + +Some of the API operations exposed by Kubernetes involve transfer of binary +streams between the client and a container, including attach, exec, portforward, +and logging. The API therefore exposes certain operations over upgradeable HTTP +connections ([described in RFC 2817](https://tools.ietf.org/html/rfc2817)) via +the WebSocket and SPDY protocols. These actions are exposed as subresources with +their associated verbs (exec, log, attach, and portforward) and are requested +via a GET (to support JavaScript in a browser) and POST (semantically accurate). + +There are two primary protocols in use today: + +1. Streamed channels + + When dealing with multiple independent binary streams of data such as the +remote execution of a shell command (writing to STDIN, reading from STDOUT and +STDERR) or forwarding multiple ports the streams can be multiplexed onto a +single TCP connection. Kubernetes supports a SPDY based framing protocol that +leverages SPDY channels and a WebSocket framing protocol that multiplexes +multiple channels onto the same stream by prefixing each binary chunk with a +byte indicating its channel. The WebSocket protocol supports an optional +subprotocol that handles base64-encoded bytes from the client and returns +base64-encoded bytes from the server and character based channel prefixes ('0', +'1', '2') for ease of use from JavaScript in a browser. + +2. Streaming response + + The default log output for a channel of streaming data is an HTTP Chunked +Transfer-Encoding, which can return an arbitrary stream of binary data from the +server. Browser-based JavaScript is limited in its ability to access the raw +data from a chunked response, especially when very large amounts of logs are +returned, and in future API calls it may be desirable to transfer large files. +The streaming API endpoints support an optional WebSocket upgrade that provides +a unidirectional channel from the server to the client and chunks data as binary +WebSocket frames. An optional WebSocket subprotocol is exposed that base64 +encodes the stream before returning it to the client. + +Clients should use the SPDY protocols if their clients have native support, or +WebSockets as a fallback. Note that WebSockets is susceptible to Head-of-Line +blocking and so clients must read and process each message sequentially. In +the future, an HTTP/2 implementation will be exposed that deprecates SPDY. + + +## Validation + +API objects are validated upon receipt by the apiserver. Validation errors are +flagged and returned to the caller in a `Failure` status with `reason` set to +`Invalid`. In order to facilitate consistent error messages, we ask that +validation logic adheres to the following guidelines whenever possible (though +exceptional cases will exist). + +* Be as precise as possible. +* Telling users what they CAN do is more useful than telling them what they +CANNOT do. +* When asserting a requirement in the positive, use "must". Examples: "must be +greater than 0", "must match regex '[a-z]+'". Words like "should" imply that +the assertion is optional, and must be avoided. +* When asserting a formatting requirement in the negative, use "must not". +Example: "must not contain '..'". Words like "should not" imply that the +assertion is optional, and must be avoided. +* When asserting a behavioral requirement in the negative, use "may not". +Examples: "may not be specified when otherField is empty", "only `name` may be +specified". +* When referencing a literal string value, indicate the literal in +single-quotes. Example: "must not contain '..'". +* When referencing another field name, indicate the name in back-quotes. +Example: "must be greater than `request`". +* When specifying inequalities, use words rather than symbols. Examples: "must +be less than 256", "must be greater than or equal to 0". Do not use words +like "larger than", "bigger than", "more than", "higher than", etc. +* When specifying numeric ranges, use inclusive ranges when possible. + diff --git a/contributors/devel/sig-architecture/api_changes.md b/contributors/devel/sig-architecture/api_changes.md new file mode 100644 index 000000000..1f46d298b --- /dev/null +++ b/contributors/devel/sig-architecture/api_changes.md @@ -0,0 +1,1007 @@ +*This document is oriented at developers who want to change existing APIs. +A set of API conventions, which applies to new APIs and to changes, can be +found at [API Conventions](api-conventions.md). + +**Table of Contents** + +- [So you want to change the API?](#so-you-want-to-change-the-api) + - [Operational overview](#operational-overview) + - [On compatibility](#on-compatibility) + - [Backward compatibility gotchas](#backward-compatibility-gotchas) + - [Incompatible API changes](#incompatible-api-changes) + - [Changing versioned APIs](#changing-versioned-apis) + - [Edit types.go](#edit-typesgo) + - [Edit defaults.go](#edit-defaultsgo) + - [Edit conversion.go](#edit-conversiongo) + - [Changing the internal structures](#changing-the-internal-structures) + - [Edit types.go](#edit-typesgo-1) + - [Edit validation.go](#edit-validationgo) + - [Edit version conversions](#edit-version-conversions) + - [Generate protobuf objects](#generate-protobuf-objects) + - [Edit json (un)marshaling code](#edit-json-unmarshaling-code) + - [Making a new API Version](#making-a-new-api-version) + - [Making a new API Group](#making-a-new-api-group) + - [Update the fuzzer](#update-the-fuzzer) + - [Update the semantic comparisons](#update-the-semantic-comparisons) + - [Implement your change](#implement-your-change) + - [Write end-to-end tests](#write-end-to-end-tests) + - [Examples and docs](#examples-and-docs) + - [Alpha, Beta, and Stable Versions](#alpha-beta-and-stable-versions) + - [Adding Unstable Features to Stable Versions](#adding-unstable-features-to-stable-versions) + + +# So you want to change the API? + +Before attempting a change to the API, you should familiarize yourself with a +number of existing API types and with the [API conventions](api-conventions.md). +If creating a new API type/resource, we also recommend that you first send a PR +containing just a proposal for the new API types. + +The Kubernetes API has two major components - the internal structures and +the versioned APIs. The versioned APIs are intended to be stable, while the +internal structures are implemented to best reflect the needs of the Kubernetes +code itself. + +What this means for API changes is that you have to be somewhat thoughtful in +how you approach changes, and that you have to touch a number of pieces to make +a complete change. This document aims to guide you through the process, though +not all API changes will need all of these steps. + +## Operational overview + +It is important to have a high level understanding of the API system used in +Kubernetes in order to navigate the rest of this document. + +As mentioned above, the internal representation of an API object is decoupled +from any one API version. This provides a lot of freedom to evolve the code, +but it requires robust infrastructure to convert between representations. There +are multiple steps in processing an API operation - even something as simple as +a GET involves a great deal of machinery. + +The conversion process is logically a "star" with the internal form at the +center. Every versioned API can be converted to the internal form (and +vice-versa), but versioned APIs do not convert to other versioned APIs directly. +This sounds like a heavy process, but in reality we do not intend to keep more +than a small number of versions alive at once. While all of the Kubernetes code +operates on the internal structures, they are always converted to a versioned +form before being written to storage (disk or etcd) or being sent over a wire. +Clients should consume and operate on the versioned APIs exclusively. + +To demonstrate the general process, here is a (hypothetical) example: + + 1. A user POSTs a `Pod` object to `/api/v7beta1/...` + 2. The JSON is unmarshalled into a `v7beta1.Pod` structure + 3. Default values are applied to the `v7beta1.Pod` + 4. The `v7beta1.Pod` is converted to an `api.Pod` structure + 5. The `api.Pod` is validated, and any errors are returned to the user + 6. The `api.Pod` is converted to a `v6.Pod` (because v6 is the latest stable +version) + 7. The `v6.Pod` is marshalled into JSON and written to etcd + +Now that we have the `Pod` object stored, a user can GET that object in any +supported api version. For example: + + 1. A user GETs the `Pod` from `/api/v5/...` + 2. The JSON is read from etcd and unmarshalled into a `v6.Pod` structure + 3. Default values are applied to the `v6.Pod` + 4. The `v6.Pod` is converted to an `api.Pod` structure + 5. The `api.Pod` is converted to a `v5.Pod` structure + 6. The `v5.Pod` is marshalled into JSON and sent to the user + +The implication of this process is that API changes must be done carefully and +backward-compatibly. + +## On compatibility + +Before talking about how to make API changes, it is worthwhile to clarify what +we mean by API compatibility. Kubernetes considers forwards and backwards +compatibility of its APIs a top priority. Compatibility is *hard*, especially +handling issues around rollback-safety. This is something every API change +must consider. + +An API change is considered compatible if it: + + * adds new functionality that is not required for correct behavior (e.g., +does not add a new required field) + * does not change existing semantics, including: + * the semantic meaning of default values *and behavior* + * interpretation of existing API types, fields, and values + * which fields are required and which are not + * mutable fields do not become immutable + * valid values do not become invalid + * explicitly invalid values do not become valid + +Put another way: + +1. Any API call (e.g. a structure POSTed to a REST endpoint) that succeeded +before your change must succeed after your change. +2. Any API call that does not use your change must behave the same as it did +before your change. +3. Any API call that uses your change must not cause problems (e.g. crash or +degrade behavior) when issued against an API servers that do not include your +change. +4. It must be possible to round-trip your change (convert to different API +versions and back) with no loss of information. +5. Existing clients need not be aware of your change in order for them to +continue to function as they did previously, even when your change is in use. +6. It must be possible to rollback to a previous version of API server that +does not include your change and have no impact on API objects which do not use +your change. API objects that use your change will be impacted in case of a +rollback. + +If your change does not meet these criteria, it is not considered compatible, +and may break older clients, or result in newer clients causing undefined +behavior. Such changes are generally disallowed, though exceptions have been +made in extreme cases (e.g. security or obvious bugs). + +Let's consider some examples. + +In a hypothetical API (assume we're at version v6), the `Frobber` struct looks +something like this: + +```go +// API v6. +type Frobber struct { + Height int `json:"height"` + Param string `json:"param"` +} +``` + +You want to add a new `Width` field. It is generally allowed to add new fields +without changing the API version, so you can simply change it to: + +```go +// Still API v6. +type Frobber struct { + Height int `json:"height"` + Width int `json:"width"` + Param string `json:"param"` +} +``` + +The onus is on you to define a sane default value for `Width` such that rules +#1 and #2 above are true - API calls and stored objects that used to work must +continue to work. + +For your next change you want to allow multiple `Param` values. You can not +simply remove `Param string` and add `Params []string` (without creating a +whole new API version) - that fails rules #1, #2, #3, and #6. Nor can you +simply add `Params []string` and use it instead - that fails #2 and #6. + +You must instead define a new field and the relationship between that field and +the existing field(s). Start by adding the new plural field: + +```go +// Still API v6. +type Frobber struct { + Height int `json:"height"` + Width int `json:"width"` + Param string `json:"param"` // the first param + Params []string `json:"params"` // all of the params +} +``` + +This new field must be inclusive of the singular field. In order to satisfy +the compatibility rules you must handle all the cases of version skew, multiple +clients, and rollbacks. This can be handled by defaulting or admission control +logic linking the fields together with context from the API operation to get as +close as possible to the user's intentions. + +Upon any mutating API operation: + * If only the singular field is specified (e.g. an older client), API logic + must populate plural[0] from the singular value, and de-dup the plural + field. + * If only the plural field is specified (e.g. a newer client), API logic must + populate the singular value from plural[0]. + * If both the singular and plural fields are specified, API logic must + validate that the singular value matches plural[0]. + * Any other case is an error and must be rejected. + +For this purpose "is specified" means the following: + * On a create or patch operation: the field is present in the user-provided input + * On an update operation: the field is present and has changed from the + current value + +Older clients that only know the singular field will continue to succeed and +produce the same results as before the change. Newer clients can use your +change without impacting older clients. The API server can be rolled back and +only objects that use your change will be impacted. + +Part of the reason for versioning APIs and for using internal types that are +distinct from any one version is to handle growth like this. The internal +representation can be implemented as: + +```go +// Internal, soon to be v7beta1. +type Frobber struct { + Height int + Width int + Params []string +} +``` + +The code that converts to/from versioned APIs can decode this into the +compatible structure. Eventually, a new API version, e.g. v7beta1, +will be forked and it can drop the singular field entirely. + +We've seen how to satisfy rules #1, #2, and #3. Rule #4 means that you can not +extend one versioned API without also extending the others. For example, an +API call might POST an object in API v7beta1 format, which uses the cleaner +`Params` field, but the API server might store that object in trusty old v6 +form (since v7beta1 is "beta"). When the user reads the object back in the +v7beta1 API it would be unacceptable to have lost all but `Params[0]`. This +means that, even though it is ugly, a compatible change must be made to the v6 +API, as above. + +For some changes, this can be challenging to do correctly. It may require multiple +representations of the same information in the same API resource, which need to +be kept in sync should either be changed. + +For example, let's say you decide to rename a field within the same API +version. In this case, you add units to `height` and `width`. You implement +this by adding new fields: + +```go +type Frobber struct { + Height *int `json:"height"` + Width *int `json:"width"` + HeightInInches *int `json:"heightInInches"` + WidthInInches *int `json:"widthInInches"` +} +``` + +You convert all of the fields to pointers in order to distinguish between unset +and set to 0, and then set each corresponding field from the other in the +defaulting logic (e.g. `heightInInches` from `height`, and vice versa). That +works fine when the user creates a sends a hand-written configuration -- +clients can write either field and read either field. + +But what about creation or update from the output of a GET, or update via PATCH +(see [In-place updates](https://kubernetes.io/docs/user-guide/managing-deployments/#in-place-updates-of-resources))? +In these cases, the two fields will conflict, because only one field would be +updated in the case of an old client that was only aware of the old field +(e.g. `height`). + +Suppose the client creates: + +```json +{ + "height": 10, + "width": 5 +} +``` + +and GETs: + +```json +{ + "height": 10, + "heightInInches": 10, + "width": 5, + "widthInInches": 5 +} +``` + +then PUTs back: + +```json +{ + "height": 13, + "heightInInches": 10, + "width": 5, + "widthInInches": 5 +} +``` + +As per the compatibility rules, the update must not fail, because it would have +worked before the change. + +## Backward compatibility gotchas + +* A single feature/property cannot be represented using multiple spec fields + simultaneously within an API version. Only one representation can be + populated at a time, and the client needs to be able to specify which field + they expect to use (typically via API version), on both mutation and read. As + above, older clients must continue to function properly. + +* A new representation, even in a new API version, that is more expressive than an + old one breaks backward compatibility, since clients that only understood the + old representation would not be aware of the new representation nor its + semantics. Examples of proposals that have run into this challenge include + [generalized label selectors](http://issues.k8s.io/341) and [pod-level security context](http://prs.k8s.io/12823). + +* Enumerated values cause similar challenges. Adding a new value to an enumerated set + is *not* a compatible change. Clients which assume they know how to handle all possible + values of a given field will not be able to handle the new values. However, removing a + value from an enumerated set *can* be a compatible change, if handled properly (treat the + removed value as deprecated but allowed). For enumeration-like fields that expect to add + new values in the future, such as `reason` fields, please document that expectation clearly + in the API field descriptions. Clients should treat such sets of values as potentially + open-ended. + +* For [Unions](api-conventions.md#unions), sets of fields where at most one should + be set, it is acceptable to add a new option to the union if the [appropriate + conventions](api-conventions.md#objects) were followed in the original object. + Removing an option requires following the [deprecation process](https://kubernetes.io/docs/reference/deprecation-policy/). + +* Changing any validation rules always has the potential of breaking some client, since it changes the + assumptions about part of the API, similar to adding new enum values. Validation rules on spec fields can + neither be relaxed nor strengthened. Strengthening cannot be permitted because any requests that previously + worked must continue to work. Weakening validation has the potential to break other consumers and generators + of the API resource. Status fields whose writers are under our control (e.g., written by non-pluggable + controllers), may potentially tighten validation, since that would cause a subset of previously valid + values to be observable by clients. + +* Do not add a new API version of an existing resource and make it the preferred version in the same + release, and do not make it the storage version. The latter is necessary so that a rollback of the + apiserver doesn't render resources in etcd undecodable after rollback. + +* Any field with a default value in one API version must have a *non-nil* default + value in all API versions. This can be split into 2 cases: + * Adding a new API version with a default value for an existing non-defaulted + field: it is required to add a default value semantically equivalent to + being unset in all previous API versions, to preserve the semantic meaning + of the value being unset. + * Adding a new field with a default value: the default values must be + semantically equivalent in all currently supported API versions. + +## Incompatible API changes + +There are times when incompatible changes might be OK, but mostly we want +changes that meet the above definitions. If you think you need to break +compatibility, you should talk to the Kubernetes API reviewers first. + +Breaking compatibility of a beta or stable API version, such as v1, is +unacceptable. Compatibility for experimental or alpha APIs is not strictly +required, but breaking compatibility should not be done lightly, as it disrupts +all users of the feature. Alpha and beta API versions may be deprecated and +eventually removed wholesale, as described in the [deprecation policy](https://kubernetes.io/docs/reference/deprecation-policy/). + +If your change is going to be backward incompatible or might be a breaking +change for API consumers, please send an announcement to +`kubernetes-dev@googlegroups.com` before the change gets in. If you are unsure, +ask. Also make sure that the change gets documented in the release notes for the +next release by labeling the PR with the "release-note-action-required" github label. + +If you found that your change accidentally broke clients, it should be reverted. + +In short, the expected API evolution is as follows: + +* `newapigroup/v1alpha1` -> ... -> `newapigroup/v1alphaN` -> +* `newapigroup/v1beta1` -> ... -> `newapigroup/v1betaN` -> +* `newapigroup/v1` -> +* `newapigroup/v2alpha1` -> ... + +While in alpha we expect to move forward with it, but may break it. + +Once in beta we will preserve forward compatibility, but may introduce new +versions and delete old ones. + +v1 must be backward-compatible for an extended length of time. + +## Changing versioned APIs + +For most changes, you will probably find it easiest to change the versioned +APIs first. This forces you to think about how to make your change in a +compatible way. Rather than doing each step in every version, it's usually +easier to do each versioned API one at a time, or to do all of one version +before starting "all the rest". + +### Edit types.go + +The struct definitions for each API are in +`staging/src/k8s.io/api///types.go`. Edit those files to reflect +the change you want to make. Note that all types and non-inline fields in +versioned APIs must be preceded by descriptive comments - these are used to +generate documentation. Comments for types should not contain the type name; API +documentation is generated from these comments and end-users should not be +exposed to golang type names. + +For types that need the generated +[DeepCopyObject](https://github.com/kubernetes/kubernetes/commit/8dd0989b395b29b872e1f5e06934721863e4a210#diff-6318847735efb6fae447e7dbf198c8b2R3767) +methods, usually only required by the top-level types like `Pod`, add this line +to the comment +([example](https://github.com/kubernetes/kubernetes/commit/39d95b9b065fffebe5b6f233d978fe1723722085#diff-ab819c2e7a94a3521aecf6b477f9b2a7R30)): + +```golang + // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +``` + +Optional fields should have the `,omitempty` json tag; fields are interpreted as +being required otherwise. + +### Edit defaults.go + +If your change includes new fields for which you will need default values, you +need to add cases to `pkg/apis///defaults.go`. + +**Note:** When adding default values to new fields, you *must* also add default +values in all API versions, instead of leaving new fields unset (e.g. `nil`) in +old API versions. This is required because defaulting happens whenever a +serialized version is read (see [#66135]). When possible, pick meaningful values +as sentinels for unset values. + +In the past the core v1 API +was special. Its `defaults.go` used to live at `pkg/api/v1/defaults.go`. +If you see code referencing that path, you can be sure its outdated. Now the core v1 api lives at +`pkg/apis/core/v1/defaults.go` which follows the above convention. + +Of course, since you have added code, you have to add a test: +`pkg/apis///defaults_test.go`. + +Do use pointers to scalars when you need to distinguish between an unset value +and an automatic zero value. For example, +`PodSpec.TerminationGracePeriodSeconds` is defined as `*int64` the go type +definition. A zero value means 0 seconds, and a nil value asks the system to +pick a default. + +Don't forget to run the tests! + +[#66135]: https://github.com/kubernetes/kubernetes/issues/66135 + +### Edit conversion.go + +Given that you have not yet changed the internal structs, this might feel +premature, and that's because it is. You don't yet have anything to convert to +or from. We will revisit this in the "internal" section. If you're doing this +all in a different order (i.e. you started with the internal structs), then you +should jump to that topic below. In the very rare case that you are making an +incompatible change you might or might not want to do this now, but you will +have to do more later. The files you want are +`pkg/apis///conversion.go` and +`pkg/apis///conversion_test.go`. + +Note that the conversion machinery doesn't generically handle conversion of +values, such as various kinds of field references and API constants. [The client +library](https://github.com/kubernetes/client-go/blob/v4.0.0-beta.0/rest/request.go#L352) +has custom conversion code for field references. You also need to add a call to +`AddFieldLabelConversionFunc` of your scheme with a mapping function that +understands supported translations, like this +[line](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/api/v1/conversion.go#L165). + +## Changing the internal structures + +Now it is time to change the internal structs so your versioned changes can be +used. + +### Edit types.go + +Similar to the versioned APIs, the definitions for the internal structs are in +`pkg/apis//types.go`. Edit those files to reflect the change you want to +make. Keep in mind that the internal structs must be able to express *all* of +the versioned APIs. + +Similar to the versioned APIs, you need to add the `+k8s:deepcopy-gen` tag to +types that need generated DeepCopyObject methods. + +## Edit validation.go + +Most changes made to the internal structs need some form of input validation. +Validation is currently done on internal objects in +`pkg/apis//validation/validation.go`. This validation is the one of the +first opportunities we have to make a great user experience - good error +messages and thorough validation help ensure that users are giving you what you +expect and, when they don't, that they know why and how to fix it. Think hard +about the contents of `string` fields, the bounds of `int` fields and the +optionality of fields. + +Of course, code needs tests - `pkg/apis//validation/validation_test.go`. + +## Edit version conversions + +At this point you have both the versioned API changes and the internal +structure changes done. If there are any notable differences - field names, +types, structural change in particular - you must add some logic to convert +versioned APIs to and from the internal representation. If you see errors from +the `serialization_test`, it may indicate the need for explicit conversions. + +Performance of conversions very heavily influence performance of apiserver. +Thus, we are auto-generating conversion functions that are much more efficient +than the generic ones (which are based on reflections and thus are highly +inefficient). + +The conversion code resides with each versioned API. There are two files: + + - `pkg/apis///conversion.go` containing manually written + conversion functions + - `pkg/apis///zz_generated.conversion.go` containing + auto-generated conversion functions + +Since auto-generated conversion functions are using manually written ones, +those manually written should be named with a defined convention, i.e. a +function converting type `X` in pkg `a` to type `Y` in pkg `b`, should be named: +`convert_a_X_To_b_Y`. + +Also note that you can (and for efficiency reasons should) use auto-generated +conversion functions when writing your conversion functions. + +Adding manually written conversion also requires you to add tests to +`pkg/apis///conversion_test.go`. + +Once all the necessary manually written conversions are added, you need to +regenerate auto-generated ones. To regenerate them run: + +```sh +make clean && make generated_files +``` + +`make clean` is important, otherwise the generated files might be stale, because +the build system uses custom cache. + +`make all` will invoke `make generated_files` as well. + +The `make generated_files` will also regenerate the `zz_generated.deepcopy.go`, +`zz_generated.defaults.go`, and `api/openapi-spec/swagger.json`. + +If regeneration is somehow not possible due to compile errors, the easiest +workaround is to remove the files causing errors and rerun the command. + +## Generate Code + +Apart from the `defaulter-gen`, `deepcopy-gen`, `conversion-gen` and +`openapi-gen`, there are a few other generators: + - `go-to-protobuf` + - `client-gen` + - `lister-gen` + - `informer-gen` + - `codecgen` (for fast json serialization with ugorji codec) + +Many of the generators are based on +[`gengo`](https://github.com/kubernetes/gengo) and share common +flags. The `--verify-only` flag will check the existing files on disk +and fail if they are not what would have been generated. + +The generators that create go code have a `--go-header-file` flag +which should be a file that contains the header that should be +included. This header is the copyright that should be present at the +top of the generated file and should be checked with the +[`repo-infra/verify/verify-boilerplane.sh`](https://git.k8s.io/repo-infra/verify/verify-boilerplate.sh) +script at a later stage of the build. + +To invoke these generators, you can run `make update`, which runs a bunch of +[scripts](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/hack/update-all.sh#L63-L78). +Please continue to read the next a few sections, because some generators have +prerequisites, also because they introduce how to invoke the generators +individually if you find `make update` takes too long to run. + +### Generate protobuf objects + +For any core API object, we also need to generate the Protobuf IDL and marshallers. +That generation is invoked with + +```sh +hack/update-generated-protobuf.sh +``` + +The vast majority of objects will not need any consideration when converting +to protobuf, but be aware that if you depend on a Golang type in the standard +library there may be additional work required, although in practice we typically +use our own equivalents for JSON serialization. The `pkg/api/serialization_test.go` +will verify that your protobuf serialization preserves all fields - be sure to +run it several times to ensure there are no incompletely calculated fields. + +### Generate Clientset + +`client-gen` is a tool to generate clientsets for top-level API objects. + +`client-gen` requires the `// +genclient` annotation on each +exported type in both the internal `pkg/apis//types.go` as well as each +specifically versioned `staging/src/k8s.io/api///types.go`. + +If the apiserver hosts your API under a different group name than the `` +in the filesystem, (usually this is because the `` in the filesystem +omits the "k8s.io" suffix, e.g., admission vs. admission.k8s.io), you can +instruct the `client-gen` to use the correct group name by adding the `// ++groupName=` annotation in the `doc.go` in both the internal +`pkg/apis//doc.go` as well as in each specifically versioned +`staging/src/k8s.io/api///types.go`. + +Once you added the annotations, generate the client with + +```sh +hack/update-codegen.sh +``` + +Note that you can use the optional `// +groupGoName=` to specify a CamelCase +custom Golang identifier to de-conflict e.g. `policy.authorization.k8s.io` and +`policy.k8s.io`. These two would both map to `Policy()` in clientsets. + +client-gen is flexible. See [this document](generating-clientset.md) if you need +client-gen for non-kubernetes API. + +### Generate Listers + +`lister-gen` is a tool to generate listers for a client. It reuses the +`//+genclient` and the `// +groupName=` annotations, so you do not need to +specify extra annotations. + +Your previous run of `hack/update-codegen.sh` has invoked `lister-gen`. + +### Generate Informers + +`informer-gen` generates the very useful Informers which watch API +resources for changes. It reuses the `//+genclient` and the +`//+groupName=` annotations, so you do not need to specify extra annotations. + +Your previous run of `hack/update-codegen.sh` has invoked `informer-gen`. + +### Edit json (un)marshaling code + +We are auto-generating code for marshaling and unmarshaling json representation +of api objects - this is to improve the overall system performance. + +The auto-generated code resides with each versioned API: + + - `staging/src/k8s.io/api///generated.proto` + - `staging/src/k8s.io/api///generated.pb.go` + +To regenerate them run: + +```sh +hack/update-generated-protobuf.sh +``` + +## Making a new API Version + +This section is under construction, as we make the tooling completely generic. + +If you are adding a new API version to an existing group, you can copy the +structure of the existing `pkg/apis//` and +`staging/src/k8s.io/api//` directories. + +Due to the fast changing nature of the project, the following content is probably out-dated: +* You can control if the version is enabled by default by update +[pkg/master/master.go](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/master/master.go#L381). +* You must add the new version to + [pkg/apis/group_name/install/install.go](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/apps/install/install.go). +* You must add the new version to + [hack/lib/init.sh#KUBE_AVAILABLE_GROUP_VERSIONS](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/hack/lib/init.sh#L53). +* You must add the new version to + [hack/update-generated-protobuf-dockerized.sh](https://github.com/kubernetes/kubernetes/blob/v1.8.2/hack/update-generated-protobuf-dockerized.sh#L44) + to generate protobuf IDL and marshallers. +* You must add the new version to + [cmd/kube-apiserver/app#apiVersionPriorities](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/cmd/kube-apiserver/app/aggregator.go#L172) +* You must setup storage for the new version in + [pkg/registry/group_name/rest](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/registry/authentication/rest/storage_authentication.go) + +You need to regenerate the generated code as instructed in the sections above. + +## Making a new API Group + +You'll have to make a new directory under `pkg/apis/` and +`staging/src/k8s.io/api`; copy the directory structure of an existing API group, +e.g. `pkg/apis/authentication` and `staging/src/k8s.io/api/authentication`; +replace "authentication" with your group name and replace versions with your +versions; replace the API kinds in +[versioned](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/staging/src/k8s.io/api/authentication/v1/register.go#L47) +and +[internal](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/authentication/register.go#L47) +register.go, and +[install.go](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/authentication/install/install.go#L43) +with your kinds. + +You'll have to add your API group/version to a few places in the code base, as +noted in [Making a new API Version](#making-a-new-api-version) section. + +You need to regenerate the generated code as instructed in the sections above. + +## Update the fuzzer + +Part of our testing regimen for APIs is to "fuzz" (fill with random values) API +objects and then convert them to and from the different API versions. This is +a great way of exposing places where you lost information or made bad +assumptions. If you have added any fields which need very careful formatting +(the test does not run validation) or if you have made assumptions such as +"this slice will always have at least 1 element", you may get an error or even +a panic from the `serialization_test`. If so, look at the diff it produces (or +the backtrace in case of a panic) and figure out what you forgot. Encode that +into the fuzzer's custom fuzz functions. Hint: if you added defaults for a +field, that field will need to have a custom fuzz function that ensures that the +field is fuzzed to a non-empty value. + +The fuzzer can be found in `pkg/api/testing/fuzzer.go`. + +## Update the semantic comparisons + +VERY VERY rarely is this needed, but when it hits, it hurts. In some rare cases +we end up with objects (e.g. resource quantities) that have morally equivalent +values with different bitwise representations (e.g. value 10 with a base-2 +formatter is the same as value 0 with a base-10 formatter). The only way Go +knows how to do deep-equality is through field-by-field bitwise comparisons. +This is a problem for us. + +The first thing you should do is try not to do that. If you really can't avoid +this, I'd like to introduce you to our `apiequality.Semantic.DeepEqual` routine. +It supports custom overrides for specific types - you can find that in +`pkg/api/helper/helpers.go`. + +There's one other time when you might have to touch this: `unexported fields`. +You see, while Go's `reflect` package is allowed to touch `unexported fields`, +us mere mortals are not - this includes `apiequality.Semantic.DeepEqual`. +Fortunately, most of our API objects are "dumb structs" all the way down - all +fields are exported (start with a capital letter) and there are no unexported +fields. But sometimes you want to include an object in our API that does have +unexported fields somewhere in it (for example, `time.Time` has unexported fields). +If this hits you, you may have to touch the `apiequality.Semantic.DeepEqual` +customization functions. + +## Implement your change + +Now you have the API all changed - go implement whatever it is that you're +doing! + +## Write end-to-end tests + +Check out the [E2E docs](e2e-tests.md) for detailed information about how to +write end-to-end tests for your feature. + +## Examples and docs + +At last, your change is done, all unit tests pass, e2e passes, you're done, +right? Actually, no. You just changed the API. If you are touching an existing +facet of the API, you have to try *really* hard to make sure that *all* the +examples and docs are updated. There's no easy way to do this, due in part to +JSON and YAML silently dropping unknown fields. You're clever - you'll figure it +out. Put `grep` or `ack` to good use. + +If you added functionality, you should consider documenting it and/or writing +an example to illustrate your change. + +Make sure you update the swagger and OpenAPI spec by running: + +```sh +hack/update-swagger-spec.sh +hack/update-openapi-spec.sh +``` + +The API spec changes should be in a commit separate from your other changes. + +## Alpha, Beta, and Stable Versions + +New feature development proceeds through a series of stages of increasing +maturity: + +- Development level + - Object Versioning: no convention + - Availability: not committed to main kubernetes repo, and thus not available +in official releases + - Audience: other developers closely collaborating on a feature or +proof-of-concept + - Upgradeability, Reliability, Completeness, and Support: no requirements or +guarantees +- Alpha level + - Object Versioning: API version name contains `alpha` (e.g. `v1alpha1`) + - Availability: committed to main kubernetes repo; appears in an official +release; feature is disabled by default, but may be enabled by flag + - Audience: developers and expert users interested in giving early feedback on +features + - Completeness: some API operations, CLI commands, or UI support may not be +implemented; the API need not have had an *API review* (an intensive and +targeted review of the API, on top of a normal code review) + - Upgradeability: the object schema and semantics may change in a later +software release, without any provision for preserving objects in an existing +cluster; removing the upgradability concern allows developers to make rapid +progress; in particular, API versions can increment faster than the minor +release cadence and the developer need not maintain multiple versions; +developers should still increment the API version when object schema or +semantics change in an [incompatible way](#on-compatibility) + - Cluster Reliability: because the feature is relatively new, and may lack +complete end-to-end tests, enabling the feature via a flag might expose bugs +with destabilize the cluster (e.g. a bug in a control loop might rapidly create +excessive numbers of object, exhausting API storage). + - Support: there is *no commitment* from the project to complete the feature; +the feature may be dropped entirely in a later software release + - Recommended Use Cases: only in short-lived testing clusters, due to +complexity of upgradeability and lack of long-term support and lack of +upgradability. +- Beta level: + - Object Versioning: API version name contains `beta` (e.g. `v2beta3`) + - Availability: in official Kubernetes releases, and enabled by default + - Audience: users interested in providing feedback on features + - Completeness: all API operations, CLI commands, and UI support should be +implemented; end-to-end tests complete; the API has had a thorough API review +and is thought to be complete, though use during beta may frequently turn up API +issues not thought of during review + - Upgradeability: the object schema and semantics may change in a later +software release; when this happens, an upgrade path will be documented; in some +cases, objects will be automatically converted to the new version; in other +cases, a manual upgrade may be necessary; a manual upgrade may require downtime +for anything relying on the new feature, and may require manual conversion of +objects to the new version; when manual conversion is necessary, the project +will provide documentation on the process + - Cluster Reliability: since the feature has e2e tests, enabling the feature +via a flag should not create new bugs in unrelated features; because the feature +is new, it may have minor bugs + - Support: the project commits to complete the feature, in some form, in a +subsequent Stable version; typically this will happen within 3 months, but +sometimes longer; releases should simultaneously support two consecutive +versions (e.g. `v1beta1` and `v1beta2`; or `v1beta2` and `v1`) for at least one +minor release cycle (typically 3 months) so that users have enough time to +upgrade and migrate objects + - Recommended Use Cases: in short-lived testing clusters; in production +clusters as part of a short-lived evaluation of the feature in order to provide +feedback +- Stable level: + - Object Versioning: API version `vX` where `X` is an integer (e.g. `v1`) + - Availability: in official Kubernetes releases, and enabled by default + - Audience: all users + - Completeness: must have conformance tests, approved by SIG Architecture, +in the appropriate conformance profile (e.g., non-portable and/or optional +features may not be in the default profile) + - Upgradeability: only [strictly compatible](#on-compatibility) changes +allowed in subsequent software releases + - Cluster Reliability: high + - Support: API version will continue to be present for many subsequent +software releases; + - Recommended Use Cases: any + +### Adding Unstable Features to Stable Versions + +When adding a feature to an object which is already Stable, the new fields and +new behaviors need to meet the Stable level requirements. If these cannot be +met, then the new field cannot be added to the object. + +For example, consider the following object: + +```go +// API v6. +type Frobber struct { + // height ... + Height *int32 `json:"height" + // param ... + Param string `json:"param" +} +``` + +A developer is considering adding a new `Width` parameter, like this: + +```go +// API v6. +type Frobber struct { + // height ... + Height *int32 `json:"height" + // param ... + Param string `json:"param" + // width ... + Width *int32 `json:"width,omitempty" +} +``` + +However, the new feature is not stable enough to be used in a stable version +(`v6`). Some reasons for this might include: + +- the final representation is undecided (e.g. should it be called `Width` or `Breadth`?) +- the implementation is not stable enough for general use (e.g. the `Area()` routine sometimes overflows.) + +The developer cannot add the new field unconditionally until stability is met. However, +sometimes stability cannot be met until some users try the new feature, and some +users are only able or willing to accept a released version of Kubernetes. In +that case, the developer has a few options, both of which require staging work +over several releases. + +#### Alpha field in existing API version + +Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons: + +* They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819) +* They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in [backward compatibility gotchas](#backward-compatibility-gotchas)) + +The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default: + +1. Add a feature gate to the API server to control enablement of the new field (and associated function): + + In [staging/src/k8s.io/apiserver/pkg/features/kube_features.go](https://git.k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/features/kube_features.go): + + ```go + // owner: @you + // alpha: v1.11 + // + // Add multiple dimensions to frobbers. + Frobber2D utilfeature.Feature = "Frobber2D" + + var defaultKubernetesFeatureGates = map[utilfeature.Feature]utilfeature.FeatureSpec{ + ... + Frobber2D: {Default: false, PreRelease: utilfeature.Alpha}, + } + ``` + +2. Add the field to the API type: + + * ensure the field is [optional](api-conventions.md#optional-vs-required) + * add the `omitempty` struct tag + * add the `// +optional` comment tag + * ensure the field is entirely absent from API responses when empty (optional fields should be pointers, anyway) + * include details about the alpha-level in the field description + + ```go + // API v6. + type Frobber struct { + // height ... + Height int32 `json:"height"` + // param ... + Param string `json:"param"` + // width indicates how wide the object is. + // This field is alpha-level and is only honored by servers that enable the Frobber2D feature. + // +optional + Width *int32 `json:"width,omitempty"` + } + ``` + +3. Before persisting the object to storage, clear disabled alpha fields on create, +and on update if the existing object does not already have a value in the field. +This prevents new usage of the feature while it is disabled, while ensuring existing data is preserved. +The recommended place to do this is in the REST storage strategy's PrepareForCreate/PrepareForUpdate methods: + + ```go + func (frobberStrategy) PrepareForCreate(ctx genericapirequest.Context, obj runtime.Object) { + frobber := obj.(*api.Frobber) + + if !utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) { + frobber.Width = nil + } + } + + func (frobberStrategy) PrepareForUpdate(ctx genericapirequest.Context, obj, old runtime.Object) { + newFrobber := obj.(*api.Frobber) + oldFrobber := old.(*api.Frobber) + + if !utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) && oldFrobber.Width == nil { + newFrobber.Width = nil + } + } + ``` + +4. In validation, validate the field if present: + + ```go + func ValidateFrobber(f *api.Frobber, fldPath *field.Path) field.ErrorList { + ... + if f.Width != nil { + ... validation of width field ... + } + ... + } + ``` + +In future Kubernetes versions: + +* if the feature progresses to beta or stable status, the feature gate can be removed or be enabled by default. +* if the schema of the alpha field must change in an incompatible way, a new field name must be used. +* if the feature is abandoned, or the field name is changed, the field should be removed from the go struct, with a tombstone comment ensuring the field name and protobuf tag are not reused: + + ```go + // API v6. + type Frobber struct { + // height ... + Height int32 `json:"height" protobuf:"varint,1,opt,name=height"` + // param ... + Param string `json:"param" protobuf:"bytes,2,opt,name=param"` + + // +k8s:deprecated=width,protobuf=3 + } + ``` + +#### New alpha API version + +Another option is to introduce a new type with an new `alpha` or `beta` version +designator, like this: + +```go +// API v7alpha1 +type Frobber struct { + // height ... + Height *int32 `json:"height"` + // param ... + Param string `json:"param"` + // width ... + Width *int32 `json:"width,omitempty"` +} +``` + +The latter requires that all objects in the same API group as `Frobber` to be +replicated in the new version, `v7alpha1`. This also requires user to use a new +client which uses the other version. Therefore, this is not a preferred option. + +A related issue is how a cluster manager can roll back from a new version +with a new feature, that is already being used by users. See +https://github.com/kubernetes/kubernetes/issues/4855. diff --git a/contributors/devel/sig-architecture/component-config-conventions.md b/contributors/devel/sig-architecture/component-config-conventions.md new file mode 100644 index 000000000..d3fe10003 --- /dev/null +++ b/contributors/devel/sig-architecture/component-config-conventions.md @@ -0,0 +1,221 @@ +# Component Configuration Conventions + +# Objective + +This document concerns the configuration of Kubernetes system components (as +opposed to the configuration of user workloads running on Kubernetes). +Component configuration is a major operational burden for operators of +Kubernetes clusters. To date, much literature has been written on and much +effort expended to improve component configuration. Despite this, the state of +component configuration remains dissonant. This document attempts to aggregate +that literature and propose a set of guidelines that component owners can +follow to improve consistency across the project. + +# Background + +Currently, component configuration is primarily driven through command line +flags. Command line driven configuration poses certain problems which are +discussed below. Attempts to improve component configuration as a whole have +been slow to make progress and have petered out (ref componentconfig api group, +configmap driven config issues). Some component owners have made use case +specific improvements on a per-need basis. Various comments in issues recommend +subsets of best design practice but no coherent, complete story exists. + +## Pain Points of Current Configuration + +Flag based configuration has poor qualities such as: + +1. Flags exist in a flat namespace, hampering the ability to organize them and expose them in helpful documentation. --help becomes useless as a reference as the number of knobs grows. It's impossible to distinguish useful knobs from cruft. +1. Flags can't easily have different values for different instances of a class. To adjust the resync period in the informers of O(n) controllers requires O(n) different flags in a global namespace. +1. Changing a process's command line necessitates a binary restart. This negatively impacts availability. +1. Flags are unsuitable for passing confidential configuration. The command line of a process is available to unprivileged process running in the host pid namespace. +1. Flags are a public API but are unversioned and unversionable. +1. Many arguments against using global variables apply to flags. + +Configuration in general has poor qualities such as: + +1. Configuration changes have the same forward/backward compatibility requirements as releases but rollout/rollback of configuration largely untested. Examples of configuration changes that might break a cluster: kubelet CNI plugin, etcd storage version. +1. Configuration options often exist only to test a specific feature where the default is reasonable for all real use cases. Examples: many sync periods. +1. Configuration options often exist to defer a "hard" design decision and to pay forward the "TODO(someone-else): think critically". +1. Configuration options are often used to workaround deficiencies of the API. For example `--register-with-labels` and `--register-with-taints` could be solved with a node initializer, if initializers existed. +1. Configuration options often exist to take testing shortcuts. There is a mentality that because a feature is opt-in, it can be released as a flag without robust testing. +1. Configuration accumulates new knobs, knobs accumulate new behaviors, knobs are forgotten and bitrot reducing code quality over time. +1. Number of configuration options is inversely proportional to test coverage. The size of the configuration state space grows >O(2^n) with the number of configuration bits. A handful of states in that space are ever tested. +1. Configuration options hamper troubleshooting efforts. On github, users frequently file tickets from environments that are neither consistent nor reproducible. + +## Types Of Configuration + +Configuration can only come from three sources: + +1. Command line flags. +1. API types serialized and stored on disk. +1. API types serialized and stored in the kubernetes API. + +Configuration options can be partitioned along certain lines. To name a few +important partitions: + +1. Bootstrap: This is configuration that is required before the component can contact the API. Examples include the kubeconfig and the filepath to the kubeconfig. +1. Dynamic vs Static: Dynamic config is config that is expected to change as part of normal operations such as a scheduler configuration or a node entering maintenance mode. Static config is config that is unlikely to change over subsequent deployments and even releases of a component. +1. Shared vs Per-Instance: Per-Instance configuration is configuration whose value is unique to the instance that the node runs on (e.g. Kubelet's `--hostname-override`). +1. Feature Gates: Feature gates are configuration options that enable a feature that has been deemed unsafe to enable by default. +1. Request context dependent: Request context dependent config is config that should probably be scoped to an attribute of the request (such as the user). We do a pretty good job of keeping these out of config and in policy objects (e.g. Quota, RBAC) but we could do more (e.g. rate limits). +1. Environment information: This is configuration that is available through downwards and OS APIs, e.g. node name, pod name, number of cpus, IP address. + +# Requirements + +Desired qualities of a configuration solution: + +1. Secure: We need to control who can change configuration. We need to control who can read sensitive configuration. +1. Manageable: We need to control which instances of a component uses which configuration, especially when those instances differ in version. +1. Reliable: Configuration pushes should just work. If they fail, they should fail early in the rollout, rollback config if possible, and alert noisily. +1. Recoverable: We need to be able to update (e.g. rollback) configuration when a component is down. +1. Monitorable: Both humans and computers need to monitor configuration; humans through json interfaces like /configz, computers through interfaces like prometheus /streamz. Confidential configuration needs to be accounted for, but can also be useful to monitor in an unredacted or partially redacted (i.e. hashed) form. +1. Verifiable: We need to be able to verify that a configuration is good. We need to verify the integrity of the received configuration and we need to validate that the encoded configuration state is sensible. +1. Auditable: We need to be able to trace the origin of a configuration change. +1. Accountable: We need to correlate a configuration push with its impact to the system. We need to be able to do this at the time of the push and later when analyzing logs. +1. Available: We should avoid high frequency configuration updates that require service disruption. We need to take into account system component SLA. +1. Scalable: We need to support distributing configuration to O(10,000) components at our current supported scalability limits. +1. Consistent: There should exist conventions that hold across components. +1. Composable: We should favor composition of configuration sources over layering/templating/inheritance. +1. Normalized: Redundant specification of configuration data should be avoided. +1. Testable: We need to be able to test the system under many different configurations. We also need to test configuration changes, both dynamic changes and those that require process restarts. +1. Maintainable: We need to push back on ever increasing cyclomatic complexity in our codebase. Each if statement and function argument added to support a configuration option negatively impacts the maintainability of our code. +1. Evolvable: We need to be able to extend our configuration API like we extend our other user facing APIs. We need to hold our configuration API to the same SLA and deprecation policy of public facing APIs. (e.g. [dynamic admission control](https://github.com/kubernetes/community/pull/611) and [hooks](https://github.com/kubernetes/kubernetes/issues/3585)) + +These don't need to be implemented immediately but are good to keep in mind. At +some point these should be ranked by priority and implemented. + +# Two Part Solution: + +## Part 1: Don't Make It Configuration + +The most effective way to reduce the operational burden of configuration is to +minimize the amount of configuration. When adding a configuration option, ask +whether alternatives might be a better fit. + +1. Policy objects: Create first class Kubernetes objects to encompass how the system should behave. These are especially useful for request context dependent configuration. We do this already in places such as RBAC and ResourceQuota but we could do more such as rate limiting. We should never hardcode groups or usermaps in configuration. +1. API features: Use (or implement) functionality of the API (e.g. think through and implement initializers instead of --register-with-label). Allowing for extension in the right places is a better way to give users control. +1. Feature discovery: Write components that introspect the existing API to decide whether to enable a feature or not. E.g. controller-manager should start an app controller if the app API is available, kubelet should enable zram if zram is set in the node spec. +1. Downwards API: Use the APIs that the OS and pod environment expose directly before opting to pass in new configuration options. +1. const's: If you don't know whether tweaking a value will be useful, make the value const. Only give it a configuration option once there becomes a need to tweak the value at runtime. +1. Autotuning: Build systems that incorporate feedback and do the best thing under the given circumstances. This makes the system more robust. (e.g. prefer congestion control, load shedding, backoff rather than explicit limiting). +1. Avoid feature flags: Turn on features when they are tested and ready for production. Don't use feature flags as a fallback for poorly tested code. +1. Configuration profiles: Instead of allowing individual configuration options to be modified, try to encompass a broader desire as a configuration profile. For example: instead of enabling individual alpha features, have an EnableAlpha option that enables all. Instead of allowing individual controller knobs to be modified, have a TestMode option that sets a broad number of parameters to be suitable for tests. + +## Part 2: Component Configuration + +### Versioning Configuration + +We create configuration API groups per component that live in the source tree of +the component. Each component has its own API group for configuration. +Components will use the same API machinery that we use for other API groups. +Configuration API serialization doesn't have the same performance requirements +as other APIs so much of the codegen can be avoided (e.g. ugorji, generated +conversions) and we can instead fallback to the reflection based implementations +where they exist. + +Configuration API groups for component config should be named according to the +scheme `.config.k8s.io`. The `.config.k8s.io` suffix serves to +disambiguate types of config API groups from served APIs. + +### Retrieving Configuration + +The primary mechanism for retrieving static configuration should be +deserialization from files. For the majority of components (with the possible +exception of the kubelet, see +[here](https://github.com/kubernetes/kubernetes/pull/29459)), these files will +be source from the configmap API and managed by the kubelet. Reliability of +this mechanism is predicated on kubelet checkpointing of pod dependencies. + + +### Structuring Configuration + +Group related options into distinct objects and subobjects. Instead of writing: + + +```yaml +kind: KubeProxyConfiguration +apiVersion: kubeproxy.config.k8s.io/v1beta3 +ipTablesSyncPeriod: 2 +ipTablesConntrackHashSize: 2 +ipTablesConntrackTableSize: 2 +``` + +Write: + +```yaml +kind: KubeProxyConfiguration +apiVersion: kubeproxy.config.k8s.io/v1beta3 +ipTables: + syncPeriod: 2 + conntrack: + hashSize: 2 + tableSize: 2 +``` + +We should avoid passing around full configuration options to deeply constructed +modules. For example, instead of calling NewSomethingController in the +controller-manager with the full controller-manager config, group relevant +config into a subobject and only pass the subobject. We should expose the +smallest possible necessary configuration to the SomethingController. + + +### Handling Different Types Of Configuration + +Above in "Type Of Configuration" we introduce a few ways to partition +configuration options. Environment information, request context depending +configuration, feature gates, and static configuration should be avoided if at +all possible using a configuration alternative. We should maintain separate +objects along these partitions and consider retrieving these configurations +from separate source (i.e. files). For example: kubeconfig (which falls into +the bootstrapping category) should not be part of the main config option (nor +should the filepath to the kubeconfig), per-instance config should be stored +separately from shared config. This allows for composition and obviates the +need for layering/templating solutions. + + +### In-Process Representation Of Configuration + +We should separate structs for flags, serializable config, and runtime config. + +1. Structs for flags should have enough information for the process startup to retrieve its full configuration. Examples include: path the kubeconfig, path to configuration file, namespace and name of configmap to use for configuration. +1. Structs for serializable configuration: This struct contains the full set of options in a serializable form (e.g. to represent an ip address instead of `net.IP`, use `string`). This is the struct that is versioned and serialized to disk using API machinery. +1. Structs for runtime: This struct holds data in the most appropriate format for execution. This field can hold non-serializable types (e.g. have a `kubeClient` field instead of a `kubeConfig` field, store ip addresses as `net.IP`). + +The flag struct is transformed into the configuration struct which is +transformed into the runtime struct. + + +### Migrating Away From Flags + +Migrating to component configuration can happen incrementally (per component). +By versioning each component's API group separately, we can allow each API +group to advance to beta and GA independently. APIs should be approved by +component owners and reviewers familiar with the component configuration +conventions. We can incentivize operators to migrate away from flags by making +new configuration options only available through the component configuration +APIs. + +# Caveats + +Proposed are not laws but guidelines and as such we've favored completeness +over consistency. There will thus be need for exceptions. + +1. Components (especially those that are not self hosted such as the kubelet) will require custom rollout strategies of new config. +1. Pod checkpointing by kubelet would allow this strategy to be simpler to make reliable. + + +# Miscellaneous Consideration + +1. **This document takes intentionally a very zealous stance against configuration.** Often configuration alternatives are not possible in Kubernetes as they are in proprietary software because Kubernetes has to run in diverse environments, with diverse users, managed by diverse operators. +1. More frequent releases of kubernetes would make "skipping the config knob" more enticing because fixing a bad guess at a const wouldn't take O(4 months) best case to rollout. Factoring in our support for old versions, it takes closer to a year. +1. Self-hosting resolves much of the distribution issue (except for maybe the Kubelet) but reliability is predicated on to-be-implemented features such as kubelet checkpointing of pod dependencies and sound operational practices such as incremental rollout of new configuration using Deployments/DaemonSets. +1. Validating config is hard. Fatal logs lead to crash loops and error logs are ignored. Both options are suboptimal. +1. Configuration needs to be updatable when components are down. +1. Naming style guide: + 1. No negatives, e.g. prefer --enable-foo over --disable-foo + 1. Use the active voice +1. We should actually enforce deprecation. Can we have a test that fails when a comment exists beyond its deadline to be removed? See [#44248](https://github.com/kubernetes/kubernetes/issues/44248) +1. Use different implementations of the same interface rather than if statements to toggle features. This makes deprecation and deletion easy, improving maintainability. +1. How does the proposed solution meet the requirements? Which desired qualities are missed? +1. Configuration changes should trigger predictable and reproducible actions. From a given system state and a given component configuration, we should be able to simulate the actions that the system will take. diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md new file mode 100644 index 000000000..46ca318df --- /dev/null +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -0,0 +1,216 @@ +# Conformance Testing in Kubernetes + +The Kubernetes Conformance test suite is a subset of e2e tests that SIG +Architecture has approved to define the core set of interoperable features that +all conformant Kubernetes clusters must support. The tests verify that the +expected behavior works as a user might encounter it in the wild. + +The process to add new conformance tests is intended to decouple the development +of useful tests from their promotion to conformance: +- Contributors write and submit e2e tests, to be approved by owning SIGs +- Tests are proven to meet the [conformance test requirements] by review + and by accumulation of data on flakiness and reliability +- A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance) + +NB: This should be viewed as a living document in a few key areas: +- The desired set of conformant behaviors is not adequately expressed by the + current set of e2e tests, as such this document is currently intended to + guide us in the addition of new e2e tests than can fill this gap +- This document currently focuses solely on the requirements for GA, + non-optional features or APIs. The list of requirements will be refined over + time to the point where it as concrete and complete as possible. +- There are currently conformance tests that violate some of the requirements + (e.g., require privileged access), we will be categorizing these tests and + deciding what to do once we have a better understanding of the situation +- Once we resolve the above issues, we plan on identifying the appropriate areas + to relax requirements to allow for the concept of conformance Profiles that + cover optional or additional behaviors + +## Conformance Test Requirements + +Conformance tests currently test only GA, non-optional features or APIs. More +specifically, a test is eligible for promotion to conformance if: + +- it tests only GA, non-optional features or APIs (e.g., no alpha or beta + endpoints, no feature flags required, no deprecated features) +- it works for all providers (e.g., no `SkipIfProviderIs`/`SkipUnlessProviderIs` + calls) +- it is non-privileged (e.g., does not require root on nodes, access to raw + network interfaces, or cluster admin permissions) +- it works without access to the public internet (short of whatever is required + to pre-pull images for conformance tests) +- it works without non-standard filesystem permissions granted to pods +- it does not rely on any binaries that would not be required for the linux + kernel or kubelet to run (e.g., can't rely on git) +- any container images used within the test support all architectures for which + kubernetes releases are built +- it passes against the appropriate versions of kubernetes as spelled out in + the [conformance test version skew policy] +- it is stable and runs consistently (e.g., no flakes) + +Examples of features which are not currently eligible for conformance tests: + +- node/platform-reliant features, eg: multiple disk mounts, GPUs, high density, + etc. +- optional features, eg: policy enforcement +- cloud-provider-specific features, eg: GCE monitoring, S3 Bucketing, etc. +- anything that requires a non-default admission plugin + +Examples of tests which are not eligible for promotion to conformance: +- anything that checks specific Events are generated, as we make no guarantees + about the contents of events, nor their delivery +- anything that checks optional Condition fields, such as Reason or Message, as + these may change over time (however it is reasonable to verify these fields + exist or are non-empty) + +Examples of areas we may want to relax these requirements once we have a +sufficient corpus of tests that define out of the box functionality in all +reasonable production worthy environments: +- tests may need to create or set objects or fields that are alpha or beta that + bypass policies that are not yet GA, but which may reasonably be enabled on a + conformant cluster (e.g., pod security policy, non-GA scheduler annotations) + +## Conformance Test Version Skew Policy + +As each new release of Kubernetes provides new functionality, the subset of +tests necessary to demonstrate conformance grows with each release. Conformance +is thus considered versioned, with the same backwards compatibility guarantees +as laid out in the [kubernetes versioning policy] + +To quote: + +> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and +> should work with v1.2, v1.3, and v1.4 clients. + +Conformance tests for a given version should be run off of the release branch +that corresponds to that version. Thus `v1.2` conformance tests would be run +from the head of the `release-1.2` branch. + +For example, suppose we're in the midst of developing kubernetes v1.3. Clusters +with the following versions must pass conformance tests built from the +following branches: + +| cluster version | master | release-1.3 | release-1.2 | release-1.1 | +| --------------- | ----- | ----------- | ----------- | ----------- | +| v1.3.0-alpha | yes | yes | yes | no | +| v1.2.x | no | no | yes | yes | +| v1.1.x | no | no | no | yes | + +## Running Conformance Tests + +Conformance tests are designed to be run even when there is no cloud provider +configured. Conformance tests must be able to be run against clusters that have +not been created with `hack/e2e.go`, just provide a kubeconfig with the +appropriate endpoint and credentials. + +These commands are intended to be run within a kubernetes directory, either +cloned from source, or extracted from release artifacts such as +`kubernetes.tar.gz`. They assume you have a valid golang installation. + +```sh +# ensure kubetest is installed +go get -u k8s.io/test-infra/kubetest + +# build test binaries, ginkgo, and kubectl first: +make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" + +# setup for conformance tests +export KUBECONFIG=/path/to/kubeconfig +export KUBERNETES_CONFORMANCE_TEST=y + +# Option A: run all conformance tests serially +kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" + +# Option B: run parallel conformance tests first, then serial conformance tests serially +GINKGO_PARALLEL=y kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" +kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" +``` + +## Kubernetes Conformance Document + +For each Kubernetes release, a Conformance Document will be generated that lists +all of the tests that comprise the conformance test suite, along with the formal +specification of each test. For an example, see the [v1.9 conformance doc]. +This document will help people understand what features are being tested without +having to look through the testcase's code directly. + + +## Promoting Tests to Conformance + +To promote a test to the conformance test suite, open a PR as follows: +- is titled "Promote xxx e2e test to Conformance" +- includes information and metadata in the description as follows: + - "/area conformance" on a newline + - "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews + @kubernetes/cncf-conformance-wg" on a new line, where sig-foo is whichever + sig owns this test + - any necessary information in the description to verify that the test meets + [conformance test requirements], such as links to reports or dashboards that + prove lack of flakiness +- contains no other modifications to test source code other than the following: + - modifies the testcase to use the `framework.ConformanceIt()` function rather + than the `framework.It()` function + - adds a comment immediately before the `ConformanceIt()` call that includes + all of the required [conformance test comment metadata] +- add the PR to SIG Architecture's [Conformance Test Review board] + + +### Conformance Test Comment Metadata + +Each conformance test must include the following piece of metadata +within its associated comment: + +- `Release`: indicates the Kubernetes release that the test was added to the + conformance test suite. If the test was modified in subsequent releases + then those releases should be included as well (comma separated) +- `Testname`: a human readable short name of the test +- `Description`: a detailed description of the test. This field must describe + the required behaviour of the Kubernetes components being tested using + [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field + is meant to be a "specification" of the tested Kubernetes features, as + such, it must be detailed enough so that readers can fully understand + the aspects of Kubernetes that are being tested without having to read + the test's code directly. Additionally, this test should provide a clear + distinction between the parts of the test that are there for the purpose + of validating Kubernetes rather than simply infrastructure logic that + is necessary to setup, or clean up, the test. + +### Sample Conformance Test + +The following snippet of code shows a sample conformance test's metadata: + +``` +/* + Release : v1.9 + Testname: Kubelet: log output + Description: By default the stdout and stderr from the process being + executed in a pod MUST be sent to the pod's logs. +*/ +framework.ConformanceIt("it should print the output to logs", func() { + ... +}) +``` + +The corresponding portion of the Kubernetes Conformance Documentfor this test +would then look like this: + +> ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) +> +> Release : v1.9 +> +> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. + +### Reporting Conformance Test Results + +Conformance test results, by provider and releases, can be viewed in the +[testgrid conformance dashboard]. If you wish to contribute test results +for your provider, please see the [testgrid conformance README] + +[kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew +[Conformance Test Review board]: https://github.com/kubernetes-sigs/architecture-tracking/projects/1 +[conformance test requirements]: #conformance-test-requirements +[conformance test metadata]: #conformance-test-metadata +[conformance test version skew policy]: #conformance-test-version-skew-policy +[testgrid conformance dashboard]: https://testgrid.k8s.io/conformance-all +[testgrid conformance README]: https://github.com/kubernetes/test-infra/blob/master/testgrid/conformance/README.md +[v1.9 conformance doc]: https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md diff --git a/contributors/devel/sig-architecture/godep.md b/contributors/devel/sig-architecture/godep.md new file mode 100644 index 000000000..4b10a7d5c --- /dev/null +++ b/contributors/devel/sig-architecture/godep.md @@ -0,0 +1,251 @@ +# Using godep to manage dependencies + +This document is intended to show a way for managing `vendor/` tree dependencies +in Kubernetes. If you do not need to manage vendored dependencies, you probably +do not need to read this. + +## Background + +As a tool, `godep` leaves much to be desired. It builds on `go get`, and adds +the ability to pin dependencies to exact git version. The `go get` tool itself +doesn't have any concept of versions, and tends to blow up if it finds a git +repo synced to anything but `master`, but that is exactly the state that +`godep` leaves repos. This is a recipe for frustration when people try to use +the tools. + +This doc will focus on predictability and reproducibility. + +## Justifications for an update + +Before you update a dependency, take a moment to consider why it should be +updated. Valid reasons include: + 1. We need new functionality that is in a later version. + 2. New or improved APIs in the dependency significantly improve Kubernetes code. + 3. Bugs were fixed that impact Kubernetes. + 4. Security issues were fixed even if they don't impact Kubernetes yet. + 5. Performance, scale, or efficiency was meaningfully improved. + 6. We need dependency A and there is a transitive dependency B. + 7. Kubernetes has an older level of a dependency that is precluding being able +to work with other projects in the ecosystem. + +## Theory of operation + +The `go` toolchain assumes a global workspace that hosts all of your Go code. + +The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`. +This reads the `Godeps.json` file, downloads all of the dependencies from the +internet, and syncs them to the specified revisions. You can then make +changes - sync to different revisions or edit Kubernetes code to use new +dependencies (and satisfy them with `go get`). When ready, you tell `godep` to +"save" everything, which it does by walking the Kubernetes code, finding all +required dependencies, copying them from `$GOPATH` into the `vendor/` directory, +and rewriting `Godeps.json`. + +This does not work well, when combined with a global Go workspace. Instead, we +will set up a private workspace for this process. + +The Kubernetes build process uses this same technique, and offers a tool called +`run-in-gopath.sh` which sets up and switches to a local, private workspace, +including setting up `$GOPATH` and `$PATH`. If you wrap commands with this +tool, they will use the private workspace, which will not conflict with other +projects and is easily cleaned up and recreated. + +To see this in action, you can run an interactive shell in this environment: + +```sh +# Run a shell, but don't run your own shell initializations. +hack/run-in-gopath.sh bash --norc --noprofile +``` + +## Restoring deps + +To extract and download dependencies into `$GOPATH` we provide a script: +`hack/godep-restore.sh`. If you run this tool, it will restore into your own +`$GOPATH`. If you wrap it in `run-in-gopath.sh` it will restore into your +`_output/` directory. + +```sh +hack/run-in-gopath.sh hack/godep-restore.sh +``` + +This script will try to optimize what it needs to download, and if it seems the +dependencies are all present already, it will return very quickly. + +If there's every any doubt about the correctness of your dependencies, you can +simply `make clean` or `rm -rf _output`, and run it again. + +Now you should have a clean copy of all of the Kubernetes dependencies. + +Downloading dependencies might take a while, so if you want to see progress +information use the `-v` flag: + +```sh +hack/run-in-gopath.sh hack/godep-restore.sh -v +``` + +## Making changes + +The most common things people need to do with deps are add and update them. +These are similar but different. + +### Adding a dep + +For the sake of examples, consider that we have discovered a wonderful Go +library at `example.com/go/frob`. The first thing you need to do is get that +code into your workspace: + +```sh +hack/run-in-gopath.sh go get -d example.com/go/frob +``` + +This will fetch, but not compile (omit the `-d` if you want to compile it now), +the library into your private `$GOPATH`. It will pull whatever the default +revision of that library is, typically the `master` branch for git repositories. +If this is not the revision you need, you can change it, for example to +`v1.0.0`: + +```sh +hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0' +``` + +Now that the code is present, you can start to use it in Kubernetes code. +Because it is in your private workspace's `$GOPATH`, it might not be part of +your own `$GOPATH`, so tools like `goimports` might not find it. This is an +unfortunate side-effect of this process. You can either add the whole private +workspace to your own `$GOPATH` or you can `go get` the library into your own +`$GOPATH` until it is properly vendored into Kubernetes. + +Another possible complication is a dep that uses `gopdep` itself. In that case, +you need to restore its dependencies, too: + +```sh +hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore' +``` + +If the transitive deps collide with Kubernetes deps, you may have to manually +resolve things. This is where the ability to run a shell in this environment +comes in handy: + +```sh +hack/run-in-gopath.sh bash --norc --noprofile +``` + +### Updating a dep + +Sometimes we already have a dep, but the version of it is wrong. Because of the +way that `godep` and `go get` interact (badly) it's generally easiest to hit it +with a big hammer: + +```sh +hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob' +hack/run-in-gopath.sh go get -d example.com/go/frob +hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0' +``` + +This will remove the code, re-fetch it, and sync to your desired version. + +### Removing a dep + +This happens almost for free. If you edit Kubernetes code and remove the last +use of a given dependency, you only need to restore and save the deps, and the +`godep` tool will figure out that you don't need that dep any more: + +## Saving deps + +Now that you have made your changes - adding, updating, or removing the use of a +dep - you need to rebuild the dependency database and make changes to the +`vendor/` directory. + +```sh +hack/run-in-gopath.sh hack/godep-save.sh +``` + +This will run through all of the primary targets for the Kubernetes project, +calculate which deps are needed, and rebuild the database. It will also +regenerate other metadata files which the project needs, such as BUILD files and +the LICENSE database. + +Commit the changes before updating deps in staging repos. + +## Saving deps in staging repos + +Kubernetes stores some code in a directory called `staging` which is handled +specially, and is not covered by the above. If you modified any code under +staging, or if you changed a dependency of code under staging (even +transitively), you'll also need to update deps there: + +```sh +./hack/update-staging-godeps.sh +``` + +Then commit the changes generated by the above script. + +## Commit messages + +Terse messages like "Update foo.org/bar to 0.42" are problematic +for maintainability. Please include in your commit message the +detailed reason why the dependencies were modified. + +Too commonly dependency changes have a ripple effect where something +else breaks unexpectedly. The first instinct during issue triage +is to revert a change. If the change was made to fix some other +issue and that issue was not documented, then a revert simply +continues the ripple by fixing one issue and reintroducing another +which then needs refixed. This can needlessly span multiple days +as CI results bubble in and subsequent patches fix and refix and +rerefix issues. This may be avoided if the original modifications +recorded artifacts of the change rationale. + +## Sanity checking + +After all of this is done, `git status` should show you what files have been +modified and added/removed. Make sure to sanity-check them with `git diff`, and +to `git add` and `git rm` them, as needed. It is commonly advised to make one +`git commit` which includes just the dependencies and Godeps files, and +another `git commit` that includes changes to Kubernetes code to use (or stop +using) the new/updated/removed dependency. These commits can go into a single +pull request. + +Before sending your PR, it's a good idea to sanity check that your +Godeps.json file and the contents of `vendor/ `are ok: + +```sh +hack/run-in-gopath.sh hack/verify-godeps.sh +``` + +All this script will do is a restore, followed by a save, and then look for +changes. If you followed the above instructions, it should be clean. If it is +not, you get to figure out why. + +## Manual updates + +It is sometimes expedient to manually fix the `Godeps.json` file to +minimize the changes. However, without great care this can lead to failures +with the verifier scripts. The kubernetes codebase does "interesting things" +with symlinks between `vendor/` and `staging/` to allow multiple Go import +paths to coexist in the same git repo. + +The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull +request. + +## Reviewing and approving dependency changes + +Particular attention to detail should be exercised when reviewing and approving +PRs that add/remove/update dependencies. Importing a new dependency should bring +a certain degree of value as there is a maintenance overhead for maintaining +dependencies into the future. + +When importing a new dependency, be sure to keep an eye out for the following: +- Is the dependency maintained? +- Does the dependency bring value to the project? Could this be done without + adding a new dependency? +- Is the target dependency the original source, or a fork? +- Is there already a dependency in the project that does something similar? +- Does the dependency have a license that is compatible with the Kubernetes + project? + +All new dependency licenses should be reviewed by either Tim Hockin (@thockin) +or the Steering Committee (@kubernetes/steering-committee) to ensure that they +are compatible with the Kubernetes project license. It is also important to note +and flag if a license has changed when updating a dependency, so that these can +also be reviewed. diff --git a/contributors/devel/sig-architecture/staging.md b/contributors/devel/sig-architecture/staging.md new file mode 100644 index 000000000..79ae762f9 --- /dev/null +++ b/contributors/devel/sig-architecture/staging.md @@ -0,0 +1,34 @@ +# Staging Directory and Publishing + +The [staging/ directory](https://git.k8s.io/kubernetes/staging) of Kubernetes contains a number of pseudo repositories ("staging repos"). They are symlinked into Kubernetes' [vendor/ directory](https://git.k8s.io/kubernetes/vendor/k8s.io) for Golang to pick them up. + +We publish the staging repos using the [publishing bot](https://git.k8s.io/publishing-bot). It uses `git filter-branch` essentially to [cut the staging directories into separate git trees](https://de.slideshare.net/sttts/cutting-the-kubernetes-monorepo-in-pieces-never-learnt-more-about-git) and pushing the new commits to the corresponding real repositories in the [kubernetes organization on Github](https://github.com/kubernetes). + +The list of staging repositories and their published branches are listed in [publisher.go inside of the bot](https://git.k8s.io/publishing-bot/cmd/publishing-bot/publisher.go). Though it is planned to move this out into the k8s.io/kubernetes repository. + +At the time of this writing, this includes the branches + +- master, +- release-1.8 / release-5.0, +- and release-1.9 / release-6.0 + +of the following staging repos in the k8s.io org: + +- api +- apiextensions-apiserver +- apimachinery +- apiserver +- client-go +- code-generator +- kube-aggregator +- metrics +- sample-apiserver +- sample-controller + +Kubernetes tags (e.g., v1.9.1-beta1) are also applied automatically to the published repositories, prefixed with kubernetes- (e.g., kubernetes-1.9.1-beta1). The client-go semver tags (on client-go only!) including release-notes are still done manually. + +The semver tags are still the (well tested) official releases. The kubernetes-1.x.y tags have limited test coverage (we have some automatic tests in place in the bot), but can be used by early adopters of client-go and the other libraries. Moreover, they help to vendor the correct version of k8s.io/api and k8s.io/apimachinery. + +If further repos under staging are need, adding them to the bot is easy. Contact one of the [owners of the bot](https://git.k8s.io/publishing-bot/OWNERS). + +Currently, the bot is hosted on the CI cluster of Redhat's OpenShift (ready to be moved out to a public CNCF cluster if we have that in the future). diff --git a/contributors/devel/staging.md b/contributors/devel/staging.md index 79ae762f9..a1cd33b77 100644 --- a/contributors/devel/staging.md +++ b/contributors/devel/staging.md @@ -1,34 +1,3 @@ -# Staging Directory and Publishing +This file has moved to https://git.k8s.io/community/contributors/devel/sig-architecture/staging.md. -The [staging/ directory](https://git.k8s.io/kubernetes/staging) of Kubernetes contains a number of pseudo repositories ("staging repos"). They are symlinked into Kubernetes' [vendor/ directory](https://git.k8s.io/kubernetes/vendor/k8s.io) for Golang to pick them up. - -We publish the staging repos using the [publishing bot](https://git.k8s.io/publishing-bot). It uses `git filter-branch` essentially to [cut the staging directories into separate git trees](https://de.slideshare.net/sttts/cutting-the-kubernetes-monorepo-in-pieces-never-learnt-more-about-git) and pushing the new commits to the corresponding real repositories in the [kubernetes organization on Github](https://github.com/kubernetes). - -The list of staging repositories and their published branches are listed in [publisher.go inside of the bot](https://git.k8s.io/publishing-bot/cmd/publishing-bot/publisher.go). Though it is planned to move this out into the k8s.io/kubernetes repository. - -At the time of this writing, this includes the branches - -- master, -- release-1.8 / release-5.0, -- and release-1.9 / release-6.0 - -of the following staging repos in the k8s.io org: - -- api -- apiextensions-apiserver -- apimachinery -- apiserver -- client-go -- code-generator -- kube-aggregator -- metrics -- sample-apiserver -- sample-controller - -Kubernetes tags (e.g., v1.9.1-beta1) are also applied automatically to the published repositories, prefixed with kubernetes- (e.g., kubernetes-1.9.1-beta1). The client-go semver tags (on client-go only!) including release-notes are still done manually. - -The semver tags are still the (well tested) official releases. The kubernetes-1.x.y tags have limited test coverage (we have some automatic tests in place in the bot), but can be used by early adopters of client-go and the other libraries. Moreover, they help to vendor the correct version of k8s.io/api and k8s.io/apimachinery. - -If further repos under staging are need, adding them to the bot is easy. Contact one of the [owners of the bot](https://git.k8s.io/publishing-bot/OWNERS). - -Currently, the bot is hosted on the CI cluster of Redhat's OpenShift (ready to be moved out to a public CNCF cluster if we have that in the future). +This file is a placeholder to preserve links. Please remove by April 24, 2019 or the release of kubernetes 1.13, whichever comes first. \ No newline at end of file diff --git a/contributors/guide/README.md b/contributors/guide/README.md index 5af355668..e91eb0fce 100644 --- a/contributors/guide/README.md +++ b/contributors/guide/README.md @@ -225,7 +225,7 @@ The location of the test code varies with type, as do the specifics of the envir * Unit: These confirm that a particular function behaves as intended. Golang includes a native ability for unit testing via the [testing](https://golang.org/pkg/testing/) package. Unit test source code can be found adjacent to the corresponding source code within a given package. For example: functions defined in [kubernetes/cmd/kubeadm/app/util/version.go](https://git.k8s.io/kubernetes/cmd/kubeadm/app/util/version.go) will have unit tests in [kubernetes/cmd/kubeadm/app/util/version_test.go](https://git.k8s.io/kubernetes/cmd/kubeadm/app/util/version_test.go). These are easily run locally by any developer on any OS. * Integration: These tests cover interactions of package components or interactions between kubernetes components and some other non-kubernetes system resource (eg: etcd). An example would be testing whether a piece of code can correctly store data to or retrieve data from etcd. Integration tests are stored in [kubernetes/test/integration/](https://git.k8s.io/kubernetes/test/integration). Running these can require the developer set up additional functionality on their development system. * End-to-end ("e2e"): These are broad tests of overall system behavior and coherence. These are more complicated as they require a functional kubernetes cluster built from the sources to be tested. A separate [document detailing e2e testing](/contributors/devel/sig-testing/e2e-tests.md) and test cases themselves can be found in [kubernetes/test/e2e/](https://git.k8s.io/kubernetes/test/e2e). -* Conformance: These are a set of testcases, currently a subset of the integration/e2e tests, that the Architecture SIG has approved to define the core set of interoperable features that all Kubernetes deployments must support. For more information on Conformance tests please see the [Conformance Testing](/contributors/devel/conformance-tests.md) Document. +* Conformance: These are a set of testcases, currently a subset of the integration/e2e tests, that the Architecture SIG has approved to define the core set of interoperable features that all Kubernetes deployments must support. For more information on Conformance tests please see the [Conformance Testing](/contributors/devel/sig-architecture/conformance-tests.md) Document. Continuous integration will run these tests either as pre-submits on PRs, post-submits against master/release branches, or both. The results appear on [testgrid](https://testgrid.k8s.io). diff --git a/contributors/guide/coding-conventions.md b/contributors/guide/coding-conventions.md index c424b9d8a..880c1833c 100644 --- a/contributors/guide/coding-conventions.md +++ b/contributors/guide/coding-conventions.md @@ -55,7 +55,7 @@ the name of the directory in which the .go file exists. sync.Mutex`). When multiple locks are present, give each lock a distinct name following Go conventions - `stateLock`, `mapLock` etc. - - [API changes](/contributors/devel/api_changes.md) + - [API changes](/contributors/devel/sig-architecture/api_changes.md) - [API conventions](/contributors/devel/api-conventions.md) @@ -119,7 +119,7 @@ respectively. Actual application examples belong in /examples. - Go code for normal third-party dependencies is managed using [Godep](https://github.com/tools/godep) and is described in the kubernetes -[godep guide](/contributors/devel/godep.md) +[godep guide](/contributors/devel/sig-architecture/godep.md) - Other third-party code belongs in `/third_party` - forked third party Go code goes in `/third_party/forked` diff --git a/sig-architecture/README.md b/sig-architecture/README.md index accb0dba7..f7d629c0b 100644 --- a/sig-architecture/README.md +++ b/sig-architecture/README.md @@ -100,7 +100,7 @@ Establishing and documenting conventions for system and user-facing APIs, define * [Kubernetes Design and Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md) * [Design principles](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/principles.md) -* [API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md) +* [API conventions](/contributors/devel/sig-architecture/api-conventions.md) * [API Review process](https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md) * [Deprecation policy](https://kubernetes.io/docs/reference/deprecation-policy/) @@ -111,7 +111,7 @@ Please see the [Design documentation](https://github.com/kubernetes-sigs/archite Reviewing, approving, and driving changes to the conformance test suite; reviewing, guiding, and creating new conformance profiles * [Conformance Tests](https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/conformance.txt) -* [Test Guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/conformance-tests.md) +* [Test Guidelines](/contributors/devel/sig-architecture/conformance-tests.md) Please see the [Conformance Test Review](https://github.com/kubernetes-sigs/architecture-tracking/projects/1) tracking board to follow the work for this sub-project. Please reach out to folks in the [OWNERS](https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/OWNERS) file if you are interested in joining this effort. There is a lot of overlap with the [Kubernetes Software Conformance Working Group](https://github.com/cncf/k8s-conformance/blob/master/README-WG.md) with this sub project as well. The github group [cncf-conformance-wg](https://github.com/orgs/kubernetes/teams/cncf-conformance-wg) enumerates the folks on this working group. Look for the `area/conformance` label in the kubernetes repositories to mark [issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aissue+is%3Aopen+label%3Aarea%2Fconformance) and [PRs](https://github.com/kubernetes/kubernetes/pulls?q=is%3Apr+is%3Aopen+label%3Aarea%2Fconformance) diff --git a/sig-architecture/api-review-process.md b/sig-architecture/api-review-process.md index f3f23a621..5a675caa7 100644 --- a/sig-architecture/api-review-process.md +++ b/sig-architecture/api-review-process.md @@ -4,7 +4,7 @@ # Process Overview and Motivations -Due to the importance of preserving usability and consistency in Kubernetes APIs, all changes and additions require expert oversight. The API review process is intended to maintain logical and functional integrity of the API over time, the consistency of user experience and the ability of previously written tools to function with new APIs. Wherever possible, the API review process should help change submitters follow [established conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md), and not simply reject without cause. +Due to the importance of preserving usability and consistency in Kubernetes APIs, all changes and additions require expert oversight. The API review process is intended to maintain logical and functional integrity of the API over time, the consistency of user experience and the ability of previously written tools to function with new APIs. Wherever possible, the API review process should help change submitters follow [established conventions](/contributors/devel/sig-architecture/api-conventions.md), and not simply reject without cause. Because expert reviewer bandwidth is extremely limited, the process provides a curated backlog with highest priority issues at the top. While this does mean some changes may be delayed in favor of other higher priority ones, this will help maintain critical project velocity, transparency, and equilibrium. Ideally, those whose API review priority is shifted in a release-impacting way will be proactively notified by the reviewers. @@ -227,9 +227,9 @@ Aspiring reviewers should reach out the moderator on slack. The moderator will * [Updating-docs-for-feature-changes.md](https://git.k8s.io/sig-release/release-team/role-handbooks/documentation-guides/updating-docs-for-feature-changes.md#when-making-api-changes) -* [https://github.com/kubernetes/community/blob/be9eeca6ee3becfa5b4c96bedf62b5b3ff5b1f8d/contributors/devel/api_changes.md](https://github.com/kubernetes/community/blob/be9eeca6ee3becfa5b4c96bedf62b5b3ff5b1f8d/contributors/devel/api_changes.md) +* [Api_changes.md](/contributors/devel/sig-architecture/api_changes.md) -* [https://github.com/kubernetes/community/blob/be9eeca6ee3becfa5b4c96bedf62b5b3ff5b1f8d/contributors/devel/api-conventions.md](https://github.com/kubernetes/community/blob/be9eeca6ee3becfa5b4c96bedf62b5b3ff5b1f8d/contributors/devel/api-conventions.md) +* [Api-conventions.md](/contributors/devel/sig-architecture/api-conventions.md) * [Pull-requests.md](https://github.com/kubernetes/community/blob/a74d906f0121c78114d79a3ac105aa2d36e24b57/contributors/devel/pull-requests.md#2-smaller-is-better-small-commits-small-prs) - should be updated to specifically call out API changes as important