diff --git a/contributors/design-proposals/architecture/architecture.md b/contributors/design-proposals/architecture/architecture.md index e93742356..1fad518e6 100644 --- a/contributors/design-proposals/architecture/architecture.md +++ b/contributors/design-proposals/architecture/architecture.md @@ -217,7 +217,7 @@ agent. Each node runs a container runtime, which is responsible for downloading images and running containers. Kubelet does not link in the base container runtime. Instead, we're defining a -[Container Runtime Interface](/contributors/devel/container-runtime-interface.md) to control the +[Container Runtime Interface](/contributors/devel/sig-node/container-runtime-interface.md) to control the underlying runtime and facilitate pluggability of that layer. This decoupling is needed in order to maintain clear component boundaries, facilitate testing, and facilitate pluggability. Runtimes supported today, either upstream or by forks, include at least docker (for Linux and Windows), diff --git a/contributors/design-proposals/auth/apparmor.md b/contributors/design-proposals/auth/apparmor.md index a88154bb1..f87d374b3 100644 --- a/contributors/design-proposals/auth/apparmor.md +++ b/contributors/design-proposals/auth/apparmor.md @@ -268,7 +268,7 @@ already underway for Docker, called ## Container Runtime Interface Other container runtimes will likely add AppArmor support eventually, so the -[Container Runtime Interface](/contributors/devel/container-runtime-interface.md) (CRI) needs to be made compatible +[Container Runtime Interface](/contributors/devel/sig-node/container-runtime-interface.md) (CRI) needs to be made compatible with this design. The two important pieces are a way to report whether AppArmor is supported by the runtime, and a way to specify the profile to load (likely through the `LinuxContainerConfig`). diff --git a/contributors/design-proposals/instrumentation/core-metrics-pipeline.md b/contributors/design-proposals/instrumentation/core-metrics-pipeline.md index 1c9d9f70c..1ca5dbd99 100644 --- a/contributors/design-proposals/instrumentation/core-metrics-pipeline.md +++ b/contributors/design-proposals/instrumentation/core-metrics-pipeline.md @@ -29,7 +29,7 @@ This document proposes a design for the set of metrics included in an eventual C "Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things. ["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of kubernetes constructs like pods or volumes. ["Summary API":](https://git.k8s.io/kubernetes/pkg/kubelet/apis/stats/v1alpha1/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. -["CRI":](/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). +["CRI":](/contributors/devel/sig-node/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). "Core Metrics": A set of metrics described in the [Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) whose purpose is to provide metrics for first-class resource isolation and utilization features, including [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) and node resource management. "Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc). "First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec. diff --git a/contributors/design-proposals/node/cri-dockershim-checkpoint.md b/contributors/design-proposals/node/cri-dockershim-checkpoint.md index 85db4c893..9f3a10b5a 100644 --- a/contributors/design-proposals/node/cri-dockershim-checkpoint.md +++ b/contributors/design-proposals/node/cri-dockershim-checkpoint.md @@ -4,7 +4,7 @@ [#34672](https://github.com/kubernetes/kubernetes/issues/34672) ## Background -[Container Runtime Interface (CRI)](../devel/container-runtime-interface.md) +[Container Runtime Interface (CRI)](/contributors/devel/sig-node/container-runtime-interface.md) is an ongoing project to allow container runtimes to integrate with kubernetes via a newly-defined API. [Dockershim](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/dockershim) diff --git a/contributors/devel/container-runtime-interface.md b/contributors/devel/container-runtime-interface.md index 1a121c9e9..30ffb4473 100644 --- a/contributors/devel/container-runtime-interface.md +++ b/contributors/devel/container-runtime-interface.md @@ -1,136 +1,3 @@ -# CRI: the Container Runtime Interface - -## What is CRI? - -CRI (_Container Runtime Interface_) consists of a -[protobuf API](https://git.k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto), -specifications/requirements (to-be-added), -and [libraries](https://git.k8s.io/kubernetes/pkg/kubelet/server/streaming) -for container runtimes to integrate with kubelet on a node. CRI is currently in Alpha. - -In the future, we plan to add more developer tools such as the CRI validation -tests. - -## Why develop CRI? - -Prior to the existence of CRI, container runtimes (e.g., `docker`, `rkt`) were -integrated with kubelet through implementing an internal, high-level interface -in kubelet. The entrance barrier for runtimes was high because the integration -required understanding the internals of kubelet and contributing to the main -Kubernetes repository. More importantly, this would not scale because every new -addition incurs a significant maintenance overhead in the main Kubernetes -repository. - -Kubernetes aims to be extensible. CRI is one small, yet important step to enable -pluggable container runtimes and build a healthier ecosystem. - -## How to use CRI? - -For Kubernetes 1.6+: - -1. Start the image and runtime services on your node. You can have a single - service acting as both image and runtime services. -2. Set the kubelet flags - - Pass the unix socket(s) to which your services listen to kubelet: - `--container-runtime-endpoint` and `--image-service-endpoint`. - - Use the "remote" runtime by `--container-runtime=remote`. - -CRI is still young and we are actively incorporating feedback from developers -to improve the API. Although we strive to maintain backward compatibility, -developers should expect occasional API breaking changes. - -*For Kubernetes 1.5, additional flags are required:* - - Set apiserver flag `--feature-gates=StreamingProxyRedirects=true`. - - Set kubelet flag `--experimental-cri=true`. - -## Does Kubelet use CRI today? - -Yes, Kubelet always uses CRI except for using the rktnetes integration. - -The old, pre-CRI Docker integration was removed in 1.7. - -## Specifications, design documents and proposals - -The Kubernetes 1.5 [blog post on CRI](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) -serves as a general introduction. - - -Below is a mixed list of CRI specifications/requirements, design docs and -proposals. We are working on adding more documentation for the API. - - - [Original proposal](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/container-runtime-interface-v1.md) - - [Networking](/contributors/devel/kubelet-cri-networking.md) - - [Container metrics](/contributors/devel/cri-container-stats.md) - - [Exec/attach/port-forward streaming requests](https://docs.google.com/document/d/1OE_QoInPlVCK9rMAx9aybRmgFiVjHpJCHI9LrfdNM_s/edit?usp=sharing) - - [Container stdout/stderr logs](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/kubelet-cri-logging.md) - -## Work-In-Progress CRI runtimes - - - [cri-o](https://github.com/kubernetes-incubator/cri-o) - - [rktlet](https://github.com/kubernetes-incubator/rktlet) - - [frakti](https://github.com/kubernetes/frakti) - - [cri-containerd](https://github.com/kubernetes-incubator/cri-containerd) - -## [Status update](#status-update) -### Kubernetes v1.7 release (Docker-CRI integration GA, container metrics API) - - The Docker CRI integration has been promoted to GA. - - The legacy, non-CRI Docker integration has been completely removed from - Kubelet. The deprecated `--enable-cri` flag has been removed. - - CRI has been extended to support collecting container metrics from the - runtime. - -### Kubernetes v1.6 release (Docker-CRI integration Beta) - **The Docker CRI integration has been promoted to Beta, and been enabled by -default in Kubelet**. - - **Upgrade**: It is recommended to drain your node before upgrading the - Kubelet. If you choose to perform in-place upgrade, the Kubelet will - restart all Kubernetes-managed containers on the node. - - **Resource usage and performance**: There is no performance regression - in our measurement. The memory usage of Kubelet increases slightly - (~0.27MB per pod) due to the additional gRPC serialization for CRI. - - **Disable**: To disable the Docker CRI integration and fall back to the - old implementation, set `--enable-cri=false`. Note that the old - implementation has been *deprecated* and is scheduled to be removed in - the next release. You are encouraged to migrate to CRI as early as - possible. - - **Others**: The Docker container naming/labeling scheme has changed - significantly in 1.6. This is perceived as implementation detail and - should not be relied upon by any external tools or scripts. - -### Kubernetes v1.5 release (CRI v1alpha1) - - - [v1alpha1 version](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/api/v1alpha1/runtime/api.proto) of CRI is released. - -#### [CRI known issues](#cri-1.5-known-issues): - - - [#27097](https://github.com/kubernetes/kubernetes/issues/27097): Container - metrics are not yet defined in CRI. - - [#36401](https://github.com/kubernetes/kubernetes/issues/36401): The new - container log path/format is not yet supported by the logging pipeline - (e.g., fluentd, GCL). - - CRI may not be compatible with other experimental features (e.g., Seccomp). - - Streaming server needs to be hardened. - - [#36666](https://github.com/kubernetes/kubernetes/issues/36666): - Authentication. - - [#36187](https://github.com/kubernetes/kubernetes/issues/36187): Avoid - including user data in the redirect URL. - -#### [Docker CRI integration known issues](#docker-cri-1.5-known-issues) - - - Docker compatibility: Support only Docker v1.11 and v1.12. - - Network: - - [#35457](https://github.com/kubernetes/kubernetes/issues/35457): Does - not support host ports. - - [#37315](https://github.com/kubernetes/kubernetes/issues/37315): Does - not support bandwidth shaping. - - Exec/attach/port-forward (streaming requests): - - [#35747](https://github.com/kubernetes/kubernetes/issues/35747): Does - not support `nsenter` as the exec handler (`--exec-handler=nsenter`). - - Also see [CRI 1.5 known issues](#cri-1.5-known-issues) for limitations - on CRI streaming. - -## Contacts - - - Email: sig-node (kubernetes-sig-node@googlegroups.com) - - Slack: https://kubernetes.slack.com/messages/sig-node +This file has moved to https://git.k8s.io/community/contributors/devel/sig-node/container-runtime-interface.md. +This file is a placeholder to preserve links. Please remove by April 28, 2019 or the release of kubernetes 1.13, whichever comes first. \ No newline at end of file diff --git a/contributors/devel/sig-node/container-runtime-interface.md b/contributors/devel/sig-node/container-runtime-interface.md new file mode 100644 index 000000000..1a121c9e9 --- /dev/null +++ b/contributors/devel/sig-node/container-runtime-interface.md @@ -0,0 +1,136 @@ +# CRI: the Container Runtime Interface + +## What is CRI? + +CRI (_Container Runtime Interface_) consists of a +[protobuf API](https://git.k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto), +specifications/requirements (to-be-added), +and [libraries](https://git.k8s.io/kubernetes/pkg/kubelet/server/streaming) +for container runtimes to integrate with kubelet on a node. CRI is currently in Alpha. + +In the future, we plan to add more developer tools such as the CRI validation +tests. + +## Why develop CRI? + +Prior to the existence of CRI, container runtimes (e.g., `docker`, `rkt`) were +integrated with kubelet through implementing an internal, high-level interface +in kubelet. The entrance barrier for runtimes was high because the integration +required understanding the internals of kubelet and contributing to the main +Kubernetes repository. More importantly, this would not scale because every new +addition incurs a significant maintenance overhead in the main Kubernetes +repository. + +Kubernetes aims to be extensible. CRI is one small, yet important step to enable +pluggable container runtimes and build a healthier ecosystem. + +## How to use CRI? + +For Kubernetes 1.6+: + +1. Start the image and runtime services on your node. You can have a single + service acting as both image and runtime services. +2. Set the kubelet flags + - Pass the unix socket(s) to which your services listen to kubelet: + `--container-runtime-endpoint` and `--image-service-endpoint`. + - Use the "remote" runtime by `--container-runtime=remote`. + +CRI is still young and we are actively incorporating feedback from developers +to improve the API. Although we strive to maintain backward compatibility, +developers should expect occasional API breaking changes. + +*For Kubernetes 1.5, additional flags are required:* + - Set apiserver flag `--feature-gates=StreamingProxyRedirects=true`. + - Set kubelet flag `--experimental-cri=true`. + +## Does Kubelet use CRI today? + +Yes, Kubelet always uses CRI except for using the rktnetes integration. + +The old, pre-CRI Docker integration was removed in 1.7. + +## Specifications, design documents and proposals + +The Kubernetes 1.5 [blog post on CRI](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) +serves as a general introduction. + + +Below is a mixed list of CRI specifications/requirements, design docs and +proposals. We are working on adding more documentation for the API. + + - [Original proposal](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/container-runtime-interface-v1.md) + - [Networking](/contributors/devel/kubelet-cri-networking.md) + - [Container metrics](/contributors/devel/cri-container-stats.md) + - [Exec/attach/port-forward streaming requests](https://docs.google.com/document/d/1OE_QoInPlVCK9rMAx9aybRmgFiVjHpJCHI9LrfdNM_s/edit?usp=sharing) + - [Container stdout/stderr logs](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/kubelet-cri-logging.md) + +## Work-In-Progress CRI runtimes + + - [cri-o](https://github.com/kubernetes-incubator/cri-o) + - [rktlet](https://github.com/kubernetes-incubator/rktlet) + - [frakti](https://github.com/kubernetes/frakti) + - [cri-containerd](https://github.com/kubernetes-incubator/cri-containerd) + +## [Status update](#status-update) +### Kubernetes v1.7 release (Docker-CRI integration GA, container metrics API) + - The Docker CRI integration has been promoted to GA. + - The legacy, non-CRI Docker integration has been completely removed from + Kubelet. The deprecated `--enable-cri` flag has been removed. + - CRI has been extended to support collecting container metrics from the + runtime. + +### Kubernetes v1.6 release (Docker-CRI integration Beta) + **The Docker CRI integration has been promoted to Beta, and been enabled by +default in Kubelet**. + - **Upgrade**: It is recommended to drain your node before upgrading the + Kubelet. If you choose to perform in-place upgrade, the Kubelet will + restart all Kubernetes-managed containers on the node. + - **Resource usage and performance**: There is no performance regression + in our measurement. The memory usage of Kubelet increases slightly + (~0.27MB per pod) due to the additional gRPC serialization for CRI. + - **Disable**: To disable the Docker CRI integration and fall back to the + old implementation, set `--enable-cri=false`. Note that the old + implementation has been *deprecated* and is scheduled to be removed in + the next release. You are encouraged to migrate to CRI as early as + possible. + - **Others**: The Docker container naming/labeling scheme has changed + significantly in 1.6. This is perceived as implementation detail and + should not be relied upon by any external tools or scripts. + +### Kubernetes v1.5 release (CRI v1alpha1) + + - [v1alpha1 version](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/api/v1alpha1/runtime/api.proto) of CRI is released. + +#### [CRI known issues](#cri-1.5-known-issues): + + - [#27097](https://github.com/kubernetes/kubernetes/issues/27097): Container + metrics are not yet defined in CRI. + - [#36401](https://github.com/kubernetes/kubernetes/issues/36401): The new + container log path/format is not yet supported by the logging pipeline + (e.g., fluentd, GCL). + - CRI may not be compatible with other experimental features (e.g., Seccomp). + - Streaming server needs to be hardened. + - [#36666](https://github.com/kubernetes/kubernetes/issues/36666): + Authentication. + - [#36187](https://github.com/kubernetes/kubernetes/issues/36187): Avoid + including user data in the redirect URL. + +#### [Docker CRI integration known issues](#docker-cri-1.5-known-issues) + + - Docker compatibility: Support only Docker v1.11 and v1.12. + - Network: + - [#35457](https://github.com/kubernetes/kubernetes/issues/35457): Does + not support host ports. + - [#37315](https://github.com/kubernetes/kubernetes/issues/37315): Does + not support bandwidth shaping. + - Exec/attach/port-forward (streaming requests): + - [#35747](https://github.com/kubernetes/kubernetes/issues/35747): Does + not support `nsenter` as the exec handler (`--exec-handler=nsenter`). + - Also see [CRI 1.5 known issues](#cri-1.5-known-issues) for limitations + on CRI streaming. + +## Contacts + + - Email: sig-node (kubernetes-sig-node@googlegroups.com) + - Slack: https://kubernetes.slack.com/messages/sig-node +