k8s-ci-robot
GitHub
commit
9dd9ae0058
|
|
@ -0,0 +1,323 @@
|
||||||
|
---
|
||||||
|
kep-number: 14 FIXME(13)
|
||||||
|
title: Runtime Class
|
||||||
|
authors:
|
||||||
|
- "@tallclair"
|
||||||
|
owning-sig: sig-node
|
||||||
|
participating-sigs:
|
||||||
|
- sig-architecture
|
||||||
|
reviewers:
|
||||||
|
- TBD
|
||||||
|
approvers:
|
||||||
|
- TBD
|
||||||
|
editor: TBD
|
||||||
|
creation-date: 2018-06-19
|
||||||
|
status: provisional
|
||||||
|
---
|
||||||
|
|
||||||
|
# Runtime Class
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
* [Summary](#summary)
|
||||||
|
* [Motivation](#motivation)
|
||||||
|
* [Goals](#goals)
|
||||||
|
* [Non\-Goals](#non-goals)
|
||||||
|
* [User Stories](#user-stories)
|
||||||
|
* [Proposal](#proposal)
|
||||||
|
* [API](#api)
|
||||||
|
* [Runtime Handler](#runtime-handler)
|
||||||
|
* [Versioning, Updates, and Rollouts](#versioning-updates-and-rollouts)
|
||||||
|
* [Implementation Details](#implementation-details)
|
||||||
|
* [Risks and Mitigations](#risks-and-mitigations)
|
||||||
|
* [Graduation Criteria](#graduation-criteria)
|
||||||
|
* [Implementation History](#implementation-history)
|
||||||
|
* [Appendix](#appendix)
|
||||||
|
* [Examples of runtime variation](#examples-of-runtime-variation)
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
`RuntimeClass` is a new cluster-scoped resource that surfaces container runtime properties to the
|
||||||
|
control plane. RuntimeClasses are assigned to pods through a `runtimeClass` field on the
|
||||||
|
`PodSpec`. This provides a new mechanism for supporting multiple runtimes in a cluster and/or node.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
There is growing interest in using different runtimes within a cluster. [Sandboxes][] are the
|
||||||
|
primary motivator for this right now, with both Kata containers and gVisor looking to integrate with
|
||||||
|
Kubernetes. Other runtime models such as Windows containers or even remote runtimes will also
|
||||||
|
require support in the future. RuntimeClass provides a way to select between different runtimes
|
||||||
|
configured in the cluster and surface their properties (both to the cluster & the user).
|
||||||
|
|
||||||
|
In addition to selecting the runtime to use, supporting multiple runtimes raises other problems to
|
||||||
|
the control plane level, including: accounting for runtime overhead, scheduling to nodes that
|
||||||
|
support the runtime, and surfacing which optional features are supported by different
|
||||||
|
runtimes. Although these problems are not tackled by this initial proposal, RuntimeClass provides a
|
||||||
|
cluster-scoped resource tied to the runtime that can help solve these problems in a future update.
|
||||||
|
|
||||||
|
[Sandboxes]: https://docs.google.com/document/d/1QQ5u1RBDLXWvC8K3pscTtTRThsOeBSts_imYEoRyw8A/edit
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
|
||||||
|
- Provide a mechanism for surfacing container runtime properties to the control plane
|
||||||
|
- Support multiple runtimes per-cluster, and provide a mechanism for users to select the desired
|
||||||
|
runtime
|
||||||
|
|
||||||
|
### Non-Goals
|
||||||
|
|
||||||
|
- RuntimeClass is NOT RuntimeComponentConfig.
|
||||||
|
- RuntimeClass is NOT a general policy mechanism.
|
||||||
|
- RuntimeClass is NOT "NodeClass". Although different nodes may run different runtimes, in general
|
||||||
|
RuntimeClass should not be a cross product of runtime properties and node properties.
|
||||||
|
|
||||||
|
The following goals are out-of-scope for the initial implementation, but may be explored in a future
|
||||||
|
iteration:
|
||||||
|
|
||||||
|
- Surfacing support for optional features by runtimes, and surfacing errors caused by
|
||||||
|
incompatible features & runtimes earlier.
|
||||||
|
- Automatic runtime or feature discovery - initially RuntimeClasses are manually defined (by the
|
||||||
|
cluster admin or provider), and are asserted to be an accurate representation of the runtime.
|
||||||
|
- Scheduling in heterogeneous clusters - it is possible to operate a heterogeneous cluster
|
||||||
|
(different runtime configurations on different nodes) through scheduling primitives like
|
||||||
|
`NodeAffinity` and `Taints+Tolerations`, but the user is responsible for setting these up and
|
||||||
|
automatic runtime-aware scheduling is out-of-scope.
|
||||||
|
- Define standardized or conformant runtime classes - although I would like to declare some
|
||||||
|
predefined RuntimeClasses with specific properties, doing so is out-of-scope for this initial KEP.
|
||||||
|
- [Pod Overhead][] - Although RuntimeClass is likely to be the configuration mechanism of choice,
|
||||||
|
the details of how pod resource overhead will be implemented is out of scope for this KEP.
|
||||||
|
- Provide a mechanism to dynamically register or provision additional runtimes.
|
||||||
|
- Requiring specific RuntimeClasses according to policy. This should be addressed by other
|
||||||
|
cluster-level policy mechanisms, such as PodSecurityPolicy.
|
||||||
|
- "Fitting" a RuntimeClass to pod requirements - In other words, specifying runtime properties and
|
||||||
|
letting the system match an appropriate RuntimeClass, rather than explicitly assigning a
|
||||||
|
RuntimeClass by name. This approach can increase portability, but can be added seamlessly in a
|
||||||
|
future iteration.
|
||||||
|
|
||||||
|
[Pod Overhead]: https://docs.google.com/document/d/1EJKT4gyl58-kzt2bnwkv08MIUZ6lkDpXcxkHqCvvAp4/edit
|
||||||
|
|
||||||
|
### User Stories
|
||||||
|
|
||||||
|
- As a cluster operator, I want to provide multiple runtime options to support a wide variety of
|
||||||
|
workloads. Examples include native linux containers, "sandboxed" containers, and windows
|
||||||
|
containers.
|
||||||
|
- As a cluster operator, I want to provide stable rolling upgrades of runtimes. For
|
||||||
|
example, rolling out an update with backwards incompatible changes or previously unsupported
|
||||||
|
features.
|
||||||
|
- As an application developer, I want to select the runtime that best fits my workload.
|
||||||
|
- As an application developer, I don't want to study the nitty-gritty details of different runtime
|
||||||
|
implementations, but rather choose from pre-configured classes.
|
||||||
|
- As an application developer, I want my application to be portable across clusters that use similar
|
||||||
|
but different variants of a "class" of runtimes.
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
The initial design includes:
|
||||||
|
|
||||||
|
- `RuntimeClass` API resource definition
|
||||||
|
- `RuntimeClass` pod field for specifying the RuntimeClass the pod should be run with
|
||||||
|
- Kubelet implementation for fetching & interpreting the RuntimeClass
|
||||||
|
- CRI API & implementation for passing along the [RuntimeHandler](#runtime-handler).
|
||||||
|
|
||||||
|
### API
|
||||||
|
|
||||||
|
`RuntimeClass` is a new cluster-scoped resource in the `node.k8s.io` API group.
|
||||||
|
|
||||||
|
> _The `node.k8s.io` API group would eventually hold the Node resource when `core` is retired.
|
||||||
|
> Alternatives considered: `runtime.k8s.io`, `cluster.k8s.io`_
|
||||||
|
|
||||||
|
_(This is a simplified declaration, syntactic details will be covered in the API PR review)_
|
||||||
|
|
||||||
|
```go
|
||||||
|
type RuntimeClass struct {
|
||||||
|
metav1.TypeMeta
|
||||||
|
// ObjectMeta minimally includes the RuntimeClass name, which is used to reference the class.
|
||||||
|
// Namespace should be left blank.
|
||||||
|
metav1.ObjectMeta
|
||||||
|
|
||||||
|
Spec RuntimeClassSpec
|
||||||
|
}
|
||||||
|
|
||||||
|
type RuntimeClassSpec struct {
|
||||||
|
// RuntimeHandler specifies the underlying runtime the CRI calls to handle pod and/or container
|
||||||
|
// creation. The possible values are specific to a given configuration & CRI implementation.
|
||||||
|
// The empty string is equivalent to the default behavior.
|
||||||
|
// +optional
|
||||||
|
RuntimeHandler string
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The runtime is selected by the pod by specifying the RuntimeClass in the PodSpec. Once the pod is
|
||||||
|
scheduled, the RuntimeClass cannot be changed.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type PodSpec struct {
|
||||||
|
...
|
||||||
|
// RuntimeClassName refers to a RuntimeClass object with the same name,
|
||||||
|
// which should be used to run this pod.
|
||||||
|
// +optional
|
||||||
|
RuntimeClassName string
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `legacy` RuntimeClass name is reserved. The legacy RuntimeClass is defined to be fully backwards
|
||||||
|
compatible with current Kubernetes. This means that the legacy runtime does not specify any
|
||||||
|
RuntimeHandler or perform any feature validation (all features are "supported").
|
||||||
|
|
||||||
|
```go
|
||||||
|
const (
|
||||||
|
// RuntimeClassNameLegacy is a reserved RuntimeClass name. The legacy
|
||||||
|
// RuntimeClass does not specify a runtime handler or perform any
|
||||||
|
// feature validation.
|
||||||
|
RuntimeClassNameLegacy = "legacy"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
An unspecified RuntimeClassName `""` is equivalent to the `legacy` RuntimeClass, though the field is
|
||||||
|
not defaulted to `legacy` (to leave room for configurable defaults in a future update).
|
||||||
|
|
||||||
|
#### Runtime Handler
|
||||||
|
|
||||||
|
The `RuntimeHandler` is passed to the CRI as part of the `RunPodSandboxRequest`:
|
||||||
|
|
||||||
|
```proto
|
||||||
|
message RunPodSandboxRequest {
|
||||||
|
// Configuration for creating a PodSandbox.
|
||||||
|
PodSandboxConfig config = 1;
|
||||||
|
// Named runtime configuration to use for this PodSandbox.
|
||||||
|
string RuntimeHandler = 2;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The RuntimeHandler is provided as a mechanism for CRI implementations to select between different
|
||||||
|
predetermined configurations. The initial use case is replacing the experimental pod annotations
|
||||||
|
currently used for selecting a sandboxed runtime by various CRI implementations:
|
||||||
|
|
||||||
|
| CRI Runtime | Pod Annotation |
|
||||||
|
| ------------|-------------------------------------------------------------|
|
||||||
|
| CRIO | io.kubernetes.cri-o.TrustedSandbox: "false" |
|
||||||
|
| containerd | io.kubernetes.cri.untrusted-workload: "true" |
|
||||||
|
| frakti | runtime.frakti.alpha.kubernetes.io/OSContainer: "true"<br>runtime.frakti.alpha.kubernetes.io/Unikernel: "true" |
|
||||||
|
| windows | experimental.windows.kubernetes.io/isolation-type: "hyperv" |
|
||||||
|
|
||||||
|
These implementations could stick with scheme ("trusted" and "untrusted"), but the preferred
|
||||||
|
approach is a non-binary one wherein arbitrary handlers can be configured with a name that can be
|
||||||
|
matched against the specified RuntimeHandler. For example, containerd might have a configuration
|
||||||
|
corresponding to a "kata-runtime" handler:
|
||||||
|
|
||||||
|
```
|
||||||
|
[plugins.cri.containerd.kata-runtime]
|
||||||
|
runtime_type = "io.containerd.runtime.v1.linux"
|
||||||
|
runtime_engine = "/opt/kata/bin/kata-runtime"
|
||||||
|
runtime_root = ""
|
||||||
|
```
|
||||||
|
|
||||||
|
This non-binary approach is more flexible: it can still map to a binary RuntimeClass selection
|
||||||
|
(e.g. `sandboxed` or `untrusted` RuntimeClasses), but can also support multiple parallel sandbox
|
||||||
|
types (e.g. `kata-containers` or `gvisor` RuntimeClasses).
|
||||||
|
|
||||||
|
### Versioning, Updates, and Rollouts
|
||||||
|
|
||||||
|
Getting upgrades and rollouts right is a very nuanced and complicated problem. For the initial alpha
|
||||||
|
implementation, we will kick the can down the road by making the `RuntimeClassSpec` **immutable**,
|
||||||
|
thereby requiring changes to be pushed as a newly named RuntimeClass instance. This means that pods
|
||||||
|
must be updated to reference the new RuntimeClass, and comes with the advantage of native support
|
||||||
|
for rolling updates through the same mechanisms as any other application update. The
|
||||||
|
`RuntimeClassName` pod field is also immutable post scheduling.
|
||||||
|
|
||||||
|
This conservative approach is preferred since it's much easier to relax constraints in a backwards
|
||||||
|
compatible way than tighten them. We should revisit this decision prior to graduating RuntimeClass
|
||||||
|
to beta.
|
||||||
|
|
||||||
|
### Implementation Details
|
||||||
|
|
||||||
|
The Kubelet uses an Informer to keep a local cache of all RuntimeClass objects. When a new pod is
|
||||||
|
added, the Kubelet resolves the Pod's RuntimeClass against the local RuntimeClass cache. Once
|
||||||
|
resolved, the RuntimeHandler field is passed to the CRI as part of the
|
||||||
|
[`RunPodSandboxRequest`][]. At that point, the interpretation of the RuntimeHandler is left to the
|
||||||
|
CRI implementation, but it should be cached if needed for subsequent calls.
|
||||||
|
|
||||||
|
If the RuntimeClass cannot be resolved (e.g. doesn't exist) at Pod creation, then the request will
|
||||||
|
be rejected in admission (controller to be detailed in a following update). If the RuntimeClass
|
||||||
|
cannot be resolved by the Kubelet when `RunPodSandbox` should be called, then the Kubelet will fail
|
||||||
|
the Pod. The admission check on a replica recreation will prevent the scheduler from thrashing. If
|
||||||
|
the `RuntimeHandler` is not recognized by the CRI implementation, then `RunPodSandbox` will return
|
||||||
|
an error.
|
||||||
|
|
||||||
|
[RunPodSandboxRequest]: https://github.com/kubernetes/kubernetes/blob/b05a61e299777c2030fbcf27a396aff21b35f01b/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto#L344
|
||||||
|
|
||||||
|
### Risks and Mitigations
|
||||||
|
|
||||||
|
**Scope creep.** RuntimeClass has a fairly broad charter, but it should not become a default
|
||||||
|
dumping ground for every new feature exposed by the node. For each feature, careful consideration
|
||||||
|
should be made about whether it belongs on the Pod, Node, RuntimeClass, or some other resource. The
|
||||||
|
[non-goals](#non-goals) should be kept in mind when considering RuntimeClass features.
|
||||||
|
|
||||||
|
**Becoming a general policy mechanism.** RuntimeClass should not be used a replacement for
|
||||||
|
PodSecurityPolicy. The use cases for defining multiple RuntimeClasses for the same underlying
|
||||||
|
runtime implementation should be extremely limited (generally only around updates & rollouts). To
|
||||||
|
enforce this, no authorization or restrictions are placed directly on RuntimeClass use; in order to
|
||||||
|
restrict a user to a specific RuntimeClass, you must use another policy mechanism such as
|
||||||
|
PodSecurityPolicy.
|
||||||
|
|
||||||
|
**Pushing complexity to the user.** RuntimeClass is a new resource in order to hide the complexity
|
||||||
|
of runtime configuration from most users (aside from the cluster admin or provisioner). However, we
|
||||||
|
are still side-stepping the issue of precisely defining specific types of runtimes like
|
||||||
|
"Sandboxed". However, it is still up for debate whether precisely defining such runtime categories
|
||||||
|
is even possible. RuntimeClass allows us to decouple this specification from the implementation, but
|
||||||
|
it is still something I hope we can address in a future iteration through the concept of pre-defined
|
||||||
|
or "conformant" RuntimeClasses.
|
||||||
|
|
||||||
|
**Non-portability.** We are already in a world of non-portability for many features (see [examples
|
||||||
|
of runtime variation](#examples-of-runtime-variation). Future improvements to RuntimeClass can help
|
||||||
|
address this issue by formally declaring supported features, or matching the runtime that supports a
|
||||||
|
given workload automitaclly. Another issue is that pods need to refer to a RuntimeClass by name,
|
||||||
|
which may not be defined in every cluster. This is something that can be addressed through
|
||||||
|
pre-defined runtime classes (see previous risk), and/or by "fitting" pod requirements to compatible
|
||||||
|
RuntimeClasses.
|
||||||
|
|
||||||
|
## Graduation Criteria
|
||||||
|
|
||||||
|
Alpha:
|
||||||
|
|
||||||
|
- Everything described in the current proposal
|
||||||
|
- [CRI validation test][cri-validation]
|
||||||
|
|
||||||
|
[cri-validation]: https://github.com/kubernetes-incubator/cri-tools/blob/master/docs/validation.md
|
||||||
|
|
||||||
|
Beta:
|
||||||
|
|
||||||
|
- Major runtimes support RuntimeClass
|
||||||
|
- RuntimeClasses are configured in the E2E environment with test coverage of a non-legacy RuntimeClass
|
||||||
|
- The update & upgrade story is revisited, and a longer-term approach is implemented as necessary.
|
||||||
|
- The cluster admin can choose which RuntimeClass is the default in a cluster.
|
||||||
|
- Additional requirements TBD
|
||||||
|
|
||||||
|
## Implementation History
|
||||||
|
|
||||||
|
- 2018-06-11: SIG-Node decision to move forward with proposal
|
||||||
|
- 2018-06-19: Initial KEP published.
|
||||||
|
|
||||||
|
## Appendix
|
||||||
|
|
||||||
|
### Examples of runtime variation
|
||||||
|
|
||||||
|
- Linux Security Module (LSM) choice - Kubernetes supports both AppArmor & SELinux options on pods,
|
||||||
|
but those are mutually exclusive, and support of either is not required by the runtime. The
|
||||||
|
default configuration is also not well defined.
|
||||||
|
- Seccomp-bpf - Kubernetes has alpha support for specifying a seccomp profile, but the default is
|
||||||
|
defined by the runtime, and support is not guaranteed.
|
||||||
|
- Windows containers - isolation features are very OS-specific, and most of the current features are
|
||||||
|
limited to linux. As we build out Windows container support, we'll need to add windows-specific
|
||||||
|
features as well.
|
||||||
|
- Host namespaces (Network,PID,IPC) may not be supported by virtualization-based runtimes
|
||||||
|
(e.g. Kata-containers & gVisor).
|
||||||
|
- Per-pod and Per-container resource overhead varies by runtime.
|
||||||
|
- Device support (e.g. GPUs) varies wildly by runtime & nodes.
|
||||||
|
- Supported volume types varies by node - it remains TBD whether this information belongs in
|
||||||
|
RuntimeClass.
|
||||||
|
- The list of default capabilities is defined in Docker, but not Kubernetes. Future runtimes may
|
||||||
|
have differing defaults, or support a subset of capabilities.
|
||||||
|
- `Privileged` mode is not well defined, and thus may have differing implementations.
|
||||||
|
- Support for resource over-commit and dynamic resource sizing (e.g. Burstable vs Guaranteed
|
||||||
|
workloads)
|
||||||
Loading…
Reference in New Issue