kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update EphemeralContainers in apiserver not kubelet

This commit is contained in:
Lee Verberne 2018-03-21 15:54:52 +01:00
parent caa072ead7
commit 946532db82
1 changed files with 85 additions and 139 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

224
contributors/design-proposals/node/troubleshoot-running-pods.md
View File

@ -1,6 +1,6 @@
# Troubleshoot Running Pods # Troubleshoot Running Pods
* Status: Pending * Status: Pending Implementation
* Version: Alpha * Version: Alpha
* Implementation Owner: @verb * Implementation Owner: @verb
@ -102,33 +102,22 @@ subsequently be used to reattach and is reported by `kubectl describe`.
### Kubernetes API Changes ### Kubernetes API Changes
There has been much discussion about how this fits best into the Kubernetes API. There has been much discussion about how this fits best into the Kubernetes API.
The consensus is for an imperative "debug this pod" action that's implemented The consensus is for an imperative "debug this pod" action whereby Kubernetes
mostly in the kubelet. In order to avoid new dependencies in the kubelet, this creates a new, temporary container in a pod on command. In order to avoid new
will be implemented in the Core API. Three possible implementations follow, and dependencies in the kubelet, this will be implemented in the Core API. Three
additional implementations that were evaluated and dismissed are at the end of possible implementations follow, and additional implementations that were
this document. evaluated and dismissed are at the end of this document.
All of the proposed solutions implement the user-level concept of a _Debug All of the proposed solutions implement the user-level concept of a _Debug
Container_ using the API-level concept of an _Ephemeral Container_. The API Container_ using the API-level concept of an _Ephemeral Container_. The API
doesn't prescribe how an Ephemeral Container is used. It could conceivably see doesn't prescribe how an Ephemeral Container is used. It could conceivably see
use other than Debug Containers, but we don't currently have other use cases. use other than Debug Containers, but we don't currently have other use cases.
#### Chosen Solution: POST to an Existing Pod #### Chosen Solution: Subresource to Update PodStatus
We're modifying an existing pod, so this fits as a subresource of the target An Ephemeral Container is not part of the pod specification as it's not part of
pod. We will create a new top-level object that contains a `v1.Container` and the intended state of the pod, but we describe it using the same primitives as
`POST` to the subresource to create the Debug Container. Since this is a `POST`, `PodSpec`. An `EphemeralContainer` contains a Spec, a Status and a Target:
we cannot upgrade the connection to streaming if we want to continue supporting
web socket clients.
Rather than using an existing subresource like `/exec` and conditionally stream
based on `PodExecOptions`, we will create a new subresource with consistent
streaming behavior. A new subresource has the added benefit of being able to
entirely hide interface entirely behind a feature flag by conditionally
registering the new subresource.
A `v1.Container` by itself lacks type and object metadata, so we will create a
new type:
``` ```
// EphemeralContainer describes a container to attach to a running pod for troubleshooting. // EphemeralContainer describes a container to attach to a running pod for troubleshooting.
@ -152,26 +141,37 @@ type EphemeralContainer struct {
} }
``` ```
All Ephemeral Containers that have been created in a pod are listed in the pod's
status:
```
type PodStatus struct {
...
// List of user-initiated ephemeral containers that have been run in this pod.
// +optional
EphemeralContainers []EphemeralContainer `json:"commands,omitempty" protobuf:"bytes,11,rep,name=ephemeralContainers"`
}
```
To create a new Ephemeral Container, one appends a new `EphemeralContainer` with
the desired `v1.Container` as `Spec` in `Pod.Status` and updates the `Pod` in
the API. This is accomplished via a new subresource, `/ephemeralcontainers`,
which enforces the append-only semantics and authorization. This is similar to
the `/status` subresource used by the kubelet to modify pod status.
**Note that Ephemeral Containers are not regular containers and should not be **Note that Ephemeral Containers are not regular containers and should not be
used to build services.** They lack guarantees for resources or execution, and used to build services.** They lack guarantees for resources or execution, and
many of the fields of `v1.Container` will not be allowed for Debug Containers. A many of the fields of `v1.Container` will not be allowed for Debug Containers. A
request will be rejected if any field is set other than the following pod update will fail validation if any field is set other than the following
whitelisted fields: `Name`, `Image`, `Command`, `Args`, `WorkingDir`, `Env`, whitelisted fields: `Name`, `Image`, `Command`, `Args`, `WorkingDir`, `Env`,
`EnvFrom`, `ImagePullPolicy`, `SecurityContext`. `TTY` and `Stdin` are always `EnvFrom`, `ImagePullPolicy`, `SecurityContext`. `TTY` and `Stdin` are always
enabled for Debug Containers and will be ignored. enabled for Debug Containers and will be ignored.
The new `/ephemeralcontainers` subresource allows the following: Once the pod object is updated, the kubelet worker watching this pod will launch
the Ephemeral Container and update its status. The client is expected to watch
1. A `POST` of a `EphemeralContainer` to for the creation of the container status and then attach to the console of a
`/api/v1/namespaces/$NS/pods/$POD_NAME/ephemeralcontainers` to create an debug container using the existing attach endpoint,
Ephemeral Container running in pod `$POD_NAME`.
1. Support for stopping an Ephemeral Container **could be supported in the
future** by a `DELETE` of
`/api/v1/namespaces/$NS/pods/$POD_NAME/ephemeralcontainers/$NAME`.
Once created, it is the responsibility of the client to watch for the
`EphemeralContainer` to appear in the `PodStatus` and then attach to the console
of a debug container using the existing attach endpoint,
`/api/v1/namespaces/$NS/pods/$POD_NAME/attach`. Note that any output of the new `/api/v1/namespaces/$NS/pods/$POD_NAME/attach`. Note that any output of the new
container between its creation and subsequent attach will not be replayed and container between its creation and subsequent attach will not be replayed and
can only be viewed using `kubectl log`. can only be viewed using `kubectl log`.
@ -211,7 +211,7 @@ exec`. To kill an Ephemeral Container one would `attach` and exit the process
interactively or create a new Ephemeral Container to send a signal with interactively or create a new Ephemeral Container to send a signal with
`kill(1)` to the original process. `kill(1)` to the original process.
#### Alternative 2: Declarative Configuration #### Alternative 2: Ephemeral Container Controller
Using subresources is an imperative style API where the client instructs the Using subresources is an imperative style API where the client instructs the
kubelet to perform an action, but in general Kubernetes prefers declarative APIs kubelet to perform an action, but in general Kubernetes prefers declarative APIs
@ -244,31 +244,16 @@ enforce, and SIG Node strongly prefers to minimize kubelet complexity.
### Ephemeral Container Status ### Ephemeral Container Status
`EphemeralContainer` is included in a new field in `PodStatus`: We wish for the kubelet to be able to construct `PodStatus` without relying on
prior state, so we will store `EphemeralContainer.Spec` &
``` `EphemeralContainer.TargetContainerName` as runtime metadata. The kubelet
type PodStatus struct { currently persists container metadata as CRI
...
// List of user-initiated ephemeral containers that have been run in this pod.
// +optional
EphemeralContainers []EphemeralContainer `json:"commands,omitempty" protobuf:"bytes,11,rep,name=ephemeralContainers"`
}
```
The kubelet should be able to construct a complete `PodStatus` with no prior
state using information stored in the container runtime.
`EphemeralContainer.Status` introduces no new data, but the kubelet must also
now populate `EphemeralContainer.Spec` &
`EphemeralContainer.TargetContainerName`.
The kubelet already persists container metadata as CRI
[labels](https://github.com/kubernetes/kubernetes/blob/v1.10.0-alpha.0/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto#L606) [labels](https://github.com/kubernetes/kubernetes/blob/v1.10.0-alpha.0/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto#L606)
and and
[annotations](https://github.com/kubernetes/kubernetes/blob/v1.10.0-alpha.0/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto#L613). [annotations](https://github.com/kubernetes/kubernetes/blob/v1.10.0-alpha.0/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto#L613).
The entire v1.Container used in the request will be serialized and stored as a The entire v1.Container used in the request will be serialized and stored as a
runtime annotation. The value of `TargetContainerName` will be stored as a runtime annotation. The value of `TargetContainerName` will be stored as a
runtime label. Persisting this data in the runtime means it will survive kubelet runtime label. Persisting this data in the runtime means it survives kubelet
restarts. restarts.
At least for the Docker runtime, this is [an intended use of docker At least for the Docker runtime, this is [an intended use of docker
@ -279,72 +264,45 @@ size. Because the container spec may be examined in security sensitive contexts
like admission control, we will conservatively limit the size of the spec to 32K like admission control, we will conservatively limit the size of the spec to 32K
and add a 32K minimum label length test to runtime qualification. and add a 32K minimum label length test to runtime qualification.
`EphemeralContainer` is populated by the kubelet in the same way as regular `EphemeralContainer.Status` is populated by the kubelet in the same way as
container statuses. This is sent to the API server and displayed by `kubectl regular container statuses. This is sent to the API server and displayed by
describe pod`. `kubectl describe pod`.
### Creating Debug Containers ### Creating Debug Containers
1. `kubectl` invokes the new API as described in the preceding section. 1. `kubectl` constructs `EphemeralContainer.Spec` and
1. The API server checks for name collisions with existing running containers `EphemeralContainer.TargetContainerName` based on command line arguments. It
(in both `PodSpec` and `PodStatus.EphemeralContainers`), performs `PUT`s the modified pod to the pod's `/ephemeralcontainers`.
validation, admission control and proxies the connection to the kubelet's 1. The apiserver discards changes to fields other than
`/ephemeralContainers/$NS/$POD_NAME` endpoint. `Pod.Status.EphemeralContainers` and validates the pod update.
1. Since a name collision could happen in the interval between container 1. Update validation fails if existing Ephemeral Containers are removed or
creation and PodStatus being published to the API server, the kubelet changed, or if the new Ephemeral Container has a non-empty status.
will perform an additional check for name collision. 1. Pod validation fails if container spec contains fields disallowed for
1. It is permissible to replace an exited container with one of the same Ephemeral Containers, has the same name as a container in the spec, or
name. has the same name as another running Ephemeral Container. (see below)
1. The kubelet request handler opens an error channel and signals the pod's 1. API resource versioning resolves update races.
sync worker with `UpdatePodOptions` that include the `EphemeralContainer` in 1. The kubelet's pod watcher notices the update and triggers a `syncPod()`.
a new field and a callback in the existing During the sync, the kubelet calls `kuberuntime.StartEphemeralContainer()`
`UpdatePodOptions.OnCompleteFunc`. for any Ephemeral Container with an empty status.
1. The pod sync worker runs the existing `syncPod()` with a new 1. `StartEphemeralContainer()` uses the existing `startContainer()` method,
`SyncPodType` of `SyncPodDebug`. which gains support for targeting the namespaces of a container by name.
1. The request handler blocks (with timeout) on receiving an error from
`syncPod()` via the callback. During this time, `syncPod()` starts the
ephemeral container, including fetching an image if necessary, and
publishes a new `PodStatus`.
1. Timeout is configured by the cluster administrator and defaults to 2
minutes.
1. `syncPod()` again checks for container name collision and starts an
ephemeral container via the new `kuberuntime.StartEphemeralContainer()`.
1. `StartEphemeralContainer()` call uses the existing `startContainer()`
method, which gains support for targeting the namespaces of a container
by name.
1. `syncPod()` runs only from a dedicated pod worker, resolving any races
for container creation.
1. After initial creation, future invocations of `syncPod()` will publish 1. After initial creation, future invocations of `syncPod()` will publish
its ContainerStatus but otherwise ignore the Ephemeral Container. It its ContainerStatus but otherwise ignore the Ephemeral Container. It
will exist for the life of the pod sandbox or it exits and is garbage will exist for the life of the pod sandbox or it exits and is garbage
collected. In no event will it be restarted. collected. In no event will it be restarted.
1. `syncPod()` then finishes a regular sync, publishing an updated PodStatus 1. `syncPod()` finishes a regular sync, publishing an updated PodStatus (which
(which includes the new `EphemeralContainer`) by its normal, existing means. includes the new `EphemeralContainer`) by its normal, existing means.
The pod worker sends its exit status to the request worker.
1. The request worker receives a (hopefully `nil`) `error` and returns it to
the client.
1. `OnCompleteFunc` is not guaranteed to be called, so if the request
worker times out it will check the pod's `PodStatus` to see if the Debug
Container was started prior to returning an error. If this happens the
`EphemeralContainer.Spec` must be compared to verify it was the same one
as requested.
1. The client performs an attach to the debug container's console. 1. The client performs an attach to the debug container's console.
The apiserver detects container name collisions with both containers in the pod
spec and other running Debug Containers by checking
`PodStatus.EphemeralContainers`. In a race to create two Debug Containers with
the same name, the API server will pass both requests and the kubelet will
reject one in the synchronized pod worker.
There are no limits on the number of Debug Containers that can be created in a There are no limits on the number of Debug Containers that can be created in a
pod, but exceeding a pod's resource allocation may cause the pod to be evicted. pod, but exceeding a pod's resource allocation may cause the pod to be evicted.
### Restarting and Reattaching Debug Containers ### Restarting and Reattaching Debug Containers
Debug Containers will never be restarted automatically. It is possible to Debug Containers will never be restarted automatically. It is possible to
replace a Debug Container that has exited by re-using a Debug Container name. It "restart" a Debug Container by by re-using the name of a Debug Container that
is an error to attempt to replace a Debug Container that is still running, which has exited. It is an error to re-use the name of a Debug Container that is still
is detected by both the API server and the kubelet. running, which is detected by API server validation.
One can reattach to a Debug Container using `kubectl attach`. When supported by One can reattach to a Debug Container using `kubectl attach`. When supported by
a runtime, multiple clients can attach to a single debug container and share the a runtime, multiple clients can attach to a single debug container and share the
@ -352,7 +310,7 @@ terminal. This is supported by Docker.
### Killing Debug Containers ### Killing Debug Containers
Debug containers will not be killed automatically until the pod (specifically, Debug containers will not be killed automatically unless the pod (specifically,
the pod sandbox) is destroyed. Debug Containers will stop when their command the pod sandbox) is destroyed. Debug Containers will stop when their command
exits, such as exiting a shell. Unlike `kubectl exec`, processes in Debug exits, such as exiting a shell. Unlike `kubectl exec`, processes in Debug
Containers will not receive an EOF if their connection is interrupted. Containers will not receive an EOF if their connection is interrupted.
@ -367,23 +325,16 @@ are necessary in the kubelet:
the pod spec. the pod spec.
1. As an exception to the above, `SyncPod()` will kill Debug Containers when 1. As an exception to the above, `SyncPod()` will kill Debug Containers when
the pod sandbox changes since a lone Debug Container in an abandoned sandbox the pod sandbox changes since a lone Debug Container in an abandoned sandbox
is not useful. Debug Containers are not automatically started in the new is not useful. Debug Containers are not started automatically in the new
sandbox. sandbox.
1. `convertStatusToAPIStatus()` must sort Debug Containers status into 1. `convertStatusToAPIStatus()` must sort Debug Containers status into
`EphemeralContainerStatuses` similar to as it does for `EphemeralContainer.Status` similar to as it does for
`InitContainerStatuses` `InitContainerStatuses`
1. The kubelet must preserve `ContainerStatus` on debug containers for
reporting.
1. Debug Containers must be excluded from calculation of pod phase and 1. Debug Containers must be excluded from calculation of pod phase and
condition condition
It's worth noting some things that do not change: `KillPod()` already operates on all running containers returned by the runtime
and requires no changes
1. `KillPod()` already operates on all running containers returned by the
runtime.
1. Containers created prior to this feature being enabled will have a
`containerType` of `""`. Since this does not match `"EPHEMERAL"` the special
handling of Debug Containers is backwards compatible.
### Security Considerations ### Security Considerations
@ -392,10 +343,8 @@ Debug Containers have no additional privileges above what is available to any
spec except that it is created on demand. spec except that it is created on demand.
Admission plugins must be updated to guard `/ephemeralcontainers`. In Admission plugins must be updated to guard `/ephemeralcontainers`. In
particular, they should enforce the same container image policy on the `Image` particular, they should enforce the same container image policy on the
parameter as is enforced for regular containers. During the alpha phase we will `EphemeralContainer.Spec` parameter as is enforced for regular containers.
additionally support a container image whitelist as a kubelet flag to allow
cluster administrators to easily constrain debug container images.
### Additional Consideration ### Additional Consideration
@ -421,7 +370,7 @@ cluster administrators to easily constrain debug container images.
#### Goals and Non-Goals for Alpha Release #### Goals and Non-Goals for Alpha Release
We're targeting an alpha release in Kubernetes 1.10 that includes the following We're targeting an alpha release in Kubernetes 1.11 that includes the following
basic functionality: basic functionality:
* Support in the kubelet for creating debug containers in a running pod * Support in the kubelet for creating debug containers in a running pod
@ -429,14 +378,15 @@ basic functionality:
* `kubectl describe pod` will list status of debug containers running in a pod * `kubectl describe pod` will list status of debug containers running in a pod
Functionality will be hidden behind an alpha feature flag and disabled by Functionality will be hidden behind an alpha feature flag and disabled by
default. The following are explicitly out of scope for the alpha release, but default.
must be resolved prior to beta release:
* There's no guarantee that exited Debug Containers won't be garbage collected Since the kubelet stores Debug Container metadata as runtime labels, it's lost
as regular containers, so they may disappear from the list of when Debug Containers are garbage collected. For the alpha release we will rely
`EphemeralContainers`. on the apiserver to store the `EphemeralContainer` for garbage collected
* We could improve reliability of `UpdatePodOptions.OnCompleteFunc` by containers. The kubelet will preserve any `EphemeralContainer` it doesn't
prioritizing based on `SyncPodType`. recognize when updating status. In the event that a `PodStatus` is lost and we
need to regenerate it from scratch, `EphemeralContainers` will only contain
Debug Containers that have not been garbage collected.
#### Kubernetes API Changes #### Kubernetes API Changes
@ -444,8 +394,7 @@ The following changes must be implemented in the API:
1. `v1.EphemeralContainer` will be added and `v1.PodStatus` will be extended as 1. `v1.EphemeralContainer` will be added and `v1.PodStatus` will be extended as
described above. described above.
1. The new subresource will be added to the pods API, validation added and 1. The new subresource will be added to the pods API.
proxied to the kubelet.
1. The API server must check for Ephemeral Containers when validating `attach`. 1. The API server must check for Ephemeral Containers when validating `attach`.
#### kubelet Implementation #### kubelet Implementation
@ -462,19 +411,16 @@ following steps:
be identified as a debug container. be identified as a debug container.
1. `kuberuntimemanager` gains a new `StartEphemeralContainer()` which calls the 1. `kuberuntimemanager` gains a new `StartEphemeralContainer()` which calls the
existing `startContainer()`. existing `startContainer()`.
1. The kubelet gains a `RunDebugContainer()` method which accepts a 1. `syncPod()` will call `StartEphemeralContainer()` to start the Debug
`v1.EphemeralContainer` and triggers a pod sync to create the debug Container. The existing `generateAPIPodStatus()` will be updated to also
container. The existing `generateAPIPodStatus()` will be update to also populate `EphemeralContainers.Status`.
populate `EphemeralContainers`.
1. The kubelet API gains the new `/ephemeralContainers/` endpoint to create the
Debug Container.
#### kubectl changes #### kubectl changes
In anticipation of this change, [#46151](https://pr.k8s.io/46151) added a In anticipation of this change, [#46151](https://pr.k8s.io/46151) added a
`kubectl alpha` command to contain alpha features. We will add `kubectl alpha `kubectl alpha` command to contain alpha features. We will add `kubectl alpha
debug` to invoke Debug Containers. `kubectl` does not use feature gates, so debug` to invoke Debug Containers. `kubectl` does not use feature gates, so
`kubectl alpha debug` will be visible by default in `kubectl` 1.10 and return an `kubectl alpha debug` will be visible by default in `kubectl` 1.11 and return an
error when used on a cluster with the feature disabled. error when used on a cluster with the feature disabled.
`kubectl describe pod` will report the contents of `EphemeralContainers` when `kubectl describe pod` will report the contents of `EphemeralContainers` when