KEP for TTL-after-finished controller
This commit is contained in:
Janet Kuo
parent
a119d94f47
commit
80e8c2bda7
|
|
@ -1 +1 @@
|
||||||
26
|
27
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,296 @@
|
||||||
|
---
|
||||||
|
kep-number: 26
|
||||||
|
title: TTL After Finished
|
||||||
|
authors:
|
||||||
|
- "@janetkuo"
|
||||||
|
owning-sig: sig-apps
|
||||||
|
participating-sigs:
|
||||||
|
- sig-api-machinery
|
||||||
|
reviewers:
|
||||||
|
- @enisoc
|
||||||
|
- @tnozicka
|
||||||
|
approvers:
|
||||||
|
- @kow3ns
|
||||||
|
editor: TBD
|
||||||
|
creation-date: 2018-08-16
|
||||||
|
last-updated: 2018-08-16
|
||||||
|
status: provisional
|
||||||
|
see-also:
|
||||||
|
- n/a
|
||||||
|
replaces:
|
||||||
|
- n/a
|
||||||
|
superseded-by:
|
||||||
|
- n/a
|
||||||
|
---
|
||||||
|
|
||||||
|
# TTL After Finished Controller
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template.
|
||||||
|
[Tools for generating][] a table of contents from markdown are available.
|
||||||
|
|
||||||
|
* [TTL After Finished Controller](#ttl-after-finished-controller)
|
||||||
|
* [Table of Contents](#table-of-contents)
|
||||||
|
* [Summary](#summary)
|
||||||
|
* [Motivation](#motivation)
|
||||||
|
* [Goals](#goals)
|
||||||
|
* [Proposal](#proposal)
|
||||||
|
* [Concrete Use Cases](#concrete-use-cases)
|
||||||
|
* [Detailed Design](#detailed-design)
|
||||||
|
* [Feature Gate](#feature-gate)
|
||||||
|
* [API Object](#api-object)
|
||||||
|
* [Validation](#validation)
|
||||||
|
* [User Stories](#user-stories)
|
||||||
|
* [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
|
||||||
|
* [TTL Controller](#ttl-controller)
|
||||||
|
* [Finished Jobs](#finished-jobs)
|
||||||
|
* [Finished Pods](#finished-pods)
|
||||||
|
* [Owner References](#owner-references)
|
||||||
|
* [Risks and Mitigations](#risks-and-mitigations)
|
||||||
|
* [Graduation Criteria](#graduation-criteria)
|
||||||
|
* [Implementation History](#implementation-history)
|
||||||
|
|
||||||
|
[Tools for generating]: https://github.com/ekalinin/github-markdown-toc
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
We propose a TTL mechanism to limit the lifetime of finished resource objects,
|
||||||
|
including Jobs and Pods, to make it easy for users to clean up old Jobs/Pods
|
||||||
|
after they finish. The TTL timer starts when the Job/Pod finishes, and the
|
||||||
|
finished Job/Pod will be cleaned up after the TTL expires.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
In Kubernetes, finishable resources, such as Jobs and Pods, are often
|
||||||
|
frequently-created and short-lived. If a Job or Pod isn't controlled by a
|
||||||
|
higher-level resource (e.g. CronJob for Jobs or Job for Pods), or owned by some
|
||||||
|
other resources, it's difficult for the users to clean them up automatically,
|
||||||
|
and those Jobs and Pods can accumulate and overload a Kubernetes cluster very
|
||||||
|
easily. Even if we can avoid the overload issue by implementing a cluster-wide
|
||||||
|
(global) resource quota, users won't be able to create new resources without
|
||||||
|
cleaning up old ones first. See [#64470][].
|
||||||
|
|
||||||
|
The design of this proposal can be later generalized to other finishable
|
||||||
|
frequently-created, short-lived resources, such as completed Pods or finished
|
||||||
|
custom resources.
|
||||||
|
|
||||||
|
[#64470]: https://github.com/kubernetes/kubernetes/issues/64470
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
|
||||||
|
Make it easy to for the users to specify a time-based clean up mechanism for
|
||||||
|
finished resource objects.
|
||||||
|
* It's configurable at resource creation time and after the resource is created.
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
[K8s Proposal: TTL controller for finished Jobs and Pods][]
|
||||||
|
|
||||||
|
[K8s Proposal: TTL controller for finished Jobs and Pods]: https://docs.google.com/document/d/1U6h1DrRJNuQlL2_FYY_FdkQhgtTRn1kEylEOHRoESTc/edit
|
||||||
|
|
||||||
|
### Concrete Use Cases
|
||||||
|
|
||||||
|
* [Kubeflow][] needs to clean up old finished Jobs (K8s Jobs, TF Jobs, Argo
|
||||||
|
workflows, etc.), see [#718][].
|
||||||
|
|
||||||
|
* [Prow][] needs to clean up old completed Pods & finished Jobs. Currently implemented with Prow sinker.
|
||||||
|
|
||||||
|
* [Apache Spark on Kubernetes][] needs proper cleanup of terminated Spark executor Pods.
|
||||||
|
|
||||||
|
* Jenkins Kubernetes plugin creates slave pods that execute builds. It needs a better way to clean up old completed Pods.
|
||||||
|
|
||||||
|
[Kubeflow]: https://github.com/kubeflow
|
||||||
|
[#718]: https://github.com/kubeflow/tf-operator/issues/718
|
||||||
|
[Prow]: https://github.com/kubernetes/test-infra/tree/master/prow
|
||||||
|
[Apache Spark on Kubernetes]: http://spark.apache.org/docs/latest/running-on-kubernetes.html
|
||||||
|
|
||||||
|
### Detailed Design
|
||||||
|
|
||||||
|
#### Feature Gate
|
||||||
|
|
||||||
|
This will be launched as an alpha feature first, with feature gate
|
||||||
|
`TTLAfterFinished`.
|
||||||
|
|
||||||
|
#### API Object
|
||||||
|
|
||||||
|
We will add the following API fields to `JobSpec` (`Job`'s `.spec`).
|
||||||
|
|
||||||
|
```go
|
||||||
|
type JobSpec struct {
|
||||||
|
// ttlSecondsAfterFinished limits the lifetime of a Job that has finished
|
||||||
|
// execution (either Complete or Failed). If this field is set, once the Job
|
||||||
|
// finishes, it will be deleted after ttlSecondsAfterFinished expires. When
|
||||||
|
// the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will
|
||||||
|
// be honored. If this field is unset, ttlSecondsAfterFinished will not
|
||||||
|
// expire. If this field is set to zero, ttlSecondsAfterFinished expires
|
||||||
|
// immediately after the Job finishes.
|
||||||
|
// This field is alpha-level and is only honored by servers that enable the
|
||||||
|
// TTLAfterFinished feature.
|
||||||
|
// +optional
|
||||||
|
TTLSecondsAfterFinished *int32
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This allows Jobs to be cleaned up after they finish and provides time for
|
||||||
|
asynchronous clients to observe Jobs' final states before they are deleted.
|
||||||
|
|
||||||
|
|
||||||
|
Similarly, we will add the following API fields to `PodSpec` (`Pod`'s `.spec`).
|
||||||
|
|
||||||
|
```go
|
||||||
|
type PodSpec struct {
|
||||||
|
// ttlSecondsAfterFinished limits the lifetime of a Pod that has finished
|
||||||
|
// execution (either Succeeded or Failed). If this field is set, once the Pod
|
||||||
|
// finishes, it will be deleted after ttlSecondsAfterFinished expires. When
|
||||||
|
// the Pod is being deleted, its lifecycle guarantees (e.g. finalizers) will
|
||||||
|
// be honored. If this field is unset, ttlSecondsAfterFinished will not
|
||||||
|
// expire. If this field is set to zero, ttlSecondsAfterFinished expires
|
||||||
|
// immediately after the Pod finishes.
|
||||||
|
// This field is alpha-level and is only honored by servers that enable the
|
||||||
|
// TTLAfterFinished feature.
|
||||||
|
// +optional
|
||||||
|
TTLSecondsAfterFinished *int32
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
##### Validation
|
||||||
|
|
||||||
|
Because Job controller depends on Pods to exist to work correctly. In Job
|
||||||
|
validation, `ttlSecondsAfterFinished` of its pod template shouldn't be set, to
|
||||||
|
prevent users from breaking their Jobs. Users should set TTL seconds on a Job,
|
||||||
|
instead of Pods owned by a Job.
|
||||||
|
|
||||||
|
It is common for higher level resources to call generic PodSpec validation;
|
||||||
|
therefore, in PodSpec validation, `ttlSecondsAfterFinished` is only allowed to
|
||||||
|
be set on a PodSpec with a `restartPolicy` that is either `OnFailure` or `Never`
|
||||||
|
(i.e. not `Always`).
|
||||||
|
|
||||||
|
### User Stories
|
||||||
|
|
||||||
|
The users keep creating Jobs in a small Kubernetes cluster with 4 nodes.
|
||||||
|
The Jobs accumulates over time, and 1 year later, the cluster ended up with more
|
||||||
|
than 100k old Jobs. This caused etcd hiccups, long high latency etcd requests,
|
||||||
|
and eventually made the cluster unavailable.
|
||||||
|
|
||||||
|
The problem could have been avoided easily with TTL controller for Jobs.
|
||||||
|
|
||||||
|
The steps are as easy as:
|
||||||
|
|
||||||
|
1. When creating Jobs, the user sets Jobs' `.spec.ttlSecondsAfterFinished` to
|
||||||
|
3600 (i.e. 1 hour).
|
||||||
|
1. The user deploys Jobs as usual.
|
||||||
|
1. After a Job finishes, the result is observed asynchronously within an hour
|
||||||
|
and stored elsewhere.
|
||||||
|
1. The TTL collector cleans up Jobs 1 hour after they complete.
|
||||||
|
|
||||||
|
### Implementation Details/Notes/Constraints
|
||||||
|
|
||||||
|
#### TTL Controller
|
||||||
|
We will add a TTL controller for finished Jobs and finished Pods. We considered
|
||||||
|
adding it in Job controller, but decided not to, for the following reasons:
|
||||||
|
|
||||||
|
1. Job controller should focus on managing Pods based on the Job's spec and pod
|
||||||
|
template, but not cleaning up Jobs.
|
||||||
|
1. We also need the TTL controller to clean up finished Pods, and we consider
|
||||||
|
generalizing TTL controller later for custom resources.
|
||||||
|
|
||||||
|
The TTL controller utilizes informer framework, watches all Jobs and Pods, and
|
||||||
|
read Jobs and Pods from a local cache.
|
||||||
|
|
||||||
|
#### Finished Jobs
|
||||||
|
|
||||||
|
When a Job is created or updated:
|
||||||
|
|
||||||
|
1. Check its `.status.conditions` to see if it has finished (`Complete` or
|
||||||
|
`Failed`). If it hasn't finished, do nothing.
|
||||||
|
1. Otherwise, if the Job has finished, check if Job's
|
||||||
|
`.spec.ttlSecondsAfterFinished` field is set. Do nothing if the TTL field is
|
||||||
|
not set.
|
||||||
|
1. Otherwise, if the TTL field is set, check if the TTL has expired, i.e.
|
||||||
|
`.spec.ttlSecondsAfterFinished` + the time when the Job finishes
|
||||||
|
(`.status.conditions.lastTransitionTime`) > now.
|
||||||
|
1. If the TTL hasn't expired, delay re-enqueuing the Job after a computed amount
|
||||||
|
of time when it will expire. The computed time period is:
|
||||||
|
(`.spec.ttlSecondsAfterFinished` + `.status.conditions.lastTransitionTime` -
|
||||||
|
now).
|
||||||
|
1. If the TTL has expired, `GET` the Job from API server to do final sanity
|
||||||
|
checks before deleting it.
|
||||||
|
1. Check if the freshly got Job's TTL has expired. This field may be updated
|
||||||
|
before TTL controller observes the new value in its local cache.
|
||||||
|
* If it hasn't expired, it is not safe to delete the Job. Delay re-enqueue
|
||||||
|
the Job after a computed amount of time when it will expire.
|
||||||
|
1. Delete the Job if passing the sanity checks.
|
||||||
|
|
||||||
|
#### Finished Pods
|
||||||
|
|
||||||
|
When a Pod is created or updated:
|
||||||
|
1. Check its `.status.phase` to see if it has finished (`Succeeded` or `Failed`).
|
||||||
|
If it hasn't finished, do nothing.
|
||||||
|
1. Otherwise, if the Pod has finished, check if Pod's
|
||||||
|
`.spec.ttlSecondsAfterFinished` field is set. Do nothing if the TTL field is
|
||||||
|
not set.
|
||||||
|
1. Otherwise, if the TTL field is set, check if the TTL has expired, i.e.
|
||||||
|
`.spec.ttlSecondsAfterFinished` + the time when the Pod finishes (max of all
|
||||||
|
of its containers termination time
|
||||||
|
`.containerStatuses.state.terminated.finishedAt`) > now.
|
||||||
|
1. If the TTL hasn't expired, delay re-enqueuing the Pod after a computed amount
|
||||||
|
of time when it will expire. The computed time period is:
|
||||||
|
(`.spec.ttlSecondsAfterFinished` + the time when the Pod finishes - now).
|
||||||
|
1. If the TTL has expired, `GET` the Pod from API server to do final sanity
|
||||||
|
checks before deleting it.
|
||||||
|
1. Check if the freshly got Pod's TTL has expired. This field may be updated
|
||||||
|
before TTL controller observes the new value in its local cache.
|
||||||
|
* If it hasn't expired, it is not safe to delete the Pod. Delay re-enqueue
|
||||||
|
the Pod after a computed amount of time when it will expire.
|
||||||
|
1. Delete the Pod if passing the sanity checks.
|
||||||
|
|
||||||
|
#### Owner References
|
||||||
|
|
||||||
|
We have considered making TTL controller leave a Job/Pod around even after its
|
||||||
|
TTL expires, if the Job/Pod has any owner specified in its
|
||||||
|
`.metadata.ownerReferences`.
|
||||||
|
|
||||||
|
We decided not to block deletion on owners, because the purpose of
|
||||||
|
`.metadata.ownerReferences` is for cascading deletion, but not for keeping an
|
||||||
|
owner's dependents alive. If the Job is owned by a CronJob, the Job can be
|
||||||
|
cleaned up based on CronJob's history limit (i.e. the number of dependent Jobs
|
||||||
|
to keep), or CronJob can choose not to set history limit but set the TTL of its
|
||||||
|
Job template to clean up Jobs after TTL expires instead of based on the history
|
||||||
|
limit capacity.
|
||||||
|
|
||||||
|
Therefore, a Job/Pod can be deleted after its TTL expires, even if it still has
|
||||||
|
owners.
|
||||||
|
|
||||||
|
Similarly, the TTL won't block deletion from generic garbage collector. This
|
||||||
|
means that when a Job's or Pod's owners are gone, generic garbage collector will
|
||||||
|
delete it, even if it hasn't finished or its TTL hasn't expired.
|
||||||
|
|
||||||
|
### Risks and Mitigations
|
||||||
|
|
||||||
|
Risks:
|
||||||
|
* Time skew may cause TTL controller to clean up resource objects at the wrong
|
||||||
|
time.
|
||||||
|
|
||||||
|
Mitigations:
|
||||||
|
* In Kubernetes, it's required to run NTP on all nodes ([#6159][]) to avoid time
|
||||||
|
skew. We will also document this risk.
|
||||||
|
|
||||||
|
[#6159]: https://github.com/kubernetes/kubernetes/issues/6159#issuecomment-93844058
|
||||||
|
|
||||||
|
## Graduation Criteria
|
||||||
|
|
||||||
|
We want to implement this feature for Pods/Jobs first to gather feedback, and
|
||||||
|
decide whether to generalize it to custom resources. This feature can be
|
||||||
|
promoted to beta after we finalize the decision for whether to generalize it or
|
||||||
|
not, and when it satisfies users' need for cleaning up finished resource
|
||||||
|
objects, without regressions.
|
||||||
|
|
||||||
|
This will be promoted to GA once it's gone a sufficient amount of time as beta
|
||||||
|
with no changes.
|
||||||
|
|
||||||
|
[umbrella issues]: https://github.com/kubernetes/kubernetes/issues/42752
|
||||||
|
|
||||||
|
## Implementation History
|
||||||
|
|
||||||
|
TBD
|
||||||
Loading…
Reference in New Issue