kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1813 from jbeda/kep-update 

Update/clarify KEP process
This commit is contained in:
k8s-ci-robot 2018-02-22 08:41:40 -08:00 committed by GitHub
parent 320c827df9 d23b9c22e8
commit f2b3dd8263
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 211 additions and 156 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
keps/0000-kep-template.md
View File

@ -1,5 +1,5 @@
---
kep-number: draft-YYYYMMDD
kep-number: 0
title: My First KEP
authors:
- "@janedoe"
@ -16,7 +16,7 @@ approvers:
editor: TBD
creation-date: yyyy-mm-dd
last-updated: yyyy-mm-dd
status: accepted
status: provisional
see-also:
- KEP-1
- KEP-2
@ -37,16 +37,25 @@ The title should be lowercased and spaces/punctuation should be replaced with `-
As the KEP is approved and an official KEP number is allocated, the file should be renamed.
To get started with this template:
* Make a copy of this template.
Name it `draft-YYYYMMDD-my-title.md`.
* Create a PR in the [`kubernetes/community`](https://github.com/kubernetes/community) repo.
* Check in early.
Do this once the document holds together and general direction is understood by many in the sponsoring SIG.
View anything marked as a draft as a working document.
1. **Pick a hosting SIG.**
Make sure that the problem space is something the SIG is interested in taking up.
KEPs should not be checked in without a sponsoring SIG.
1. **Allocate a KEP number.**
Do this by (a) taking the next number in the `NEXT_KEP_NUMBER` file and (b) incrementing that number.
Include the updated `NEXT_KEP_NUMBER` file in your PR.
1. **Make a copy of this template.**
Name it `NNNN-YYYYMMDD-my-title.md` where `NNNN` is the KEP number that was allocated.
1. **Fill out the "overview" sections.**
This includes the Summary and Motivation sections.
These should be easy if you've preflighted the idea of the KEP with the appropriate SIG.
1. **Create a PR.**
Assign it to folks in the SIG that are sponsoring this process.
1. **Merge early.**
Avoid getting hung up on specific details and instead aim to get the goal of the KEP merged quickly.
The best way to do this is to just start with the "Overview" sections and fill out details incrementally in follow on PRs.
View anything marked as a `provisional` as a working document and subject to change.
Aim for single topic PRs to keep discussions focused.
If you disagree with what is already in a document, open a new PR with suggested changes.
* A KEP number can be assigned at any time by (even in the first PR) (a) taking the next number in the NEXT_KEP_NUMBER file and (b) incrementing that number.
These PRs should be approved quickly to minimize merge conflicts.
The canonical place for the latest set of instructions (and the likely source of this file) is [here](/keps/0000-kep-template.md).

4
keps/0001-kubernetes-enhancement-proposal-process.md
View File

@ -164,7 +164,7 @@ Metadata items:
KEP filename. See the template for instructions and details.
* **status** Required
* The current state of the KEP.
* Must be one of `accepted`, `implementable`, `implemented`, `deferred`, `rejected`, `withdrawn`, or `replaced`.
* Must be one of `provisional`, `implementable`, `implemented`, `deferred`, `rejected`, `withdrawn`, or `replaced`.
* **authors** Required
* A list of authors for the KEP.
This is simply the github ID.
@ -222,7 +222,7 @@ Metadata items:
A KEP has the following states
- `accepted`: The KEP has been proposed and is actively being defined.
- `provisional`: The KEP has been proposed and is actively being defined.
This is the starting state while the KEP is being fleshed out and actively defined and discussed.
The owning SIG has accepted that this is work that needs to be done.
- `implementable`: The approvers have approved this KEP for implementation.

142
keps/0002-controller-manager.md
View File

@ -1,5 +1,5 @@
```
kep-number: draft-20171127
---
kep-number: 2
title: Cloud Provider Controller Manager
authors:
- "@cheftako"
@ -15,10 +15,10 @@ reviewers:
approvers:
- "@thockin"
editor: TBD
status: approved
status: provisional
replaces:
- contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md
```
---
# Remove Cloud Provider Code From Kubernetes Core
@ -47,35 +47,35 @@ replaces:
## Summary
We want to remove any cloud provider specific logic from the kubernetes/kubernetes repo. We want to restructure the code
to make is easy for any cloud provider to extend the kubernetes core in a consistent manner for their cloud. New cloud
to make is easy for any cloud provider to extend the kubernetes core in a consistent manner for their cloud. New cloud
providers should look at the [Creating a Custom Cluster from Scratch](https://kubernetes.io/docs/getting-started-guides/scratch/#cloud-provider)
and the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L31)
which will need to be implemented.
and the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L31)
which will need to be implemented.
## Motivation
We are trying to remove any dependencies from Kubernetes Core to any specific cloud provider. Currently we have seven
such dependencies. To prevent this number from growing we have locked Kubernetes Core to the addition of any new
dependencies. This means all new cloud providers have to implement all their pieces outside of the Core.
We are trying to remove any dependencies from Kubernetes Core to any specific cloud provider. Currently we have seven
such dependencies. To prevent this number from growing we have locked Kubernetes Core to the addition of any new
dependencies. This means all new cloud providers have to implement all their pieces outside of the Core.
However everyone still ends up consuming the current set of seven in repo dependencies. For the seven in repo cloud
providers any changes to their specific cloud provider code requires OSS PR approvals and a deployment to get those
providers any changes to their specific cloud provider code requires OSS PR approvals and a deployment to get those
changes in to an official build. The relevant dependencies require changes in the following areas.
- [Kube Controller Manager](https://kubernetes.io/docs/reference/generated/kube-controller-manager/) - Track usages of [CMServer.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/options/options.go)
- [Kube Controller Manager](https://kubernetes.io/docs/reference/generated/kube-controller-manager/) - Track usages of [CMServer.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/options/options.go)
- [API Server](https://kubernetes.io/docs/reference/generated/kube-apiserver/) - Track usages of [ServerRunOptions.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go)
- [Kubelet](https://kubernetes.io/docs/reference/generated/kubelet/) - Track usages of [KubeletFlags.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go)
- [How Cloud Provider Functionality is deployed to and enabled in the cluster](https://kubernetes.io/docs/setup/pick-right-solution/#hosted-solutions) - Track usage from [PROVIDER_UTILS](https://github.com/kubernetes/kubernetes/blob/master/cluster/kube-util.sh)
For the cloud providers who are in repo, moving out would allow them to more quickly iterate on their solution and
decouple cloud provider fixes from open source releases. Moving the cloud provider code out of the open source
processes means that these processes do not need to load/run unnecessary code for the environment they are in.
For the cloud providers who are in repo, moving out would allow them to more quickly iterate on their solution and
decouple cloud provider fixes from open source releases. Moving the cloud provider code out of the open source
processes means that these processes do not need to load/run unnecessary code for the environment they are in.
We would like to abstract a core controller manager library so help standardize the behavior of the cloud
controller managers produced by each cloud provider. We would like to minimize the number and scope of controllers
controller managers produced by each cloud provider. We would like to minimize the number and scope of controllers
running in the cloud controller manager so as to minimize the surface area for per cloud provider deviation.
### Goals
- Get to a point where we do not load the cloud interface for any of kubernetes core processes.
- Get to a point where we do not load the cloud interface for any of kubernetes core processes.
- Remove all cloud provider specific code from kubernetes/kubernetes.
- Have a generic controller manager library available for use by the per cloud provider controller managers.
- Move the cloud provider specific controller manager logic into repos appropriate for those cloud providers.
@ -96,33 +96,33 @@ Forcing cloud providers to use the generic cloud manager.
### Controller Manager Changes
For the controller manager we would like to create a set of common code which can be used by both the cloud controller
manager and the kube controller manager. The cloud controller manager would then be responsible for running controllers
manager and the kube controller manager. The cloud controller manager would then be responsible for running controllers
whose function is specific to cloud provider functionality. The kube controller manager would then be responsible
for running all controllers whose function was not related to a cloud provider.
In order to create a 100% cloud independent controller manager, the controller-manager will be split into multiple binaries.
1. Cloud dependent controller-manager binaries
2. Cloud independent controller-manager binaries - This is the existing `kube-controller-manager` that is being shipped
2. Cloud independent controller-manager binaries - This is the existing `kube-controller-manager` that is being shipped
with kubernetes releases.
The cloud dependent binaries will run those loops that rely on cloudprovider in a separate process(es) within the kubernetes control plane.
The cloud dependent binaries will run those loops that rely on cloudprovider in a separate process(es) within the kubernetes control plane.
The rest of the controllers will be run in the cloud independent controller manager.
The decision to run entire controller loops, rather than only the very minute parts that rely on cloud provider was made
because it makes the implementation simple. Otherwise, the shared data structures and utility functions have to be
disentangled, and carefully separated to avoid any concurrency issues. This approach among other things, prevents code
The decision to run entire controller loops, rather than only the very minute parts that rely on cloud provider was made
because it makes the implementation simple. Otherwise, the shared data structures and utility functions have to be
disentangled, and carefully separated to avoid any concurrency issues. This approach among other things, prevents code
duplication and improves development velocity.
Note that the controller loop implementation will continue to reside in the core repository. It takes in
cloudprovider.Interface as an input in its constructor. Vendor maintained cloud-controller-manager binary could link
Note that the controller loop implementation will continue to reside in the core repository. It takes in
cloudprovider.Interface as an input in its constructor. Vendor maintained cloud-controller-manager binary could link
these controllers in, as it serves as a reference form of the controller implementation.
There are four controllers that rely on cloud provider specific code. These are node controller, service controller,
route controller and attach detach controller. Copies of each of these controllers have been bundled together into
one binary. The cloud dependent binary registers itself as a controller, and runs the cloud specific controller loops
There are four controllers that rely on cloud provider specific code. These are node controller, service controller,
route controller and attach detach controller. Copies of each of these controllers have been bundled together into
one binary. The cloud dependent binary registers itself as a controller, and runs the cloud specific controller loops
with the user-agent named "external-controller-manager".
RouteController and serviceController are entirely cloud specific. Therefore, it is really simple to move these two
RouteController and serviceController are entirely cloud specific. Therefore, it is really simple to move these two
controller loops out of the cloud-independent binary and into the cloud dependent binary.
NodeController does a lot more than just talk to the cloud. It does the following operations -
@ -131,12 +131,12 @@ NodeController does a lot more than just talk to the cloud. It does the followin
2. Monitor Node Status
3. Node Pod Eviction
While Monitoring Node status, if the status reported by kubelet is either 'ConditionUnknown' or 'ConditionFalse', then
the controller checks if the node has been deleted from the cloud provider. If it has already been deleted from the
cloud provider, then it deletes the nodeobject without waiting for the `monitorGracePeriod` amount of time. This is the
While Monitoring Node status, if the status reported by kubelet is either 'ConditionUnknown' or 'ConditionFalse', then
the controller checks if the node has been deleted from the cloud provider. If it has already been deleted from the
cloud provider, then it deletes the nodeobject without waiting for the `monitorGracePeriod` amount of time. This is the
only operation that needs to be moved into the cloud dependent controller manager.
Finally, The attachDetachController is tricky, and it is not simple to disentangle it from the controller-manager
Finally, The attachDetachController is tricky, and it is not simple to disentangle it from the controller-manager
easily, therefore, this will be addressed with Flex Volumes (Discussed under a separate section below)
@ -160,17 +160,17 @@ Among these controller loops, the following are cloud provider dependent.
- [routeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/route)
- [serviceController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/service)
The nodeController uses the cloudprovider to check if a node has been deleted from the cloud. If cloud provider reports
a node as deleted, then this controller immediately deletes the node from kubernetes. This check removes the need to
The nodeController uses the cloudprovider to check if a node has been deleted from the cloud. If cloud provider reports
a node as deleted, then this controller immediately deletes the node from kubernetes. This check removes the need to
wait for a specific amount of time to conclude that an inactive node is actually dead.
The volumeController uses the cloudprovider to create, delete, attach and detach volumes to nodes. For instance, the
logic for provisioning, attaching, and detaching a EBS volume resides in the AWS cloudprovider. The volumeController
The volumeController uses the cloudprovider to create, delete, attach and detach volumes to nodes. For instance, the
logic for provisioning, attaching, and detaching a EBS volume resides in the AWS cloudprovider. The volumeController
uses this code to perform its operations.
The routeController configures routes for hosts in the cloud provider.
The serviceController maintains a list of currently active nodes, and is responsible for creating and deleting
The serviceController maintains a list of currently active nodes, and is responsible for creating and deleting
LoadBalancers in the underlying cloud.
### Kubelet Changes
@ -185,28 +185,28 @@ Moving on to the kubelet, the following cloud provider dependencies exist in kub
- Periodically poll the cloud provider to figure out if the node has any new IP addresses associated with it
- It sets a condition that makes the node unschedulable until cloud routes are configured.
- It allows the cloud provider to post process DNS settings
The majority of the calls by the kubelet to the cloud is done during the initialization of the Node Object. The other
The majority of the calls by the kubelet to the cloud is done during the initialization of the Node Object. The other
uses are for configuring Routes (in case of GCE), scrubbing DNS, and periodically polling for IP addresses.
All of the above steps, except the Node initialization step can be moved into a controller. Specifically, IP address
All of the above steps, except the Node initialization step can be moved into a controller. Specifically, IP address
polling, and configuration of Routes can be moved into the cloud dependent controller manager.
[Scrubbing DNS was found to be redundant](https://github.com/kubernetes/kubernetes/pull/36785). So, it can be disregarded. It is being removed.
Finally, Node initialization needs to be addressed. This is the trickiest part. Pods will be scheduled even on
uninitialized nodes. This can lead to scheduling pods on incompatible zones, and other weird errors. Therefore, an
approach is needed where kubelet can create a Node, but mark it as "NotReady". Then, some asynchronous process can
update it and mark it as ready. This is now possible because of the concept of Taints.
This approach requires kubelet to be started with known taints. This will make the node unschedulable until these
taints are removed. The external cloud controller manager will asynchronously update the node objects and remove the
taints.
Finally, Node initialization needs to be addressed. This is the trickiest part. Pods will be scheduled even on
uninitialized nodes. This can lead to scheduling pods on incompatible zones, and other weird errors. Therefore, an
approach is needed where kubelet can create a Node, but mark it as "NotReady". Then, some asynchronous process can
update it and mark it as ready. This is now possible because of the concept of Taints.
This approach requires kubelet to be started with known taints. This will make the node unschedulable until these
taints are removed. The external cloud controller manager will asynchronously update the node objects and remove the
taints.
### API Server Changes
Finally, in the kube-apiserver, the cloud provider is used for transferring SSH keys to all of the nodes, and within an a
dmission controller for setting labels on persistent volumes.
dmission controller for setting labels on persistent volumes.
Kube-apiserver uses the cloud provider for two purposes
@ -215,30 +215,30 @@ Kube-apiserver uses the cloud provider for two purposes
### Volume Managent Changes
Volumes need cloud providers, but they only need **specific** cloud providers. The majority of volume management logic
resides in the controller manager. These controller loops need to be moved into the cloud-controller manager. The cloud
controller manager also needs a mechanism to read parameters for initilization from cloud config. This can be done via
config maps.
Volumes need cloud providers, but they only need **specific** cloud providers. The majority of volume management logic
resides in the controller manager. These controller loops need to be moved into the cloud-controller manager. The cloud
controller manager also needs a mechanism to read parameters for initilization from cloud config. This can be done via
config maps.
There are two entirely different approach to refactoring volumes -
[Flex Volumes](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md) and
There are two entirely different approach to refactoring volumes -
[Flex Volumes](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md) and
[CSI Container Storage Interface](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md). There is an undergoing effort to move all
of the volume logic from the controller-manager into plugins called Flex Volumes. In the Flex volumes world, all of the
vendor specific code will be packaged in a separate binary as a plugin. After discussing with @thockin, this was
of the volume logic from the controller-manager into plugins called Flex Volumes. In the Flex volumes world, all of the
vendor specific code will be packaged in a separate binary as a plugin. After discussing with @thockin, this was
decidedly the best approach to remove all cloud provider dependency for volumes out of kubernetes core. Some of the discovery
information for this can be found at [https://goo.gl/CtzpVm](https://goo.gl/CtzpVm).
### Deployment Changes
This change will introduce new binaries to the list of binaries required to run kubernetes. The change will be designed
such that these binaries can be installed via `kubectl apply -f` and the appropriate instances of the binaries will be
This change will introduce new binaries to the list of binaries required to run kubernetes. The change will be designed
such that these binaries can be installed via `kubectl apply -f` and the appropriate instances of the binaries will be
running.
Issues such as monitoring, configuring the new binaries will generally be left to cloud provider. However they should
ensure that test runs upload the logs for these new processes to [test grid](https://k8s-testgrid.appspot.com/).
Applying the cloud controller manager is the only step that is different in the upgrade process.
In order to complete the upgrade process, you need to apply the cloud-controller-manager deployment to the setup.
Applying the cloud controller manager is the only step that is different in the upgrade process.
In order to complete the upgrade process, you need to apply the cloud-controller-manager deployment to the setup.
A deployment descriptor file will be provided with this change. You need to apply this change using
```
@ -247,7 +247,7 @@ kubectl apply -f cloud-controller-manager.yml
This will start the cloud specific controller manager in your kubernetes setup.
The downgrade steps are also the same as before for all the components except the cloud-controller-manager.
The downgrade steps are also the same as before for all the components except the cloud-controller-manager.
In case of the cloud-controller-manager, the deployment should be deleted using
```
@ -263,7 +263,7 @@ Make sure that you consider the impact of this feature from the point of view of
## Graduation Criteria
How will we know that this has succeeded?
Gathering user feedback is crucial for building high quality experiences and SIGs have the important responsibility of
Gathering user feedback is crucial for building high quality experiences and SIGs have the important responsibility of
setting milestones for stability and completeness.
Hopefully the content previously contained in [umbrella issues][] will be tracked in the `Graduation Criteria` section.
@ -287,6 +287,6 @@ Major milestones might include
One alternate to consider is the use of a side-car. The cloud-interface in tree could then be a [GRPC](https://github.com/grpc/grpc-go)
call out to that side-car. We could then leave the Kube API Server, Kube Controller Manager and Kubelet pretty much as is.
We would still need separate repos to hold the code for the side care and to handle cluster setup for the cloud provider.
However we believe that different cloud providers will (already) want different control loops. As such we are likely to need
something like the cloud controller manager anyway. From the perspective it seems easier to centralize the effort in that
direction. In addition it should limit the proliferation of new processes across the entire cluster.
However we believe that different cloud providers will (already) want different control loops. As such we are likely to need
something like the cloud controller manager anyway. From the perspective it seems easier to centralize the effort in that
direction. In addition it should limit the proliferation of new processes across the entire cluster.

58
keps/README.md
View File

@ -1,7 +1,59 @@
# Kubernetes Enhancement Proposals (KEPs)
This directory contains both the template and process for KEPs.
A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordiante on new efforts for the Kubernetes project.
You can read the full details of the project in [KEP-1](0001-kubernetes-enhancement-proposal-process.md).
This process is still in an _alpha_ state and is opt-in for those that want to provide feedback for the process.
This process is still in a _beta_ state and is opt-in for those that want to provide feedback for the process.
See [KEP-1](0001-kubernetes-enhancement-proposal-process.md) for details of this process.
## Quick start for the KEP process
1. Socialize an idea with a sponsoring SIG.
Make sure that others think the work is worth taking up and will help review the KEP and any code changes required.
2. Follow the process outlined in the [KEP template](0000-kep-template.md)
## FAQs
### Do I have to use the KEP process?
No... but we hope that you will.
Over time having a rich set of KEPs in one place will make it easier for people to track what is going in the community and find a structured historic record.
KEPs are only required when the changes are wide ranging and impact most of the project.
These changes are usually coordinated through SIG-Architecture.
It is up to any specific SIG if they want to use the KEP process and when.
The process is available to SIGs to use but not required.
### Why would I want to use the KEP process?
Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community.
As such, we want to build a well curated set of clear proposals in a common format with useful metadata.
Benefits to KEP users (in the limit):
* Exposure on a kubernetes blessed web site that is findable via web search engines.
* Cross indexing of KEPs so that users can find connections and the current status of any KEP.
* A clear process with approvers and reviewers for making decisions.
This will lead to more structured decisions that stick as there is a discoverable record around the decisions.
We are inspired by IETF RFCs, Pyton PEPs and Rust RFCs.
See [KEP-1](0001-kubernetes-enhancement-proposal-process.md) for more details.
### Do I put my KEP in the root KEP directory or a SIG subdirectory?
If the KEP is mainly restricted to one SIG's purview then it should be in a KEP directory for that SIG.
If the KEP is widely impacting much of Kubernetes, it should be put at the root of this directory.
If in doubt ask [SIG-Architecture](../sig-architecture/README.md) and they can advise.
### What will it take for KEPs to "graduate" out of "beta"?
Things we'd like to see happen to consider KEPs well on their way:
* A set of KEPs that show healthy process around describing an effort and recording decisions in a reasonable amount of time.
* KEPs exposed on a searchable and indexable web site.
* Presubmit checks for KEPs around metadata format and markdown validity.
Even so, the process can evolve. As we find new techniques we can improve our processes.
### My FAQ isn't answered here!
The KEP process is still evolving!
If something is missing or not answered here feel free to reach out to [SIG-Architecture](../sig-architecture/README.md).
If you want to propose a change to the KEP process you can open a PR on [KEP-1](0001-kubernetes-enhancement-proposal-process.md) with your proposal.

23
keps/sig-cluster-lifecycle/0003-cluster-api.md
View File

@ -1,11 +1,7 @@
# Kubernetes Cluster Management API
## Metadata
```
---
kep-number: 3
title: Kubernetes Cluster Management API
status: accepted
status: provisional
authors:
- "@roberthbailey"
- "@pipejakob"
@ -14,12 +10,13 @@ reviewers:
- "@thockin"
approvers:
- "@roberthbailey"
editor:
editor:
- "@roberthbailey"
creation-date: 2018-01-19
last-updated: 2018-01-22
---
```
# Kubernetes Cluster Management API
## Table of Contents
@ -48,16 +45,16 @@ last-updated: 2018-01-22
## Summary
We are building a set of Kubernetes cluster management APIs to enable common cluster lifecycle operations (install, upgrade, repair, delete) across disparate environments.
We represent nodes and other infrastructure in Kubernetes-style APIs to enable higher level controllers to update the desired state of the cluster (e.g. the autoscaling controller requesting additional machines) and reconcile the world with that state (e.g. communicating with cloud providers to create or delete virtual machines).
We are building a set of Kubernetes cluster management APIs to enable common cluster lifecycle operations (install, upgrade, repair, delete) across disparate environments.
We represent nodes and other infrastructure in Kubernetes-style APIs to enable higher level controllers to update the desired state of the cluster (e.g. the autoscaling controller requesting additional machines) and reconcile the world with that state (e.g. communicating with cloud providers to create or delete virtual machines).
With the full state of the cluster represented as API objects, Kubernetes installers can use them as a common configuration language, and more sophisticated tooling can be built in an environment-agnostic way.
## Motivation
Kubernetes has a common set of APIs (see the [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md)) to orchestrate containers regardless of deployment mechanism or cloud provider.
Kubernetes also has APIs for handling some infrastructure, like load-balancers, ingress rules, or persistent volumes, but not for creating new machines.
Kubernetes has a common set of APIs (see the [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md)) to orchestrate containers regardless of deployment mechanism or cloud provider.
Kubernetes also has APIs for handling some infrastructure, like load-balancers, ingress rules, or persistent volumes, but not for creating new machines.
As a result, the deployment mechanisms that manage Kubernetes clusters each have unique APIs and implementations for how to handle lifecycle events like cluster creation or deletion, master upgrades, and node upgrades.
Additionally, the cluster-autoscaler is responsible not only for determining when the cluster should be scaled, but also responsible for adding capacity to the cluster by interacting directly with the cloud provider to perform the scaling.
Additionally, the cluster-autoscaler is responsible not only for determining when the cluster should be scaled, but also responsible for adding capacity to the cluster by interacting directly with the cloud provider to perform the scaling.
When another component needs to create or destroy virtual machines, like the node auto provisioner, it would similarly need to reimplement the logic for interacting with the supported cloud providers (or reuse the same code to prevent duplication).
### Goals
@ -81,7 +78,7 @@ When another component needs to create or destroy virtual machines, like the nod
* Should a single Kubernetes cluster only house definitions for itself?
* If so, that removes the ability to have a single cluster control the reconciliation of infrastructure for other clusters.
* However, with the concurrent [Cluster Registry](https://docs.google.com/a/google.com/document/d/1Oi9EO3Jwtp69obakl-9YpLkP764GZzsz95XJlX1a960/edit) project, a good separation of responsibilities would be that the Cluster Registry API is responsible for indexing multiple clusters, each of which would only have to know about itself. In order to achieve cross-cluster reconciliation, a controller would need to integrate with a Cluster Registry for discovery.
* Should a cluster’s control plane definition should be housed within that same cluster.
* Should a cluster’s control plane definition should be housed within that same cluster.
* If the control plane becomes unhealthy, then it won’t be able to rectify itself without external intervention. If the control plane configuration lives elsewhere, and the controllers reconciling its state are able to act in the face of control plane failure, then this API could be used to fix a misconfigured control plane that is unresponsive.
* Should our representation of Nodes allow declarative versioning of non-Kubernetes packages, like the container runtime, the Linux kernel, etc.?
* It potentially enables the use case of smaller, in-place upgrades to nodes without changing the node image.

111
keps/sig-cluster-lifecycle/0004-bootstrap-checkpointing.md
View File

@ -1,19 +1,15 @@
# Kubernetes Bootstrap Checkpointing Proposal
## Metadata
```
---
kep-number: 4
title: Kubernetes Bootstrap Checkpointing Proposal
status: implemented
authors:
- "@timothysc"
owning-sig: sig-cluster-lifecycle
participating-sigs:
owning-sig: sig-cluster-lifecycle
participating-sigs:
- sig-node
reviewers:
- "@yujuhong"
- "@luxas"
- "@luxas"
- "@roberthbailey"
approvers:
- "@yujuhong"
@ -22,8 +18,9 @@ editor:
name: @timothysc
creation-date: 2017-10-20
last-updated: 2018-01-23
---
```
# Kubernetes Bootstrap Checkpointing Proposal
## Table of Contents
@ -39,19 +36,19 @@ last-updated: 2018-01-23
## Summary
There are several methods to deploy a kubernetes cluster, one method that
offers some unique advantages is self hosting. The purpose of this proposal
is to outline a method to checkpoint specific annotated pods, namely the
control plane components, for the purpose of enabling self hosting.
There are several methods to deploy a kubernetes cluster, one method that
offers some unique advantages is self hosting. The purpose of this proposal
is to outline a method to checkpoint specific annotated pods, namely the
control plane components, for the purpose of enabling self hosting.
The details of self hosting are beyond the scope of this proposal, and are
outlined in the references listed below:
The details of self hosting are beyond the scope of this proposal, and are
outlined in the references listed below:
- [Self Hosted Kubernetes][0]
- [Kubeadm Upgrades][1]
Extra details on this proposal, and its history, can be found in the links
below:
Extra details on this proposal, and its history, can be found in the links
below:
- [Bootstrap Checkpointing Draft 1][2]
- [Bootstrap Checkpointing Draft 2][3]
@ -59,88 +56,88 @@ below:
## Objectives
The scope of this proposal is **bounded**, but has the potential for broader
reuse in the future. The reader should be mindful of the explicitly stated
The scope of this proposal is **bounded**, but has the potential for broader
reuse in the future. The reader should be mindful of the explicitly stated
[Non-Goals](#non-goals) that are listed below.
### Goals
- Provide a basic framework for recording annotated *Pods* to the filesystem.
- Ensure that a restart of the kubelet checks for existence of these files
- Provide a basic framework for recording annotated *Pods* to the filesystem.
- Ensure that a restart of the kubelet checks for existence of these files
and loads them on startup.
### Non-Goals
- This is not a generic checkpointing mechanism for arbitrary resources.
- This is not a generic checkpointing mechanism for arbitrary resources.
(e.g. Secrets) Such changes require wider discussions.
- This will not checkpoint internal kubelet state.
- This proposal does not cover self hosted kubelet(s). It is beyond the
- This will not checkpoint internal kubelet state.
- This proposal does not cover self hosted kubelet(s). It is beyond the
scope of this proposal, and comes with it's own unique set of challenges.
## Proposal
The enablement of this feature is gated by a single command line flag that
is passed to the kubelet on startup, ```--bootstrap-checkpoint-path``` ,
and will be denoted that it is ```[Alpha]```.
The enablement of this feature is gated by a single command line flag that
is passed to the kubelet on startup, ```--bootstrap-checkpoint-path``` ,
and will be denoted that it is ```[Alpha]```.
### User Stories
#### Pod Submission to Running
- On submission of a Pod, via kubeadm or an operator, an annotation
```node.kubernetes.io/bootstrap-checkpoint=true``` is added to that Pod, which
indicates that it should be checkpointed by the kubelet. When the kubelet
receives a notification from the apiserver that a new pod is to run, it will
inspect the ```--bootstrap-checkpoint-path``` flag to determine if
checkpointing is enabled. Finally, the kubelet will perform an atomic
- On submission of a Pod, via kubeadm or an operator, an annotation
```node.kubernetes.io/bootstrap-checkpoint=true``` is added to that Pod, which
indicates that it should be checkpointed by the kubelet. When the kubelet
receives a notification from the apiserver that a new pod is to run, it will
inspect the ```--bootstrap-checkpoint-path``` flag to determine if
checkpointing is enabled. Finally, the kubelet will perform an atomic
write of a ```Pod_UID.yaml``` file when the afore mentioned annotation exists.
The scope of this annotation is bounded and will not be promoted to a field.
The scope of this annotation is bounded and will not be promoted to a field.
#### Pod Deletion
- On detected deletion of a Pod, the kubelet will remove the associated
checkpoint from the filesystem. Any failure to remove a pod, or file, will
- On detected deletion of a Pod, the kubelet will remove the associated
checkpoint from the filesystem. Any failure to remove a pod, or file, will
result in an error notification in the kubelet logs.
#### Cold Start
- On a cold start, the kubelet will check the value of
```--bootstrap-checkpoint-path```. If the value is specified, it will read in
the contents of the that directory and startup the appropriate Pod. Lastly,
the kubelet will then pull the list of pods from the api-server and rectify
- On a cold start, the kubelet will check the value of
```--bootstrap-checkpoint-path```. If the value is specified, it will read in
the contents of the that directory and startup the appropriate Pod. Lastly,
the kubelet will then pull the list of pods from the api-server and rectify
what is supposed to be running according to what is bound, and will go through
its normal startup procedure.
### Implementation Constraints
Due to its opt-in behavior, administrators will need to take the same precautions
Due to its opt-in behavior, administrators will need to take the same precautions
necessary in segregating master nodes, when enabling the bootstrap annotation.
Please see [WIP Implementation][4] for more details.
## Graduation Criteria
Graduating this feature is a responsibility of sig-cluster-lifecycle and
sig-node to determine over the course of the 1.10 and 1.11 releases. History
has taught us that initial implementations often have a tendency overlook use
cases and require refinement. It is the goal of this proposal to have an
initial alpha implementation of bootstrap checkpoining in the 1.9 cycle,
and further refinement will occur after we have validated it across several
deployments.
Graduating this feature is a responsibility of sig-cluster-lifecycle and
sig-node to determine over the course of the 1.10 and 1.11 releases. History
has taught us that initial implementations often have a tendency overlook use
cases and require refinement. It is the goal of this proposal to have an
initial alpha implementation of bootstrap checkpoining in the 1.9 cycle,
and further refinement will occur after we have validated it across several
deployments.
## Testing
Testing of this feature will occur in three parts.
- Unit testing of standard code behavior
## Testing
Testing of this feature will occur in three parts.
- Unit testing of standard code behavior
- Simple node-e2e test to ensure restart recovery
- (TODO) E2E test w/kubeadm self hosted master restart recovery of an apiserver.
- (TODO) E2E test w/kubeadm self hosted master restart recovery of an apiserver.
## Implementation History
- 20171020 - 1.9 draft proposal
- 20171101 - 1.9 accepted proposal
- 20171020 - 1.9 draft proposal
- 20171101 - 1.9 accepted proposal
- 20171114 - 1.9 alpha implementation code complete
## Unresolved Questions
* None at this time.
* None at this time.
[0]: /contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md
[1]: https://github.com/kubernetes/community/pull/825
[2]: https://docs.google.com/document/d/1hhrCa_nv0Sg4O_zJYOnelE8a5ClieyewEsQM6c7-5-o/edit?ts=5988fba8#
[3]: https://docs.google.com/document/d/1qmK0Iq4fqxnd8COBFZHpip27fT-qSPkOgy1x2QqjYaQ/edit?ts=599b797c#
[4]: https://github.com/kubernetes/kubernetes/pull/50984
[4]: https://github.com/kubernetes/kubernetes/pull/50984