kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add read repair approach

This commit is contained in:
Chao Xu 2017-05-16 14:57:04 -07:00
parent 07b4aa81d8
commit 679c2c7d9d
1 changed files with 41 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
contributors/design-proposals/dynamic-admission-control-configuration.md
View File

@ -2,15 +2,16 @@
## Background ## Background
[#132](https://github.com/kubernetes/community/pull/132) proposed making The extensible admission control
admission control extensible. In the proposal, the `initializer admission [proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md)
controller` and the `generic webhook admission controller` are the two proposed making admission control extensible. In the proposal, the `initializer
admission controller` and the `generic webhook admission controller` are the two
controllers that set default initializers and external admission hooks for controllers that set default initializers and external admission hooks for
resources newly created. These two admission controllers are in the same binary resources newly created. These two admission controllers are in the same binary
as the apiserver. This [section](https://github.com/smarterclayton/community/blob/be132e88f7597ab3927b788a3de6d5ab6de673d2/contributors/design-proposals/admission_control_extension.md#dynamic-configuration) as the apiserver. This
of #132 gave a preliminary design of the dynamic configuration of the list of [section](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md#dynamic-configuration)
the default admission controls. This document hashes out the implementation gave a preliminary design of the dynamic configuration of the list of the
details. default admission controls. This document hashes out the implementation details.
## Goals ## Goals
@ -26,10 +27,12 @@ details.
## Specification ## Specification
We assume initializers could be "fail open". We need to update #132 if this is We assume initializers could be "fail open". We need to update the extensible
accepted. admission control
[proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md)
if this is accepted.
The schema is copied from The schema is copied from
[#132](https://github.com/kubernetes/community/pull/132) with a few [#132](https://github.com/kubernetes/community/pull/132) with a few
modifications. modifications.
@ -168,14 +171,37 @@ This will block the entire cluster. We have a few options:
## Handling fail-open initializers ## Handling fail-open initializers
#132 assumed initializers always failed closed. It is dangerous since crashed The original [proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md) assumed initializers always failed closed. It is dangerous since crashed
initializers can block the whole cluster. We propose to allow initializers to initializers can block the whole cluster. We propose to allow initializers to
fail open, and in 1.7, let all initializers fail open. fail open, and in 1.7, let all initializers fail open. We considered the two
approaches to implement the fail open initializers.
#### 1. apiserver + read repair
In the initializer prototype
[PR](https://github.com/kubernetes/kubernetes/pull/36721), the apiserver that
handles the CREATE request
[watches](https://github.com/kubernetes/kubernetes/pull/36721/files#diff-2c081fad5c858e67c96f75adac185093R349)
the uninitialized object. We can add a timer there and let the apiserver remove
the timed out initializer.
If the apiserver crashes, then we fall back to a `read repair` mechanism. When
handling a GET request, the apiserver checks the objectMeta.CreationTimestamp of
the object, if a global intializer timeout (e.g., 10 mins) has reached, the
apiserver removes the first initializer in the object.
In HA setup, apiserver needs to take the clock drift into account as well.
Note that the fallback is only invoked when the initializer and the apiserver
crashes, so it is rare.
#### 2. use a controller
A `fail-open initializers controller` will remove the timed out fail-open A `fail-open initializers controller` will remove the timed out fail-open
initializers from objects' initializers list. Every 30s, the controller initializers from objects' initializers list. The controller uses shared
informers to track uninitialized objects. Every 30s, the controller
* lists uninitialized objects * makes a snapshot of the uninitialized objects in the informers.
* indexes the objects by the name of the first initialilzer in the objectMeta.Initializers * indexes the objects by the name of the first initialilzer in the objectMeta.Initializers
* compares with the snapshot 30s ago, finds objects whose first initializers haven't changed * compares with the snapshot 30s ago, finds objects whose first initializers haven't changed
* does a consistent read of AdmissionControllerConfiguration, finds which initializers are fail-open * does a consistent read of AdmissionControllerConfiguration, finds which initializers are fail-open
@ -183,7 +209,7 @@ initializers from objects' initializers list. Every 30s, the controller
## Future work ## Future work
1. Allow the user to POST the individual initializer/webhook, expressing the 1. allow the user to POST the individual initializer/webhook, expressing the
dependency on other initializers/webhooks, and let a controller assembles the dependency on other initializers/webhooks, and let a controller assembles the
ordered list of initializers/webhooks. ordered list of initializers/webhooks.