Merge pull request #527 from janetkuo/ds-update-history
Proposal for DaemonSet history and rollback
This commit is contained in:
Janet Kuo
GitHub
commit
951fd045ca
|
|
@ -67,7 +67,7 @@ type DaemonSetUpdateStrategy struct {
|
|||
// RollingUpdate.
|
||||
//---
|
||||
// TODO: Update this to follow our convention for oneOf, whatever we decide it
|
||||
// to be. Same as DeploymentStrategy.RollingUpdate.
|
||||
// to be. Same as Deployment `strategy.rollingUpdate`.
|
||||
// See https://github.com/kubernetes/kubernetes/issues/35345
|
||||
// +optional
|
||||
RollingUpdate *RollingUpdateDaemonSet
|
||||
|
|
@ -118,10 +118,16 @@ type DaemonSetSpec struct {
|
|||
// +optional
|
||||
MinReadySeconds int32 `json:"minReadySeconds,omitempty"`
|
||||
|
||||
// DEPRECATED.
|
||||
// A sequence number representing a specific generation of the template.
|
||||
// Populated by the system. Can be set at creation time. Read-only otherwise.
|
||||
// +optional
|
||||
TemplateGeneration int64 `json:"templateGeneration,omitempty"`
|
||||
|
||||
// The number of old history to retain to allow rollback.
|
||||
// This is a pointer to distinguish between explicit zero and not specified.
|
||||
// Defaults to 10.
|
||||
RevisionHistoryLimit *int32 `json:"revisionHistoryLimit,omitempty"`
|
||||
}
|
||||
|
||||
// DaemonSetStatus represents the current status of a daemon set.
|
||||
|
|
@ -146,13 +152,25 @@ type DaemonSetStatus struct {
|
|||
// (ready for at least minReadySeconds)
|
||||
// +optional
|
||||
NumberUnavailable int32 `json:"numberUnavailable"`
|
||||
|
||||
// Count of hash collisions for the DaemonSet. The DaemonSet controller
|
||||
// uses this field as a collision avoidance mechanism when it needs to
|
||||
// create the name for the newest ControllerRevision.
|
||||
// +optional
|
||||
CollisionCount *int64 `json:"collisionCount,omitempty"`
|
||||
}
|
||||
|
||||
const (
|
||||
// DEPRECATED: DefaultDeploymentUniqueLabelKey is used instead.
|
||||
// DaemonSetTemplateGenerationKey is the key of the labels that is added
|
||||
// to daemon set pods to distinguish between old and new pod templates
|
||||
// to daemon set pods to distinguish between old and new pod
|
||||
// during DaemonSet template update.
|
||||
DaemonSetTemplateGenerationKey string = "pod-template-generation"
|
||||
|
||||
// DefaultDaemonSetUniqueLabelKey is the default label key that is added
|
||||
// to existing DaemonSet pods to distinguish between old and new
|
||||
// DaemonSet pods during DaemonSet template updates.
|
||||
DefaultDaemonSetUniqueLabelKey string = "daemonset-controller-hash"
|
||||
)
|
||||
```
|
||||
|
||||
|
|
@ -163,27 +181,61 @@ const (
|
|||
The DaemonSet Controller will make DaemonSet updates happen. It will watch
|
||||
DaemonSets on the apiserver.
|
||||
|
||||
DaemonSet controller manages [`ControllerRevisions`](controller_history.md) for
|
||||
DaemonSet revision introspection and rollback. It's referred to as "history"
|
||||
throughout the rest of this proposal.
|
||||
|
||||
For each pending DaemonSet updates, it will:
|
||||
|
||||
1. Find all pods whose labels are matched by DaemonSet `.spec.selector`.
|
||||
- If `OwnerReference` is implemented for DaemonSets, filter out pods that
|
||||
aren't controlled by this DaemonSet too.
|
||||
1. Reconstruct DaemonSet history:
|
||||
- List existing DaemonSet history controlled by this DaemonSet
|
||||
- Find the history of DaemonSet's current target state, and create one if
|
||||
not found:
|
||||
- The `.name` of this history will be unique, generated from pod template
|
||||
hash with hash collision resolution. If history creation failed:
|
||||
- If it's because of name collision:
|
||||
- Compare history with DaemonSet current target state:
|
||||
- If they're the same, we've already created the history
|
||||
- Otherwise, bump DaemonSet `.status.collisionCount` by 1, exit and
|
||||
retry in the next sync loop
|
||||
- Otherwise, exit and retry again in the next sync loop.
|
||||
- The history will be labeled with `DefaultDaemonSetUniqueLabelKey`.
|
||||
- DaemonSet controller will add a ControllerRef in the history
|
||||
`.ownerReferences`.
|
||||
- Current history should have the largest `.revision` number amonst all
|
||||
existing history. Update `.revision` if it's not (e.g. after a rollback.)
|
||||
- If more than one current history is found, remove duplicates and relabel
|
||||
their pods' `DefaultDaemonSetUniqueLabelKey`.
|
||||
1. Sync nodes:
|
||||
- Find all nodes that should run these pods created by this DaemonSet.
|
||||
- Create daemon pods on nodes when they should have those pods running but not
|
||||
yet. Otherwise, delete running daemon pods that shouldn't be running on nodes.
|
||||
- Label new pods with current `.spec.templateGeneration` and
|
||||
`DefaultDaemonSetUniqueLabelKey` value of current history when creating them.
|
||||
1. Check `DaemonSetUpdateStrategy`:
|
||||
- If `OnDelete`: do nothing
|
||||
- If `RollingUpdate`:
|
||||
- Find all daemon pods that belong to this DaemonSet using label selectors.
|
||||
- Pods with "pod-template-generation" value equal to the DaemonSet's
|
||||
`.spec.templateGeneration` are new pods, otherwise they're old pods.
|
||||
- Note that pods without "pod-template-generation" labels (e.g. DaemonSet
|
||||
pods created before RollingUpdate strategy is implemented) will be
|
||||
seen as old pods.
|
||||
- For all pods owned by this DaemonSet:
|
||||
- If its `pod-template-generation` label value equals to DaemonSet's
|
||||
`.spec.templateGeneration`, it's a new pod (don't compare
|
||||
`DefaultDaemonSetUniqueLabelKey`, for backward compatibility).
|
||||
- Add `DefaultDaemonSetUniqueLabelKey` label to the new pod based on current
|
||||
history, if the pod doesn't have this label set yet.
|
||||
- Otherwise, if the value doesn't match, or the pod doesn't have a
|
||||
`pod-template-generation` label, check its `DefaultDaemonSetUniqueLabelKey` label:
|
||||
- If the value matches any of the history's `DefaultDaemonSetUniqueLabelKey` label,
|
||||
it's a pod generated from that history.
|
||||
- If that history matches the current target state of the DaemonSet,
|
||||
it's a new pod.
|
||||
- Otherwise, it's an old pod.
|
||||
- Otherwise, if the pod doesn't have a `DefaultDaemonSetUniqueLabelKey` label, or no
|
||||
matching history is found, it's an old pod.
|
||||
- If there are old pods found, compare `MaxUnavailable` with DaemonSet
|
||||
`.status.numberUnavailable` to see how many old daemon pods can be
|
||||
killed. Then, kill those pods in the order that unhealthy pods (failed,
|
||||
pending, not ready) are killed first.
|
||||
1. Find all nodes that should run these pods created by this DaemonSet.
|
||||
1. Create daemon pods on nodes when they should have those pods running but not
|
||||
yet. Otherwise, delete running daemon pods that shouldn't be running on nodes.
|
||||
1. Clean up old history based on `.spec.revisionHistoryLimit`
|
||||
- Always keep live history and current history
|
||||
1. Cleanup, update DaemonSet status
|
||||
- `.status.numberAvailable` = the total number of DaemonSet pods that have
|
||||
become `Ready` for `MinReadySeconds`
|
||||
|
|
@ -198,6 +250,8 @@ In DaemonSet strategy (pkg/registry/extensions/daemonset/strategy.go#PrepareForU
|
|||
increase DaemonSet's `.spec.templateGeneration` by 1 if any changes is made to
|
||||
DaemonSet's `.spec.template`.
|
||||
|
||||
This was originally implmeneted in 1.6, and kept in 1.7 for backward compatibility.
|
||||
|
||||
### kubectl
|
||||
|
||||
#### kubectl rollout
|
||||
|
|
@ -206,6 +260,9 @@ Users can use `kubectl rollout` to monitor DaemonSet updates:
|
|||
|
||||
- `kubectl rollout status daemonset/<DaemonSet-Name>`: to see the DaemonSet
|
||||
upgrade status
|
||||
- `kubectl rollout history daemonset/<DaemonSet-Name>`: to view the history of
|
||||
DaemonSet updates.
|
||||
- `kubectl rollout undo daemonset/<DaemonSet-Name>`: to rollback a DaemonSet
|
||||
|
||||
## Updating DaemonSets mid-way
|
||||
|
||||
|
|
@ -216,7 +273,7 @@ one will begin rolling out.
|
|||
|
||||
## Deleting DaemonSets
|
||||
|
||||
Deleting a DaemonSet (with cascading) will delete all its pods.
|
||||
Deleting a DaemonSet (with cascading) will delete all its pods and history.
|
||||
|
||||
|
||||
## DaemonSet Strategies
|
||||
|
|
@ -281,57 +338,13 @@ In the future, we may:
|
|||
- Support pausing DaemonSet rolling update
|
||||
- Support auto-rollback DaemonSets
|
||||
|
||||
### DaemonSet History
|
||||
### API
|
||||
|
||||
In the future, we will support DaemonSet history, so that users can view
|
||||
previous revisions and roll back when necessary.
|
||||
Implement a subresource for DaemonSet history (`daemonsets/foo/history`) that
|
||||
summarizes the information in the history.
|
||||
|
||||
One possible approach is to create a new resource called `DaemonSetRevision` to
|
||||
store DaemonSet history.
|
||||
|
||||
Another way to implement DaemonSet history is through creating `PodTemplates` as
|
||||
snapshots of DaemonSet templates, and then create them in DaemonSet controller:
|
||||
|
||||
- Find existing PodTemplates whose labels are matched by DaemonSet
|
||||
`.spec.selector`.
|
||||
- Sort those PodTemplates by creation timestamp and only retain at most
|
||||
`.spec.revisionHistoryLimit` latest PodTemplates (remove the rest).
|
||||
- Find the PodTemplate whose `.template` is the same as the DaemonSet's
|
||||
`.spec.template`.
|
||||
- If not found, create a new PodTemplate from DaemonSet's
|
||||
`.spec.template`:
|
||||
- The name will be `<DaemonSet-Name>-<template-generation>`
|
||||
- PodTemplate `.metadata.labels` will have a "pod-template-generation"
|
||||
label, value be the same as DaemonSet's `.spec.templateGeneration`.
|
||||
- PodTemplate will have revision information to avoid triggering
|
||||
unnecessary restarts on rollback, since we only roll forward and only
|
||||
increase templateGeneration.
|
||||
- PodTemplate `.metadata.annotations` will be copied from DaemonSet
|
||||
`.metadata.annotations`.
|
||||
- If the PodTemplate is found, sync its "pod-template-generation" and
|
||||
revision information with current DaemonSet.
|
||||
- DaemonSet creates pods with "pod-template-generation" label.
|
||||
|
||||
PodTemplate may need to be made an admin-only or read only resource if it's used
|
||||
to store DaemonSet history.
|
||||
|
||||
#### History Clean Up Policy
|
||||
|
||||
We also need to specify a clean up policy for the number of history to keep in
|
||||
the system, and keep only limited number of history by default (for example, 3).
|
||||
|
||||
#### kubectl
|
||||
|
||||
`kubectl` should also support DaemonSet history:
|
||||
|
||||
- `kubectl rollout undo daemonset/<DaemonSet-Name>` to roll back
|
||||
- `kubectl rollout history daemonset/<DaemonSet-Name>`: to view the history of
|
||||
DaemonSet updates
|
||||
|
||||
#### API
|
||||
|
||||
Implement a subresource for DaemonSet history (e.g. `daemonsets/foo/history`)
|
||||
that summarizes the information in the history.
|
||||
Implement a subresource for DaemonSet rollback (`daemonsets/foo/rollback`) that
|
||||
triggers a DaemonSet rollback.
|
||||
|
||||
### Tests
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue