Merge pull request #527 from janetkuo/ds-update-history

Proposal for DaemonSet history and rollback

contributors/design-proposals/daemonset-update.md

@@ -67,7 +67,7 @@ type DaemonSetUpdateStrategy struct {
 	// RollingUpdate.
 	//---
 	// TODO: Update this to follow our convention for oneOf, whatever we decide it
-	// to be. Same as DeploymentStrategy.RollingUpdate.
+	// to be. Same as Deployment `strategy.rollingUpdate`.
 	// See https://github.com/kubernetes/kubernetes/issues/35345
 	// +optional
 	RollingUpdate *RollingUpdateDaemonSet
@@ -118,10 +118,16 @@ type DaemonSetSpec struct {
 	// +optional
 	MinReadySeconds int32 `json:"minReadySeconds,omitempty"`
 
+	// DEPRECATED.
 	// A sequence number representing a specific generation of the template.
 	// Populated by the system. Can be set at creation time. Read-only otherwise. 
 	// +optional
 	TemplateGeneration int64 `json:"templateGeneration,omitempty"`
+
+	// The number of old history to retain to allow rollback.
+	// This is a pointer to distinguish between explicit zero and not specified.
+	// Defaults to 10.
+	RevisionHistoryLimit *int32 `json:"revisionHistoryLimit,omitempty"`
 }
 
 // DaemonSetStatus represents the current status of a daemon set.
@@ -146,13 +152,25 @@ type DaemonSetStatus struct {
 	// (ready for at least minReadySeconds)
 	// +optional
 	NumberUnavailable int32 `json:"numberUnavailable"`
+
+	// Count of hash collisions for the DaemonSet. The DaemonSet controller
+	// uses this field as a collision avoidance mechanism when it needs to
+	// create the name for the newest ControllerRevision.
+	// +optional
+	CollisionCount *int64 `json:"collisionCount,omitempty"`
 }
 
 const (
+	// DEPRECATED: DefaultDeploymentUniqueLabelKey is used instead.
 	// DaemonSetTemplateGenerationKey is the key of the labels that is added
-	// to daemon set pods to distinguish between old and new pod templates
+	// to daemon set pods to distinguish between old and new pod
 	// during DaemonSet template update.
 	DaemonSetTemplateGenerationKey string = "pod-template-generation"
+
+	// DefaultDaemonSetUniqueLabelKey is the default label key that is added
+	// to existing DaemonSet pods to distinguish between old and new
+	// DaemonSet pods during DaemonSet template updates.
+	DefaultDaemonSetUniqueLabelKey string = "daemonset-controller-hash"
 )
 ```
 
@@ -163,27 +181,61 @@ const (
 The DaemonSet Controller will make DaemonSet updates happen. It will watch
 DaemonSets on the apiserver. 
 
+DaemonSet controller manages [`ControllerRevisions`](controller_history.md) for
+DaemonSet revision introspection and rollback. It's referred to as "history"
+throughout the rest of this proposal.
+
 For each pending DaemonSet updates, it will:
 
-1. Find all pods whose labels are matched by DaemonSet `.spec.selector`. 
+1. Reconstruct DaemonSet history:
-   - If `OwnerReference` is implemented for DaemonSets, filter out pods that
+   - List existing DaemonSet history controlled by this DaemonSet
-     aren't controlled by this DaemonSet too.
+   - Find the history of DaemonSet's current target state, and create one if
+     not found:
+     - The `.name` of this history will be unique, generated from pod template
+       hash with hash collision resolution. If history creation failed:
+       - If it's because of name collision:
+         - Compare history with DaemonSet current target state:
+           - If they're the same, we've already created the history
+           - Otherwise, bump DaemonSet `.status.collisionCount` by 1, exit and
+             retry in the next sync loop
+       - Otherwise, exit and retry again in the next sync loop.
+     - The history will be labeled with `DefaultDaemonSetUniqueLabelKey`.
+     - DaemonSet controller will add a ControllerRef in the history
+       `.ownerReferences`.
+   - Current history should have the largest `.revision` number amonst all
+     existing history. Update `.revision` if it's not (e.g. after a rollback.)
+   - If more than one current history is found, remove duplicates and relabel
+     their pods' `DefaultDaemonSetUniqueLabelKey`.
+1. Sync nodes:
+   - Find all nodes that should run these pods created by this DaemonSet.
+   - Create daemon pods on nodes when they should have those pods running but not
+     yet. Otherwise, delete running daemon pods that shouldn't be running on nodes.
+   - Label new pods with current `.spec.templateGeneration` and
+     `DefaultDaemonSetUniqueLabelKey` value of current history when creating them.
 1. Check `DaemonSetUpdateStrategy`:
    - If `OnDelete`: do nothing
    - If `RollingUpdate`:
-     - Find all daemon pods that belong to this DaemonSet using label selectors.
+     - For all pods owned by this DaemonSet:
-       - Pods with "pod-template-generation" value equal to the DaemonSet's
+       - If its `pod-template-generation` label value equals to DaemonSet's 
-         `.spec.templateGeneration` are new pods, otherwise they're old pods.
+         `.spec.templateGeneration`, it's a new pod (don't compare
-       - Note that pods without "pod-template-generation" labels (e.g. DaemonSet
+         `DefaultDaemonSetUniqueLabelKey`, for backward compatibility).
-         pods created before RollingUpdate strategy is implemented) will be
+         - Add `DefaultDaemonSetUniqueLabelKey` label to the new pod based on current
-         seen as old pods.
+           history, if the pod doesn't have this label set yet. 
+       - Otherwise, if the value doesn't match, or the pod doesn't have a
+         `pod-template-generation` label, check its `DefaultDaemonSetUniqueLabelKey` label:
+         - If the value matches any of the history's `DefaultDaemonSetUniqueLabelKey` label,
+           it's a pod generated from that history.
+           - If that history matches the current target state of the DaemonSet,
+             it's a new pod.
+           - Otherwise, it's an old pod. 
+         - Otherwise, if the pod doesn't have a `DefaultDaemonSetUniqueLabelKey` label, or no
+           matching history is found, it's an old pod. 
      - If there are old pods found, compare `MaxUnavailable` with DaemonSet
        `.status.numberUnavailable` to see how many old daemon pods can be
        killed. Then, kill those pods in the order that unhealthy pods (failed,
        pending, not ready) are killed first.
-1. Find all nodes that should run these pods created by this DaemonSet.
+1. Clean up old history based on `.spec.revisionHistoryLimit`
-1. Create daemon pods on nodes when they should have those pods running but not
+   - Always keep live history and current history
-   yet. Otherwise, delete running daemon pods that shouldn't be running on nodes.
 1. Cleanup, update DaemonSet status  
    - `.status.numberAvailable` = the total number of DaemonSet pods that have
      become `Ready` for `MinReadySeconds`
@@ -198,6 +250,8 @@ In DaemonSet strategy (pkg/registry/extensions/daemonset/strategy.go#PrepareForU
 increase DaemonSet's `.spec.templateGeneration` by 1 if any changes is made to
 DaemonSet's `.spec.template`.
 
+This was originally implmeneted in 1.6, and kept in 1.7 for backward compatibility.
+
 ### kubectl 
 
 #### kubectl rollout 
@@ -206,6 +260,9 @@ Users can use `kubectl rollout` to monitor DaemonSet updates:
 
 - `kubectl rollout status daemonset/<DaemonSet-Name>`: to see the DaemonSet
   upgrade status 
+- `kubectl rollout history daemonset/<DaemonSet-Name>`: to view the history of
+  DaemonSet updates.
+- `kubectl rollout undo daemonset/<DaemonSet-Name>`: to rollback a DaemonSet
 
 ## Updating DaemonSets mid-way
 
@@ -216,7 +273,7 @@ one will begin rolling out.
 
 ## Deleting DaemonSets
 
-Deleting a DaemonSet (with cascading) will delete all its pods. 
+Deleting a DaemonSet (with cascading) will delete all its pods and history.
 
 
 ## DaemonSet Strategies
@@ -281,57 +338,13 @@ In the future, we may:
 - Support pausing DaemonSet rolling update
 - Support auto-rollback DaemonSets
 
-### DaemonSet History 
+### API
 
-In the future, we will support DaemonSet history, so that users can view
+Implement a subresource for DaemonSet history (`daemonsets/foo/history`) that
-previous revisions and roll back when necessary. 
+summarizes the information in the history.
 
-One possible approach is to create a new resource called `DaemonSetRevision` to
+Implement a subresource for DaemonSet rollback (`daemonsets/foo/rollback`) that
-store DaemonSet history.
+triggers a DaemonSet rollback.
-
-Another way to implement DaemonSet history is through creating `PodTemplates` as
-snapshots of DaemonSet templates, and then create them in DaemonSet controller:
-
-- Find existing PodTemplates whose labels are matched by DaemonSet
-  `.spec.selector`.
-  - Sort those PodTemplates by creation timestamp and only retain at most
-   `.spec.revisionHistoryLimit` latest PodTemplates (remove the rest).
-  - Find the PodTemplate whose `.template` is the same as the DaemonSet's
-    `.spec.template`.
-    - If not found, create a new PodTemplate from DaemonSet's
-      `.spec.template`:
-      - The name will be `<DaemonSet-Name>-<template-generation>`
-      - PodTemplate `.metadata.labels` will have a "pod-template-generation"
-        label, value be the same as DaemonSet's `.spec.templateGeneration`.
-      - PodTemplate will have revision information to avoid triggering
-        unnecessary restarts on rollback, since we only roll forward and only
-        increase templateGeneration.
-      - PodTemplate `.metadata.annotations` will be copied from DaemonSet 
-        `.metadata.annotations`.
-    - If the PodTemplate is found, sync its "pod-template-generation" and
-      revision information with current DaemonSet. 
-    - DaemonSet creates pods with "pod-template-generation" label. 
-
-PodTemplate may need to be made an admin-only or read only resource if it's used
-to store DaemonSet history. 
-
-#### History Clean Up Policy
-
-We also need to specify a clean up policy for the number of history to keep in
-the system, and keep only limited number of history by default (for example, 3).
-
-#### kubectl 
-
-`kubectl` should also support DaemonSet history:
-
-- `kubectl rollout undo daemonset/<DaemonSet-Name>` to roll back 
-- `kubectl rollout history daemonset/<DaemonSet-Name>`: to view the history of
-  DaemonSet updates
-
-#### API
-
-Implement a subresource for DaemonSet history (e.g. `daemonsets/foo/history`)
-that summarizes the information in the history.
 
 ### Tests