Merge pull request #32764 from neolit123/1.24-add-steps-for-reconf

kubeadm: add task page for cluster reconf

content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md

@@ -24,9 +24,12 @@ you can skip the default CoreDNS deployment and deploy your own variant.
 For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases).
 {{< /note >}}
 
-<!-- body -->
+{{< note >}}
+To reconfigure a cluster that has already been created see
+[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
+{{< /note >}}
 
-{{< feature-state for_k8s_version="v1.12" state="stable" >}}
+<!-- body -->
 
 ## Customizing the control plane with flags in `ClusterConfiguration`
 

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

@@ -162,6 +162,9 @@ To customize control plane components, including optional IPv6 assignment to liv
 for control plane components and etcd server, provide extra arguments to each component as documented in
 [custom arguments](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/).
 
+To reconfigure a cluster that has already been created see
+[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
+
 To run `kubeadm init` again, you must first [tear down the cluster](#tear-down).
 
 If you join a node with a different architecture to your cluster, make sure that your deployed DaemonSets

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure.md

@@ -0,0 +1,281 @@
+---
+reviewers:
+- sig-cluster-lifecycle
+title: Reconfiguring a kubeadm cluster
+content_type: task
+weight: 10
+---
+
+<!-- overview -->
+
+kubeadm does not support automated ways of reconfiguring components that
+were deployed on managed nodes. One way of automating this would be
+by using a custom [operator](/docs/concepts/extend-kubernetes/operator/).
+
+To modify the components configuration you must manually edit associated cluster
+objects and files on disk.
+
+This guide shows the correct sequence of steps that need to be performed
+to achieve kubeadm cluster reconfiguration.
+
+## {{% heading "prerequisites" %}}
+
+- You need a cluster that was deployed using kubeadm
+- Have administrator credentials (`/etc/kubernetes/admin.conf`) and network connectivity
+to a running kube-apiserver in the cluster from a host that has kubectl installed
+- Have a text editor installed on all hosts
+
+<!-- steps -->
+
+## Reconfiguring the cluster
+
+kubeadm writes a set of cluster wide component configuration options in
+ConfigMaps and other objects. These objects must be manually edited. The command `kubectl edit`
+can be used for that.
+
+The `kubectl edit` command will open a text editor where you can edit and save the object directly.
+
+You can use the environment variables `KUBECONFIG` and `KUBE_EDITOR` to specify the location of
+the kubectl consumed kubeconfig file and preferred text editor.
+
+For example:
+```
+KUBECONFIG=/etc/kubernetes/admin.conf KUBE_EDITOR=nano kubectl edit <parameters>
+```
+
+{{< note >}}
+Upon saving any changes to these cluster objects, components running on nodes may not be
+automatically updated. The steps below instruct you on how to perform that manually.
+{{< /note >}}
+
+{{< warning >}}
+Component configuration in ConfigMaps is stored as unstructured data (YAML string).
+This means that validation will not be performed upon updating the contents of a ConfigMap.
+You have to be careful to follow the documented API format for a particular
+component configuration and avoid introducing typos and YAML indentation mistakes.
+{{< /warning >}}
+
+### Applying cluster configuration changes
+
+#### Updating the `ClusterConfiguration`
+
+During cluster creation and upgrade, kubeadm writes its
+[`ClusterConfiguration`](/docs/reference/config-api/kubeadm-config.v1beta3/)
+in a ConfigMap called `kubeadm-config` in the `kube-system` namespace.
+
+To change a particular option in the `ClusterConfiguration` you can edit the ConfigMap with this command:
+
+```shell
+kubectl edit cm -n kube-system kubeadm-config
+```
+
+The configuration is located under the `data.ClusterConfiguration` key.
+
+{{< note >}}
+The `ClusterConfiguration` includes a variety of options that affect the configuration of individual
+components such as kube-apiserver, kube-scheduler, kube-controller-manager, CoreDNS, etcd and kube-proxy.
+Changes to the configuration must be reflected on node components manually.
+{{< /note >}}
+
+#### Reflecting `ClusterConfiguration` changes on control plane nodes
+
+kubeadm manages the control plane components as static Pod manifests located in
+the directory `/etc/kubernetes/manifests`.
+Any changes to the `ClusterConfiguration` under the `apiServer`, `controllerManager`, `scheduler` or `etcd`
+keys must be reflected in the associated files in the manifests directory on a control plane node.
+
+Such changes may include:
+- `extraArgs` - requires updating the list of flags passed to a component container
+- `extraMounts` - requires updated the volume mounts for a component container
+- `*SANs` - requires writing new certificates with updated Subject Alternative Names.
+
+Before proceeding with these changes, make sure you have backed up the directory `/etc/kubernetes/`.
+
+To write new certificates you can use:
+```shell
+kubeadm init phase certs <component-name> --config <config-file>
+```
+
+To write new manifest files in `/etc/kubernetes/manifests` you can use:
+
+```shell
+kubeadm init phase control-plane <component-name> --config <config-file>
+```
+
+The `<config-file>` contents must match the updated `ClusterConfiguration`.
+The `<component-name>` value must be the name of the component.
+
+{{< note >}}
+Updating a file in `/etc/kubernetes/manifests` will tell the kubelet to restart the static Pod for the corresponding component.
+Try doing these changes one node at a time to leave the cluster without downtime.
+{{< /note >}}
+
+### Applying kubelet configuration changes
+
+#### Updating the `KubeletConfiguration`
+
+During cluster creation and upgrade, kubeadm writes its
+[`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/)
+in a ConfigMap called `kubelet-config` in the `kube-system` namespace.
+
+You can edit the ConfigMap with this command:
+
+```shell
+kubectl edit cm -n kube-system kubelet-config
+```
+
+The configuration is located under the `data.kubelet` key.
+
+#### Reflecting the kubelet changes
+
+To reflect the change on kubeadm nodes you must do the following:
+- Log in to a kubeadm node
+- Run `kubeadm upgrade node phase kubelet-config` to download the latest `kubelet-config`
+ConfigMap contents into the local file `/var/lib/kubelet/config.conf`
+- Edit the file `/var/lib/kubelet/kubeadm-flags.env` to apply additional configuration with
+flags
+- Restart the kubelet service with `systemctl restart kubelet`
+
+{{< note >}}
+Do these changes one node at a time to allow workloads to be rescheduled properly.
+{{< /note >}}
+
+{{< note >}}
+During `kubeadm upgrade`, kubeadm downloads the `KubeletConfiguration` from the
+`kubelet-config` ConfigMap and overwrite the contents of `/var/lib/kubelet/config.conf`.
+This means that node local configuration must be applied either by flags in
+`/var/lib/kubelet/kubeadm-flags.env` or by manually updating the contents of
+`/var/lib/kubelet/config.conf` after `kubeadm upgrade`, and then restarting the kubelet.
+{{< /note >}}
+
+### Applying kube-proxy configuration changes
+
+#### Updating the `KubeProxyConfiguration`
+
+During cluster creation and upgrade, kubeadm writes its
+[`KubeProxyConfiguration`](/docs/reference/config-api/kube-proxy-config.v1alpha1/)
+in a ConfigMap in the `kube-system` namespace called `kube-proxy`.
+
+This ConfigMap is used by the `kube-proxy` DaemonSet in the `kube-system` namespace.
+
+To change a particular option in the `KubeProxyConfiguration`, you can edit the ConfigMap with this command:
+
+```shell
+kubectl edit cm -n kube-system kube-proxy
+```
+
+The configuration is located under the `data.config.conf` key.
+
+#### Reflecting the kube-proxy changes
+
+Once the `kube-proxy` ConfigMap is updated, you can restart all kube-proxy Pods:
+
+Obtain the Pod names:
+
+```shell
+kubectl get po -n kube-system | grep kube-proxy
+```
+
+Delete a Pod with:
+
+```shell
+kubectl delete po -n kube-system <pod-name>
+```
+
+New Pods that use the updated ConfigMap will be created.
+
+{{< note >}}
+Because kubeadm deploys kube-proxy as a DaemonSet, node specific configuration is unsupported.
+{{< /note >}}
+
+### Applying CoreDNS configuration changes
+
+#### Updating the CoreDNS Deployment and Service
+
+kubeadm deploys CoreDNS as a Deployment called `coredns` and with a Service `kube-dns`,
+both in the `kube-system` namespace.
+
+To update any of the CoreDNS settings, you can edit the Deployment and
+Service objects:
+
+```shell
+kubectl edit deployment -n kube-system coredns
+kubectl edit service -n kube-system kube-dns
+```
+
+#### Reflecting the CoreDNS changes
+
+Once the CoreDNS changes are applied you can delete the CoreDNS Pods:
+
+Obtain the Pod names:
+
+```shell
+kubectl get po -n kube-system | grep coredns
+```
+
+Delete a Pod with:
+
+```shell
+kubectl delete po -n kube-system <pod-name>
+```
+
+New Pods with the updated CoreDNS configuration will be created.
+
+{{< note >}}
+kubeadm does not allow CoreDNS configuration during cluster creation and upgrade.
+This means that if you execute `kubeadm upgrade apply`, your changes to the CoreDNS
+objects will be lost and must be reapplied.
+{{< /note >}}
+
+## Persisting the reconfiguration
+
+During the execution of `kubeadm upgrade` on a managed node, kubeadm might overwrite configuration
+that was applied after the cluster was created (reconfiguration).
+
+### Persisting Node object reconfiguration
+
+kubeadm writes Labels, Taints, CRI socket and other information on the Node object for a particular
+Kubernetes node. To change any of the contents of this Node object you can use:
+
+```shell
+kubectl edit no <node-name>
+```
+
+During `kubeadm upgrade` the contents of such a Node might get overwritten.
+If you would like to persist your modifications to the Node object after upgrade,
+you can prepare a [kubectl patch](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/)
+and apply it to the Node object:
+
+```shell
+kubectl patch no <node-name> --patch-file <patch-file>
+```
+
+#### Persisting control plane component reconfiguration
+
+The main source of control plane configuration is the `ClusterConfiguration`
+object stored in the cluster. To extend the static Pod manifests configuration,
+[patches](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#patches) can be used.
+
+These patch files must remain as files on the control plane nodes to ensure that
+they can be used by the `kubeadm upgrade ... --patches <directory>`.
+
+If reconfiguration is done to the `ClusterConfiguration` and static Pod manifests on disk,
+the set of node specific patches must be updated accordingly.
+
+#### Persisting kubelet reconfiguration
+
+Any changes to the `KubeletConfiguration` stored in `/var/lib/kubelet/config.conf` will be overwritten on
+`kubeadm upgrade` by downloading the contents of the cluster wide `kubelet-config` ConfigMap.
+To persist kubelet node specific configuration either the file `/var/lib/kubelet/config.conf`
+has to be updated manually post-upgrade or the file `/var/lib/kubelet/kubeadm-flags.env` can include flags.
+The kubelet flags override the associated `KubeletConfiguration` options, but note that
+some of the flags are deprecated.
+
+A kubelet restart will be required after changing `/var/lib/kubelet/config.conf` or
+`/var/lib/kubelet/kubeadm-flags.env`.
+
+{{% heading "whatsnext" %}}
+
+- [Upgrading kubeadm clusters](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade)
+- [Customizing components with the kubeadm API](/docs/setup/production-environment/tools/kubeadm/control-plane-flags)
+- [Certificate management with kubeadm](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs)

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md

@@ -43,7 +43,12 @@ first drain the node (or nodes) that you are upgrading. In the case of control p
 they could be running CoreDNS Pods or other critical workloads. For more information see
 [Draining nodes](/docs/tasks/administer-cluster/safely-drain-node/).
 - All containers are restarted after upgrade, because the container spec hash value is changed.
-- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded, you can execute `systemctl status kubelet`  or view the service logs with `journalctl -xeu kubelet`.
+- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded,
+you can execute `systemctl status kubelet`  or view the service logs with `journalctl -xeu kubelet`.
+- Usage of the `--config` flag of `kubeadm upgrade` with
+[kubeadm configuration API types](/docs/reference/config-api/kubeadm-config.v1beta3)
+with the purpose of reconfiguring the cluster is not recommended and can have unexpected results. Follow the steps in
+[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure) instead.
 
 <!-- steps -->