Kubeadm kubelet integration doc (#9652)
* kubeadm: Add a kubelet-kubeadm integration reference doc * kubeadm: edits for the kubelet-kubeadm integration doc * kubeadm: move the kubelet-kubeadm integration doc * Apply edits to kubelet-integration.md * Make kubelet-integration.md use a template Also apply MD formatting fixes. * Copyedit * Address feedback on copyedit * added a couple of typo fixes
This commit is contained in:
Lubomir I. Ivanov
k8s-ci-robot
parent
8f974b4df5
commit
3098e3741e
|
|
@ -0,0 +1,204 @@
|
|||
---
|
||||
reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Configuring each kubelet in your cluster using kubeadm
|
||||
content_template: templates/concept
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{ feature-state for_k8s_version="1.11" state="stable" }}
|
||||
|
||||
The lifecycle of the kubeadm CLI tool is decoupled from the
|
||||
[Kubernetes Node Agent](/docs/reference/command-line-tools-reference/kubelet), which is a daemon that runs
|
||||
on each Kubernetes master or Node. The kubeadm CLI tool is executed by the user when Kubernetes is
|
||||
initialized or upgraded, whereas the kubelet is always running in the background.
|
||||
|
||||
Since the kubelet is a daemon, it needs to be maintained by some kind of a init
|
||||
system or service manager. When the kubelet is installed using DEBs or RPMs,
|
||||
systemd is configured to manage the kubelet. You can use a different service
|
||||
manager instead, but you need to configure it manually.
|
||||
|
||||
Some kublet configuration details need to be the same across all kubelets involved in the cluster, while
|
||||
other configuration aspects need to be set on a per-kubelet basis, to accommodate the different
|
||||
characteristics of a given machine, such as OS, storage, and networking. You can manage the configuration
|
||||
of your kubelets manually, but [kubeadm now provides a `MasterConfig` API type for managing your
|
||||
kubelet configurations centrally](#configure-kubelets-using-kubeadm).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{ toc }}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Kubelet configuration patterns
|
||||
|
||||
The following sections describe patterns to kubelet configuration that are simplified by
|
||||
using kubeadm, rather than managing the kubelet configuration for each Node manually.
|
||||
|
||||
### Propogating cluster-level configuration to each kubelet
|
||||
|
||||
You can provide the kubelet with default values to be used by `kubelet init` and `kubelet join`
|
||||
commands. Interesting examples include using a different CRI runtime or setting the default subnet
|
||||
used by services.
|
||||
|
||||
If you want your services to use the subnet `10.96.0.0/12` as the default for services, you can pass
|
||||
the `--service-cidr` parameter to kubeadm:
|
||||
|
||||
```bash
|
||||
kubeadm init --service-cidr 10.96.0.0/12
|
||||
```
|
||||
|
||||
Virtual IPs for services are now allocated from this subnet. You also need to set the DNS address used
|
||||
by the kubelet, using the `--cluster-dns` flag. This setting needs to be the same for every kubelet
|
||||
on every manager and Node in the cluster. The kubelet provides a versioned, structured API object
|
||||
that can configure most parameters in the kubelet and push out this configuration to each running
|
||||
kubelet in the cluster. This object is called **the kubelet's ComponentConfig**.
|
||||
The ComponentConfig allows the user to specify flags such as the cluster DNS IP addresses expressed as
|
||||
a list of values to a camelCased key, illustrated by the following example:
|
||||
|
||||
```yaml
|
||||
apiVersion: kubelet.config.k8s.io/v1beta1
|
||||
kind: KubeletConfiguration
|
||||
clusterDNS:
|
||||
- 10.96.0.10
|
||||
```
|
||||
|
||||
See the
|
||||
[API reference for the
|
||||
kubelet ComponentConfig](https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig#KubeletConfiguration)
|
||||
for more information.
|
||||
|
||||
### Providing instance-specific configuration details
|
||||
|
||||
Some hosts require specific kubelet configurations, due to differences in hardware, operating system,
|
||||
networking, or other host-specific parameters. The following list provides a few examples.
|
||||
|
||||
- The path to the DNS resolution file, as specified by the `--resolve-conf` kubelet
|
||||
configuration flag, may differ among operating systems, or depending on whether you are using
|
||||
`systemd-resolved`. If this path is wrong, DNS resolution will fail on the Node whose kubelet
|
||||
is configured incorrectly.
|
||||
|
||||
- The Node API object `.metadata.name` is set to the machine's hostname by default,
|
||||
unless you are using a cloud provider. You can use the `--hostname-override` flag to override the
|
||||
default behavior if you need to specify a Node name different from the machine's hostname.
|
||||
|
||||
- Currently, the kubelet cannot automatically detects the cgroup driver used by the CRI runtime,
|
||||
but the value of `--cgroup-driver` must match the cgroup driver used by the CRI runtime to ensure
|
||||
the health of the kubelet.
|
||||
|
||||
- Depending on the CRI runtime your cluster uses, you may need to specify different flags to the kubelet.
|
||||
For instance, when using Docker, you need to specify flags such as `--network-plugin=cni`, but if you
|
||||
are using an external runtime, you need to specify `--container-runtime=remote` and specify the CRI
|
||||
endpoint using the `--container-runtime-path-endpoint=<path>`.
|
||||
|
||||
You can specify these flags by configuring an individual kubelet's configuration in your service manager,
|
||||
such as systemd.
|
||||
|
||||
## Configure kubelets using kubeadm
|
||||
|
||||
{{ feature-state for_k8s_version="1.11" state="stable" }}
|
||||
|
||||
The kubeadm config API type `MasterConfiguration` embeds the kubelet's ComponentConfig under
|
||||
the `.kubeletConfiguration.baseConfig` key. Any user writing a `MasterConfiguration`
|
||||
file can use this configuration key to also set the base-level configuration for all kubelets
|
||||
in the cluster.
|
||||
|
||||
### Workflow when using `kubeadm init`
|
||||
|
||||
When you call `kubeadm init`, the `.kubeletConfiguration.baseConfig` structure is marshalled to disk
|
||||
at `/var/lib/kubelet/config.yaml`, and also uploaded to a ConfigMap in the cluster. The ConfigMap
|
||||
is named `kubelet-config-1.X`, where `.X` is the minor version of the Kubernetes version you are
|
||||
initializing. A kubelet configuration file is also written to `/etc/kubernetes/kubelet.conf` with the
|
||||
baseline cluster-wide configuration for all kubelets in the cluster. This configuration file
|
||||
points to the client certificates that allow the kubelet to communicate with the API server. This
|
||||
addresses the need to
|
||||
[propogate cluster-level configuration to each kubelet](#propagating-cluster-level-configuration-to-each-kubelet).
|
||||
|
||||
To address the second pattern of
|
||||
[providing instance-specific configuration details](#providing-instance-specific-configuration-details),
|
||||
kubeadm writes an environment file to `/var/lib/kubelet/kubeadm-flags.env`, which contains a list of
|
||||
flags to pass to the kubelet when it starts. The flags are presented in the file like this:
|
||||
|
||||
```bash
|
||||
KUBELET_KUBEADM_ARGS="--flag1=value1 --flag2=value2 ..."
|
||||
```
|
||||
|
||||
In addition to the flags used when starting the kubelet, the file also contains dynamic
|
||||
parameters such as the cgroup driver and whether to use a different CRI runtime socket
|
||||
(`--cri-socket`).
|
||||
|
||||
After marshalling these two files to disk, kubeadm attempts to run the following two
|
||||
commands, if you are using systemd:
|
||||
|
||||
```bash
|
||||
systemctl daemon-reload && systemctl restart kubelet
|
||||
```
|
||||
|
||||
If the reload and restart are successful, the normal `kubeadm init` workflow continues.
|
||||
|
||||
### Workflow when using `kubeadm join`
|
||||
|
||||
When you run `kubeadm join`, kubeadm uses the Bootstrap Token credential perform
|
||||
a TLS bootstrap, which fetches the credential needed to download the
|
||||
`kubelet-config-1.X` ConfigMap and writes it to `/var/lib/kubelet/config.yaml`. The dynamic
|
||||
environment file is generated in exactly the same way as `kubeadm init`.
|
||||
|
||||
Next, `kubeadm` runs the following two commands to load the new configuration into the kubelet:
|
||||
|
||||
```bash
|
||||
systemctl daemon-reload && systemctl restart kubelet
|
||||
```
|
||||
|
||||
After the kubelet loads the new configuration, kubeadm writes the
|
||||
`/etc/kubernetes/bootstrap-kubelet.conf` KubeConfig file, which contains a CA certificate and Bootstrap
|
||||
Token. These are used by the kubelet to perform the TLS Bootstrap and obtain a unique
|
||||
credential, which is stored in `/etc/kubernetes/kubelet.conf`. When this file is written, the kubelet
|
||||
has finished performing the TLS Bootstrap.
|
||||
|
||||
## The kubelet drop-in file for systemd
|
||||
|
||||
The configuration file installed by the kubeadm DEB or RPM package is written to
|
||||
`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` and is used by systemd.
|
||||
|
||||
```none
|
||||
[Service]
|
||||
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
|
||||
--kubeconfig=/etc/kubernetes/kubelet.conf"
|
||||
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
|
||||
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating
|
||||
the KUBELET_KUBEADM_ARGS variable dynamically
|
||||
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
|
||||
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
|
||||
#the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
|
||||
# KUBELET_EXTRA_ARGS should be sourced from this file.
|
||||
EnvironmentFile=-/etc/default/kubelet
|
||||
ExecStart=
|
||||
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
|
||||
```
|
||||
|
||||
This file specifies the default locations for all of the files managed by kubeadm for the kubelet.
|
||||
|
||||
- The KubeConfig file to use for the TLS Bootstrap is `/etc/kubernetes/bootstrap-kubelet.conf`,
|
||||
but it is only used if `/etc/kubernetes/kubelet.conf` does not exist.
|
||||
- The KubeConfig file with the unique kubelet identity is `/etc/kubernetes/kubelet.conf`.
|
||||
- The file containing the kubelet's ComponentConfig is `/var/lib/kubelet/config.yaml`.
|
||||
- The dynamic environment file that contains `KUBELET_KUBEADM_ARGS` is sourced from `/var/lib/kubelet/kubeadm-flags.env`.
|
||||
- The file that can contain user-specified flag overrides with `KUBELET_EXTRA_ARGS` is sourced from
|
||||
`/etc/default/kubelet` (for DEBs), or `/etc/systconfig/kubelet` (for RPMs). `KUBELET_EXTRA_ARGS`
|
||||
is last in the flag chain and has the highest priority in the event of conflicting settings.
|
||||
|
||||
## Kubernetes binaries and package contents
|
||||
|
||||
The DEB and RPM packages shipped with the Kubernetes releases are:
|
||||
|
||||
| Package name | Description |
|
||||
|--------------|-------------|
|
||||
| `kubeadm` | Installs the `/usr/bin/kubeadm` CLI tool and [The kubelet drop-in file(#the-kubelet-drop-in-file-for-systemd) for the kubelet. |
|
||||
| `kubelet` | Installs the `/usr/bin/kubelet` binary. |
|
||||
| `kubectl` | Installs the `/usr/bin/kubectl` binary. |
|
||||
| `kubernetes-cni` | Installs the official CNI binaries into the `/opt/cni/bin` directory. |
|
||||
| `cri-tools` | Installs the `/usr/bin/crictl` binary from [https://github.com/kubernetes-incubator/cri-tools](https://github.com/kubernetes-incubator/cri-tools). |
|
||||
|
||||
{{% /capture %}}
|
||||
Loading…
Reference in New Issue