kubernetes
/
kops
mirror of https://github.com/kubernetes/kops.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update troubleshoot.md

This commit is contained in:
simonc 2022-05-26 10:51:27 +01:00 committed by GitHub
parent 074f780990
commit bc93ec53ac
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 22 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
docs/operations/troubleshoot.md
View File

@ -40,13 +40,31 @@ Often the issue is obvious such as passing incorrect CLI flags.
After resizing an etcd cluster or restoring backup, the kubernetes API can contain too many endpoints.
You can confirm this by running `kubectl get endpoints -n default kubernetes`. This command should list exactly as many IPs as you have control plane nodes.
[This bug](https://github.com/kubernetes/kubernetes/issues/86812) caueses old apiserver leases to get stuck. In order to recover from this you need to remove the leases from etcd directly:
[This bug](https://github.com/kubernetes/kubernetes/issues/86812) causes old apiserver leases to get stuck. In order to recover from this you need to remove the leases from etcd directly:
```
CONTAINER=$(kubectl get pods -n kube-system | grep etcd-manager-main | head -n 1 | awk '{print $1}')
kubectl exec -it -n kube-system $CONTAINER -- sh
cd /opt/etcd-v3.4.13-linux-amd64/
./etcdctl --cacert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt --cert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt --key=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key --endpoints=https://127.0.0.1:4001 del --prefix /registry/masterleases/
```
etcd and etcdctl are installed into directories in /opt - look for the latest version eg 3.5.1
```
# DIRNAME=/opt/etcd-v3.5.1-linux-amd64
# ETCDCTL_API=3
# alias etcdctl='$DIRNAME/etcdctl --cacert=/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.crt --cert=/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key --key=/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key --endpoints=https://127.0.0.1:4001'
```
You can get a list of the leases eg:
```
etcdctl get --prefix /registry/masterleases
```
And delete with:
```
etcdctl del /registry/masterleases/$IP_ADDRESS
```
Also you can delete all of the leases in one go...
```
./etcdctl --cacert=/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.crt --cert=/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key --key=/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key --cert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt --endpoints=https://127.0.0.1:4001 del --prefix /registry/masterleases/
```
The remaining api servers will immediately recreate their own leases.
@ -79,4 +97,4 @@ If the directory is there, but the CNI plugin and configuration is missing, it m
At this point it is worth repeating that the control plane _will work_ without CNI. Most control plane nodes do not use the pod network but communicates using the host's network. If you cannot talk to the API server, e.g running `kubectl get nodes`, the problem is not CNI.
If the API is working, and the CNI is installed through a `DaemonSet`, check that the pods are running. If pods are expected, but absent, it may be an issue with installing the CNI addon. kOps will try to install addons regularly, so run `journalctl -f` on a control plane node to spot any errors.
If the API is working, and the CNI is installed through a `DaemonSet`, check that the pods are running. If pods are expected, but absent, it may be an issue with installing the CNI addon. kOps will try to install addons regularly, so run `journalctl -f` on a control plane node to spot any errors.