kubernetes
/
kops
mirror of https://github.com/kubernetes/kops.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add troubleshooting of corrupted api server leases

This commit is contained in:
Ole Markus With 2021-02-08 12:07:34 +01:00
parent 741564e22c
commit 0c5ec01d4a
1 changed files with 16 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
docs/operations/troubleshoot.md
View File

@ -35,6 +35,22 @@ If nodeup succeeds, the core kube containers should have started. Look for the A
Often the issue is obvious such as passing incorrect CLI flags.
### API Server hangs after etcd restore
After resizing an etcd cluster or restoring backup, the kubernetes API can contain too many endpoints.
You can confirm this by running `kubectl get endpoints -n default kubernetes`. This command should list exactly as many IPs as you have control plane nodes.
[This bug](https://github.com/kubernetes/kubernetes/issues/86812) caueses old apiserver leases to get stuck. In order to recover from this you need to remove the leases from etcd directly:
```
CONTAINER=$(kubectl get pods -n kube-system | grep etcd-manager-main | head -n 1 | awk '{print $1}')
kubectl exec -it -n kube-system $CONTAINER -- sh
cd /opt/etcd-v3.4.13-linux-amd64/
./etcdctl --cacert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt --cert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt --key=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key --endpoints=https://127.0.0.1:4001 del --prefix /registry/masterleases/
```
The remaining api servers will immediately recreate their own leases.
## etcd
The API server makes use of two etcd servers, main and events.