119 lines
4.2 KiB
Markdown
119 lines
4.2 KiB
Markdown
## CSI driver debug tips
|
||
|
||
### Case#1: volume create/delete failed
|
||
> There could be multiple controller pods (only one pod is the leader), if there are no helpful logs, try to get logs from the leader controller pod.
|
||
- find csi driver controller pod
|
||
```console
|
||
$ kubectl get po -o wide -n kube-system | grep csi-smb-controller
|
||
NAME READY STATUS RESTARTS AGE IP NODE
|
||
csi-smb-controller-56bfddd689-dh5tk 5/5 Running 0 35s 10.240.0.19 k8s-agentpool-22533604-0
|
||
csi-smb-controller-56bfddd689-sl4ll 5/5 Running 0 35s 10.240.0.23 k8s-agentpool-22533604-1
|
||
```
|
||
- get pod description and logs
|
||
```console
|
||
$ kubectl describe po csi-smb-controller-56bfddd689-dh5tk -n kube-system > csi-smb-controller-description.log
|
||
$ kubectl logs csi-smb-controller-56bfddd689-dh5tk -c smb -n kube-system > csi-smb-controller.log
|
||
```
|
||
|
||
### Case#2: volume mount/unmount failed
|
||
- find csi driver pod that does the actual volume mount/unmount
|
||
```console
|
||
$ kubectl get po -o wide -n kube-system | grep csi-smb-node
|
||
NAME READY STATUS RESTARTS AGE IP NODE
|
||
csi-smb-node-cvgbs 3/3 Running 0 7m4s 10.240.0.35 k8s-agentpool-22533604-1
|
||
csi-smb-node-dr4s4 3/3 Running 0 7m4s 10.240.0.4 k8s-agentpool-22533604-0
|
||
```
|
||
|
||
- get pod description and logs
|
||
```console
|
||
$ kubectl describe po csi-smb-node-cvgbs -n kube-system > csi-smb-node-description.log
|
||
$ kubectl logs csi-smb-node-cvgbs -c smb -n kube-system > csi-smb-node.log
|
||
```
|
||
|
||
- check cifs mount inside driver
|
||
```console
|
||
kubectl exec -it csi-smb-node-cvgbs -n kube-system -c smb -- mount | grep cifs
|
||
```
|
||
|
||
- get Windows csi-proxy logs inside driver
|
||
```console
|
||
kubectl exec -it csi-smb-node-win-xxxxx -n kube-system -c smb cmd
|
||
type c:\k\csi-proxy.err.log
|
||
```
|
||
|
||
#### Update driver version quickly by editing driver deployment directly
|
||
- update controller deployment
|
||
```console
|
||
kubectl edit deployment csi-smb-controller -n kube-system
|
||
```
|
||
- update daemonset deployment
|
||
```console
|
||
kubectl edit ds csi-smb-node -n kube-system
|
||
```
|
||
change below deployment config, e.g.
|
||
```console
|
||
image: registry.k8s.io/sig-storage/smbplugin:v1.8.0
|
||
imagePullPolicy: Always
|
||
```
|
||
|
||
### troubleshooting connection failure on agent node
|
||
- On Linux node
|
||
```console
|
||
mkdir /tmp/test
|
||
sudo mount -v -t cifs //smb-server/fileshare /tmp/test -o vers=3.0,username=accountname,password=accountkey,dir_mode=0777,file_mode=0777,cache=strict,actimeo=30
|
||
```
|
||
|
||
- Check whether original smb mount directory works
|
||
```console
|
||
sudo mount | grep cifs
|
||
```
|
||
|
||
<details><summary>
|
||
Get client-side logs on Linux node if there is mount error
|
||
</summary>
|
||
|
||
```console
|
||
kubectl debug node/node-name --image=nginx
|
||
kubectl cp node-debugger-node-name-xxxx:/host/var/log/messages /tmp/messages
|
||
kubectl cp node-debugger-node-name-xxxx:/host/var/log/syslog /tmp/syslog
|
||
kubectl cp node-debugger-node-name-xxxx:/host/var/log/kern.log /tmp/kern.log
|
||
#after log collected, delete the debug pod by:
|
||
kubectl delete po node-debugger-node-name-xxxx
|
||
```
|
||
|
||
</details>
|
||
|
||
- On Windows node
|
||
```console
|
||
$User = "AZURE\USERNAME"
|
||
$PWord = ConvertTo-SecureString -String "PASSWORD" -AsPlainText -Force
|
||
$Credential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $User, $Pword
|
||
New-SmbGlobalMapping -LocalPath x: -RemotePath \\smb-server\fileshare -Credential $Credential
|
||
Get-SmbGlobalMapping
|
||
cd x:
|
||
dir
|
||
```
|
||
|
||
<details><summary>
|
||
Get client-side logs on Windows node if there is mount error
|
||
</summary>
|
||
|
||
```console
|
||
Get SMBClient events from Event Viewer under following path:
|
||
Application and Services Logs -> Microsoft -> Windows -> SMBClient
|
||
```
|
||
|
||
</details>
|
||
|
||
### Configure [csi-proxy](https://github.com/kubernetes-csi/csi-proxy#installation) on Windows node
|
||
> Start a Powershell window as admin
|
||
```console
|
||
> cd c:\k
|
||
Invoke-WebRequest https://acs-mirror.azureedge.net/csi-proxy/v1.0.2/binaries/csi-proxy-v1.0.2.tar.gz -OutFile csi-proxy.tar.gz;
|
||
tar -xvf csi-proxy.tar.gz
|
||
copy .\bin\csi-proxy.exe .
|
||
sc.exe create csiproxy binPath= "c:\k\csi-proxy.exe -windows-service -log_file=c:\k\csi-proxy.log -logtostderr=false --v=5"
|
||
sc.exe failure csiproxy reset= 0 actions= restart/10000
|
||
sc.exe start csiproxy
|
||
```
|