Azure / Azure/kubernetes-volume-drivers
Node unable to communicate with Windows file share after unscheduled pod deletion
- Dominant language
- Shell
- Stars
- 127
- Forks
- 62
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 2
Description
**What happened**:
Connecting to a windows file share using SMB3 from Linux AKS and works great initially. Have 3 nodes with multiple pods in each that all mount to the same windows file share. On a few occasions after it has been up for a few days and a pod has been deleted through the kubernetes dashboard, all pods running on that node are no longer able to connect to the windows file share. Existing pods that have not be restarted get this error when trying to navigate to the mounted share:
mount error(112): Host is down
New pods trying to start on the node get this error:
```
MountVolume.SetUp failed for volume "app-smb-share-pv" : mount command failed, status: Failure, reason: Failed to mount device at /var/lib/kubelet/pods/4a20d527-116e-4aca-a2f1-59e0c9319a89/volumes/microsoft.com~smb/app-smb-share-pv, user:XXXX, //WINSERVER/file-share, error log:Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
```
Nothing different in the smb-driver.log file
In the warn log, get these:
```
Mar 30 12:46:18 aks-nodepool1-XXX-vmss00000H kernel: [328413.074290] CIFS VFS: cifs_mount failed w/return code = -112
Mar 30 12:46:18 aks-nodepool1-XXX-vmss00000H kernel: [328413.081632] CIFS VFS: ioctl error in smb2_get_dfs_refer rc=-5
```
Can use smbclient to mount to the same windows file share without a problem so know it is not a fundamental networking issue. If I restart the node from the Azure portal, then pods in node can mount fine.
YAML files below.
Seems like something isn't gracefully happening *sometimes* when pod is deleted. Am I doing something wrong?
**What you expected to happen**:
Deleting a pod does not cause the mount for the entire node to become unusable.
**How to reproduce it**:
Start of AKS cluster that has a mount to a SMB 3 windows file share, unceremoniously delete a pod (cancel a helm deployment, delete pod using kubernetes dashboard) and other pods and any new pods on the node can no longer communicate with the Windows File Share.
**Anything else we need to know?**:
Here is the PersistentVolume YAML being used:
```
apiVersion: v1
kind: PersistentVolume
metadata:
name: app-smb-share-pv
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
storageClassName: appshare
persistentVolumeReclaimPolicy: Retain
flexVolume:
driver: "microsoft.com/smb"
secretRef:
name: smbcreds
namespace: default
options:
source: "///file-share"
mountoptions: "vers=3.0,dir_mode=0777,file_mode=0777,sec=ntlmv2,domain=MYDOMAIN"
```
Volume mount in each pod:
```
- name: appshare-volume
mountPath: /mnt/app
```
Volume in each pod:
```
- name: appshare-volume
persistentVolumeClaim:
claimName: app-share-pvc
```
**Environment**:
- Kubernetes version (use `kubectl version`): 1.15.7
- OS (e.g. from /etc/os-release): Linux
- Kernel (e.g. `uname -a`): Linux app-back-office-deployment-7fd86c5bc7-v5klp 4.15.0-1069-azure #74-Ubuntu SMP Fri Feb 7 17:22:24 UTC 2020 x86_64 GNU/Linux
- Install tools: helm
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the PersistentVolume YAML, the smb-driver.log output, and the kernel CIFS errors shown after an unscheduled pod deletion. Reproduce the failure on an AKS node and compare the mount behavior with smbclient and after a node restart; done means deleting a pod no longer prevents existing or new pods on that node from mounting the Windows file share.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, kubernetes, linux
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100