Azure / Azure/kubernetes-volume-drivers
mount hangs on start
- Dominant language
- Shell
- Stars
- 127
- Forks
- 62
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 2
Description
**What happened**:
When create I pv and pvc, I can see the smb bash script calling the mount command but it never returns. I have to add "log '{"status": "Success"}'" so the script returns the json so kubelet is happy. Even adding in an echo statement makes it work. Running the raw mount command works as well. I think it has to do some forking that is going on that kubelet is not getting back the json from the ppid.
**What you expected to happen**:
log '{"status": "Success"}' returned json to kubelet so its happy.
**How to reproduce it**:
When you run smb it will hang. Use the below configs
pv:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-smb-flexvol
namespace: foobar
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
flexVolume:
driver: "microsoft.com/smb"
secretRef:
name: smbcreds
options:
source: "//some ip/someplace"
mountoptions: "vers=3.0,dir_mode=0777,file_mode=0777"
pvc:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-smb-flexvol
namespace: foobar
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
volumeName: pv-smb-flexvol
storageClassName: ""
Of course following the normal installation you provided with the plugins dir etc.
**Anything else we need to know?**:
Here is what I had to add to make it work:
mount() {
MNTPATH="$1"
VOLUME_SRC=$(echo "$JSON"|"$JQ" -r '.source')
READ_MODE=$(echo "$JSON"|"$JQ" -r '.["kubernetes.io/readwrite"]')
MOUNT_OPTIONS=$(echo "$JSON"|"$JQ" -r '.mountoptions // empty')
USERNAME=$(echo "$JSON"|"$JQ" -r '.["kubernetes.io/secret/username"] // empty'|base64 -d)
PASSWORD=$(echo "$JSON"|"$JQ" -r '.["kubernetes.io/secret/password"] // empty'|base64 -d)
DOMAIN=$(echo "$JSON"|"$JQ" -r '.["kubernetes.io/secret/domain"] // empty'|base64 -d)
ALL_OPTIONS="username=${USERNAME},password=${PASSWORD},${READ_MODE}"
if [ -n "$DOMAIN" ]; then
ALL_OPTIONS="${ALL_OPTIONS},domain=${DOMAIN}"
fi
if [ -n "$MOUNT_OPTIONS" ]; then
ALL_OPTIONS="${ALL_OPTIONS},${MOUNT_OPTIONS}"
fi
if ismounted ; then
log '{"status": "Success"}'
exit 0
fi
echo "`date` EXEC: mkdir -p ${MNTPATH}" >> $LOG
mkdir -p ${MNTPATH} >>$LOG 2>&1
#mounting
echo "`date` EXEC: /bin/mount -t cifs ${VOLUME_SRC} ${MNTPATH}" >>$LOG
log '{"status": "Success"}' # <------------------- HERE!!!
/bin/mount -t cifs "${VOLUME_SRC}" "${MNTPATH}" -o "${ALL_OPTIONS}" >>$LOG 2>&1
if [ $? -ne 0 ]; then
errorLog=`tail -n 1 "${LOG}"`
err "{ \"status\": \"Failure\", \"message\": \"Failed to mount device ${DMDEV} at ${MNTPATH}, user:${USERNAME}, ${VOLUME_SRC}, error log:${errorLog}\" }"
exit 1
fi
log '{"status": "Success"}'
exit 0
}
As you can see i just added the log right before the mount for it to work, but adding anything really will make it work. ie, echo "foo".
**Environment**:
- Kubernetes version (use `kubectl version`):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
- OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
- Kernel (e.g. `uname -a`):
Linux ip-10-161-160-117 4.4.0-1075-aws #85-Ubuntu SMP Thu Jan 17 17:15:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
- Install tools:
- Others:
This is running jupyterhub with a shared volume, which is basically just utilizing a shared volume:
https://zero-to-jupyterhub.readthedocs.io/en/latest/user-storage.html
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the SMB bash script's mount() entry point and reproduce the hang using the provided PersistentVolume and PersistentVolumeClaim configurations. Compare behavior with and without the added log or echo around /bin/mount, and inspect how the script returns JSON to kubelet. Done means the raw mount command completes and kubelet receives the expected success response without an extra statement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, shell
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100