GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators
nvidia-driver-installer crash loop during GKE scale ups
- Dominant language
- Go
- Stars
- 252
- Forks
- 184
- Avg merge
- 1h 55m
- Merged PRs (30d)
- 2
Description
We've been using the nvidia-driver-installer on Ubuntu node groups via GKE v1.15 per the official How-to GPU instructions specified [here](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus).
```bash
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/ubuntu/daemonset-preloaded.yaml
```
The daemonset deployed via [`daemonset-preloaded.yaml`](https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/nvidia-driver-installer/ubuntu/daemonset-preloaded.yaml) appeared to work correctly for some time, however we started noticing issues last Friday when new nodes were added to the node group via cluster autoscaling. The nvidia-driver-installer daemonset pods that were scheduled to these new nodes began to crash loop, as their initContainers were exiting with non-zero exit codes.
Upon examining pod logs, it appears that the failed pods contain the following lines as their last output before exiting.
```
Verifying Nvidia installation... DONE.
ln: /root/home/kubernetes/bin/nvidia: cannot overwrite directory
```
See [here](https://pastebin.com/raw/HpnmGTaz) for full log output from one of the failed pods.
I've logged into one of the nodes and manually removed the `/root/home/kubernetes/bin/nvidia` folder (which is presumably created by the very first instance of the nvidia-driver-installer pod scheduled to a node when it comes up) but the folder re-appears and the daemonset pods continue to crash in loop. Nodes that have daemonset pods in this state don't have the drivers correctly installed, and jobs that require them fail to import CUDA due to driver issues.
We've been experiencing this issue for 4 days now with nodes that receive live production traffic. Not every node that scales up experiences this problem, but most do. If a node comes up and its nvidia-driver-installer pod begins to crash, we've had no luck bringing it out of that state. Instead we've manually marked the node as unschedulable and brought it down, hoping the next to come up won't experience the same problem.
From our perspective, nothing has changed with our cluster configuration, node group configuration, or K8s manifests that would cause this issue to start occurring. We did experience something similar for a few hours in mid December, but the issue resolved itself within a few hours and we didn't think much of it. I'm happy to provide more logs or detailed information about the errors upon request!
Any thoughts about what could be causing this?
Contributor guide
Assessment
This issue has not been assessed yet.