GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators
Nvidia driver failed while using cuda 10.0 in Kubernetes Cluster
- Dominant language
- Go
- Stars
- 252
- Forks
- 184
- Avg merge
- 1h 55m
- Merged PRs (30d)
- 2
Description
we run docker containers in GKE(Google Kubernetes Engine) with 12.x version with cuda 10.0 version and cudnn>7.6.5 . Actually the nvidia driver installed as per the docs through https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml gives 410.79 nvidia-driver in the cluster. But looks like the cuda installs different nvidia driver in the container and mismatch with the kernel version. While doing **nvidia-smi** in the container, we get:
**Failed to initialize NVML: Driver/library version mismatch**
How can we solve this issue?
Contributor guide
Assessment
This issue has not been assessed yet.