GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators
Is it okay to change Nvidia Driver version other than that given by Daemonset mentioned in GPU for Kubernetes Cluster Docs
- Dominant language
- Go
- Stars
- 252
- Forks
- 184
- Avg merge
- 1h 55m
- Merged PRs (30d)
- 2
Description
We are currently using nvidia driver daemonset as given in https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
Unfortunately it supports 410.79 nvidia driver which is compatible with kubernetes 1.12X version. But, we want to upgrade our driver to 440.x and also upgrade our cuda libraries which is currently cuda-10.0, we have also used Telsa T4 with tensor cores for GPU hardware. We are already at production but, some of our tensorflow code are not running. We have used tensorflow-gpu=2.0.0. Most surprising fact is, the same code runs perfectly on local computer with GPU support. The GPU specs used in local computer are Nivida GTX 1060 and 1080. It has become hard to track what issue might have caused the problem.
Contributor guide
Assessment
This issue has not been assessed yet.