GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators

Is it okay to change Nvidia Driver version other than that given by Daemonset mentioned in GPU for Kubernetes Cluster Docs

Open
#134 1 comment 5 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
252
Forks
184
Avg merge
1h 55m
Merged PRs (30d)
2

Description

We are currently using nvidia driver daemonset as given in https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
Unfortunately it supports 410.79 nvidia driver which is compatible with kubernetes 1.12X version. But, we want to upgrade our driver to 440.x and also upgrade our cuda libraries which is currently cuda-10.0, we have also used Telsa T4 with tensor cores for GPU hardware. We are already at production but, some of our tensorflow code are not running. We have used tensorflow-gpu=2.0.0. Most surprising fact is, the same code runs perfectly on local computer with GPU support. The GPU specs used in local computer are Nivida GTX 1060 and 1080. It has become hard to track what issue might have caused the problem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.