GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators

Nvidia driver failed while using cuda 10.0 in Kubernetes Cluster

Open
#133 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
252
Forks
184
Avg merge
1h 55m
Merged PRs (30d)
2

Description

we run docker containers in GKE(Google Kubernetes Engine) with 12.x version with cuda 10.0 version and cudnn>7.6.5 . Actually the nvidia driver installed as per the docs through https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml gives 410.79 nvidia-driver in the cluster. But looks like the cuda installs different nvidia driver in the container and mismatch with the kernel version. While doing **nvidia-smi** in the container, we get:
**Failed to initialize NVML: Driver/library version mismatch**
How can we solve this issue?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.