GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators

Update driver version: how?

Open
#107 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
252
Forks
184
Avg merge
1h 55m
Merged PRs (30d)
2

Description

When updating the driver version (by editing the DaemonSet `NVIDIA_DRIVER_VERSION` environment variable for example), the nodes need a restart/reset:
- with `stable` branche the update strategy is `OnDelete`: we need to manually delete the pods first
- the new pods then fail: the new driver installation fails on nodes where another kernel module version is already loaded.

Example:
```
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 418.40.04................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

ERROR: An NVIDIA kernel module 'nvidia-uvm' appears to already be loaded in
your kernel. This may be because it is in use (for example, by an X
server, a CUDA program, or the NVIDIA Persistence Daemon), but this
may also happen if your kernel was configured without support for
module unloading. Please be sure to exit any programs that may be
using the GPU(s) before attempting to upgrade your driver. If no
GPU-based programs are running, you know that your kernel supports
module unloading, and you still receive this message, then an error
may have occured that has corrupted an NVIDIA kernel module's usage
count, for which the simplest remedy is to reboot your computer.

ERROR: Installation has failed. Please see the file
'/usr/local/nvidia/nvidia-installer.log' for details. You may find
suggestions on fixing installation problems in the README available
on the Linux driver download page at www.nvidia.com.
```

After a node restart/reset, the installation works with the new version.

=> Maybe document somewhere how to upgrade the driver version?

Remark: since #66 (included in `master`, not in `stable`, which is documented in https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers) the update strategy changed from `OnDelete` to `RollingUpdate`; this avoids the need to manually delete the pods (which is required even with the node restart/reset); but the node restart/reset is still needed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.