GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators

`nvidia-device-plugin` failed to run on GPU nodes created by Node Auto-Provisioning

Open
#407 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
252
Forks
184
Avg merge
1h 55m
Merged PRs (30d)
2

Description

## How to reproduce

* Create a GKE cluster in Standard mode
* Enable Node Auto-Provisioning with L4 GPU capacity
* Try to create a pod with `nvidia.com/gpu` resource request
* The pod will be stuck in PodInitializing state

## Analysis

Using GPU nodes with Node-Auto-Provisioning in GKE will try to run `nvidia-gpu-device-plugin` job on GPU nodes. It's only after this job finishes successfully that the node will have allocatable `nvidia.com/gpu` resources. However, this job is stuck, and no pod with GPU requests can be scheduled onto it.

```
$ kubectl -n kube-system get pod

kube-system nvidia-gpu-device-plugin-small-cos-4fpl4 0/2 Init:0/2 0 16m

$ kubectl -n kube-system logs nvidia-gpu-device-plugin-small-cos-hzvds -c nvidia-driver-installer

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 598 100 598 0 0 273k 0 --:--:-- --:--:-- --:--:-- 291k
GPU driver auto installation is disabled.
Waiting for GPU driver libraries to be available.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.