GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators
nvidia-gpu-device-plugin gets OOM killed
- Dominant language
- Go
- Stars
- 252
- Forks
- 184
- Avg merge
- 1h 55m
- Merged PRs (30d)
- 2
Description
Hey folks,
In our experiments with running GPU loads over GKE over at [iguazio](https://www.iguazio.com) we've hit OOM kill of the NVIDIA GPU device plugin pod during a gpu load test
```
$ kubectl -n kube-system describe pod nvidia-gpu-device-plugin-ngkrv | grep OOM -A15
Reason: OOMKilled
Exit Code: 137
Started: Tue, 27 Jul 2021 11:15:02 +0300
Finished: Tue, 27 Jul 2021 16:49:32 +0300
Ready: True
Restart Count: 1
Limits:
cpu: 50m
memory: 20Mi
Requests:
cpu: 50m
memory: 20Mi
Environment:
LD_LIBRARY_PATH: /usr/local/nvidia/lib64
Mounts:
/dev from dev (rw)
```
This happened on ubuntu nodes w GPU (n1-standard-16 though I don't think it matters 😄 ), running GKE engine 1.19.9-gke.1900
I suspect the allocated resources for it (mem limits) might not be enough. Maybe up it to 40Mi?
We're using the documented way of installing it as describe in https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#ubuntu
Contributor guide
Assessment
This issue has not been assessed yet.