GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators

nvidia-gpu-device-plugin gets OOM killed

Open
#202 12 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
252
Forks
184
Avg merge
1h 55m
Merged PRs (30d)
2

Description

Hey folks,
In our experiments with running GPU loads over GKE over at [iguazio](https://www.iguazio.com) we've hit OOM kill of the NVIDIA GPU device plugin pod during a gpu load test
```
$ kubectl -n kube-system describe pod nvidia-gpu-device-plugin-ngkrv | grep OOM -A15
Reason: OOMKilled
Exit Code: 137
Started: Tue, 27 Jul 2021 11:15:02 +0300
Finished: Tue, 27 Jul 2021 16:49:32 +0300
Ready: True
Restart Count: 1
Limits:
cpu: 50m
memory: 20Mi
Requests:
cpu: 50m
memory: 20Mi
Environment:
LD_LIBRARY_PATH: /usr/local/nvidia/lib64
Mounts:
/dev from dev (rw)
```

This happened on ubuntu nodes w GPU (n1-standard-16 though I don't think it matters 😄 ), running GKE engine 1.19.9-gke.1900

I suspect the allocated resources for it (mem limits) might not be enough. Maybe up it to 40Mi?
We're using the documented way of installing it as describe in https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#ubuntu

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.