GoogleCloudPlatform / GoogleCloudPlatform/compute-gpu-installation
GKE's GPU operator mounts to directories that are no longer supported by NVIDIA's CUDA base images
- Dominant language
- Python
- Stars
- 89
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
NVIDIA's recent CUDA base images, like [12.8.1-cudnn-devel-ubuntu22.04](https://hub.docker.com/layers/nvidia/cuda/12.8.1-cudnn-devel-ubuntu22.04/images/sha256-61f6c08f2b59036cb935e56d1e31a6b64e3ae2c7ddb86d33fa0b044c7917b719) no longer include `/usr/local/nvidia/lib64` in `LD_LIBRARY_PATH` or /`usr/local/nvidia/bin` in `PATH`, so e.g. `nvidia-smi` fails on GKE.
We added these to our base images' `PATH` variables manually for compatibility for our customers who use GKE. Ideally we can remove this workaround in case it is a symptom of other latent incompatibilities with the GKE operator. EKS does not have this problem.
Please let me know if I can provide any additional information. Thanks! Ref: https://github.com/ray-project/ray/issues/54551
Contributor guide
Assessment
This issue has not been assessed yet.