GoogleCloudPlatform / GoogleCloudPlatform/kubernetes-engine-samples
Fine-tune Gemma image has issues, not detecting GPU
Open
:rotating_light:
- Dominant language
- HCL
- Stars
- 1.4k
- Forks
- 1.3k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 4
Description
https://github.com/GoogleCloudPlatform/kubernetes-engine-samples/blob/main/ai-ml/llm-finetuning-gemma/Dockerfile
The dependabot update causes the fine-tune job to fail due to not detecting the GPU.
GKE Autopilot 1.33.x
The updates causes the issue on the following image versions tested:
- `nvidia/cuda:12.9.1-runtime-ubuntu22.04`
- `nvidia/cuda:12.9.1-runtime-ubuntu24.04`
The original version `nvidia/cuda:12.2.0-runtime-ubuntu22.04` still works.
Contributor guide
Assessment
This issue has not been assessed yet.