[TRAIN-7] RuntimeError: No CUDA GPUs are available on A100; fix requires TORCH_CUDA_ARCH_LIST
Open
bug
Documentation
p1
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
## Problem
Running `examples/run_grpo.py` on an A100 node produces `RuntimeError: No CUDA GPUs are available`, even though GPUs are accessible and CUDA is in `PATH`. The fix is to export `TORCH_CUDA_ARCH_LIST="8.0"`, but this is not documented anywhere.
## Steps to reproduce
Run `examples/run_grpo.py` on an A100 node without setting `TORCH_CUDA_ARCH_LIST`. Observe the error.
## Suggested fix
Document that `TORCH_CUDA_ARCH_LIST` must be set to match the target GPU architecture. Consider detecting the GPU architecture automatically or setting a sensible default.
Contributor guide
Assessment
This issue has not been assessed yet.