NeMo RL CUDA 13.0 failed to initiailize on RTX 6000 Blackwell
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
NeMo RL container says CUDA failed to initialize and can't run training with CUDA 13.0 on RTX 6000 Blackwell
**Steps/Code to reproduce bug**
I tried running NeMo RL with the following setup:
- GPUs: 8xRTX 6000 Blackwell
- CUDA: 13.0
When the NeMo RL container loads up, it outputs:
```
CUDA Version 12.9.0.043
```
And then:
```
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available. [[ Initialization error (error 3) ]]
```
I tried running training anyway, and got this error:
```
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.
```
**Expected behavior**
Runtime error listed above
**Additional context**
I can successfully run `nvidia-smi` on the host machine and from within the NeMo RL container. They both show CUDA 13.0
Contributor guide
Assessment
This issue has not been assessed yet.