NVIDIA-NeMo / NVIDIA-NeMo/RL

NeMo RL CUDA 13.0 failed to initiailize on RTX 6000 Blackwell

Open
#1,517 1 comment 0 reactions 0 assignees View on GitHub
bug i-tme t-dependency
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

NeMo RL container says CUDA failed to initialize and can't run training with CUDA 13.0 on RTX 6000 Blackwell

**Steps/Code to reproduce bug**

I tried running NeMo RL with the following setup:

- GPUs: 8xRTX 6000 Blackwell
- CUDA: 13.0

When the NeMo RL container loads up, it outputs:

```
CUDA Version 12.9.0.043
```

And then:

```
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available. [[ Initialization error (error 3) ]]
```

I tried running training anyway, and got this error:

```
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.
```

**Expected behavior**

Runtime error listed above

**Additional context**

I can successfully run `nvidia-smi` on the host machine and from within the NeMo RL container. They both show CUDA 13.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.