Comfy-Org / Comfy-Org/ComfyUI

CUDA Malloc blacklist not working correclty

Open
#4,668 0 comments 1 reaction 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

### Expected Behavior

Any GPU that is in the blacklist, cudaMallocAsync should be disabled by default.

### Actual Behavior

Some GPU that is in the blacklist, cudaMallocAsync is enabled by default.

### Steps to Reproduce

Run `python main.py` without any flags.

### Debug Logs

```powershell
Total VRAM 8112 MB, total RAM 32013 MB
pytorch version: 2.5.0.dev20240826+cu124
Set vram state to: NORMAL_VRAM
Device: cuda:0 Quadro M4000 : cudaMallocAsync
Using pytorch cross attention
[Prompt Server] web root: /home/harryn0502/Documents/SD/uis/ComfyUI/web
/home/harryn0502/miniconda3/envs/comfy/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
Skipping loading of custom nodes

WARNING: this card most likely does not support cuda-malloc, if you get "CUDA error" please run ComfyUI with: --disable-cuda-malloc

Starting server

To see the GUI go to: http://127.0.0.1:8188
```

### Other

It seems to be a problem where when I run `nvidia-smi -L` on my terminal where GPU is Quadro M4000 the string it outputs

> GPU 0: Quadro M4000 (UUID: GPU-4aa7611d-f3e5-198c-509e-1d353e52662f).

However, if I run the same command above on my RTX 3090 Ti, the string outputs

> GPU 0: NVIDIA GeForce RTX 3090 Ti (UUID: GPU-5b707f84-2794-60ae-d1bf-3a64f1f87bf0).

I believe this is due to that at [cuda_malloc.py on line 59](https://github.com/comfyanonymous/ComfyUI/blob/34eda0f853daffdfcab04e1b3187de40f1d30bbf/cuda_malloc.py#L59), the code tries to check if the variable `names` contain a string `"NVIDIA"` before trying to set the flag `cuda_malloc` based on the blacklist. As you can see above Quadro M4000 does not contain a string `"NVIDIA"` resulting this bug.

**I recommend remove line 59 to fix this bug.**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.