Comfy-Org / Comfy-Org/ComfyUI

Default arg "cuda-malloc" causes CUDA error: operation not supported on GTX 960M GPU

Open
#940 43 comments 1 reaction 0 assignees View on GitHub
User Support
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

I have confirmed that the default arg `--cuda-malloc` causes error on my computer. I must disable it by adding `--disable-cuda-malloc` to let ComfyUI work properly.

If I don't disable it, the following CUDA error will occur when I try to gerenated an image:
```powershell
(venv) PS C:\Users\wklchris> python "${comfyuiDir}/main.py" --lowvram
Total VRAM 2048 MB, total RAM 8076 MB
Trying to enable lowvram mode because your GPU seems to have 4GB or less. If you don't want this use: --normalvram
Set vram state to: LOW_VRAM
Device: cuda:0 NVIDIA GeForce GTX 960M : cudaMallocAsync
Using pytorch cross attention
Adding extra search path checkpoints D:/Git-repos/stable-diffusion-webui\models/Stable-diffusion
Adding extra search path configs D:/Git-repos/stable-diffusion-webui\models/Stable-diffusion
Adding extra search path vae D:/Git-repos/stable-diffusion-webui\models/VAE
Adding extra search path loras D:/Git-repos/stable-diffusion-webui\models/Lora
Adding extra search path loras D:/Git-repos/stable-diffusion-webui\models/LyCORIS
Adding extra search path upscale_models D:/Git-repos/stable-diffusion-webui\models/ESRGAN
Adding extra search path upscale_models D:/Git-repos/stable-diffusion-webui\models/RealESRGAN
Adding extra search path upscale_models D:/Git-repos/stable-diffusion-webui\models/SwinIR
Adding extra search path embeddings D:/Git-repos/stable-diffusion-webui\embeddings
Adding extra search path hypernetworks D:/Git-repos/stable-diffusion-webui\models/hypernetworks
Adding extra search path controlnet D:/Git-repos/stable-diffusion-webui\models/ControlNet
Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
model_type EPS
adm 0
making attention of type 'vanilla-pytorch' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-pytorch' with 512 in_channels
left over keys: dict_keys(['model_ema.decay', 'model_ema.num_updates'])
!!! Exception during processing !!!
Traceback (most recent call last):
File "D:\Git-repos\ComfyUI\execution.py", line 145, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\Git-repos\ComfyUI\execution.py", line 75, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\Git-repos\ComfyUI\execution.py", line 68, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\Git-repos\ComfyUI\nodes.py", line 1082, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "D:\Git-repos\ComfyUI\nodes.py", line 1052, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "D:\Git-repos\ComfyUI\comfy\sample.py", line 75, in sample
comfy.model_management.load_model_gpu(model)
File "D:\Git-repos\ComfyUI\comfy\model_management.py", line 298, in load_model_gpu
accelerate.dispatch_model(real_model, device_map=device_map, main_device=torch_dev)
File "D:\Git-repos\stable-diffusion-webui\venv\lib\site-packages\accelerate\big_modeling.py", line 370, in dispatch_model
attach_align_device_hook_on_blocks(
File "D:\Git-repos\stable-diffusion-webui\venv\lib\site-packages\accelerate\hooks.py", line 498, in attach_align_device_hook_on_blocks
add_hook_to_module(module, hook)
File "D:\Git-repos\stable-diffusion-webui\venv\lib\site-packages\accelerate\hooks.py", line 155, in add_hook_to_module
module = hook.init_hook(module)
File "D:\Git-repos\stable-diffusion-webui\venv\lib\site-packages\accelerate\hooks.py", line 251, in init_hook
set_module_tensor_to_device(module, name, self.execution_device)
File "D:\Git-repos\stable-diffusion-webui\venv\lib\site-packages\accelerate\utils\modeling.py", line 147, in set_module_tensor_to_device
new_value = old_value.to(device)
RuntimeError: CUDA error: operation not supported
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

I run ComfyUI on Windows with torch 2.0 and I use a GTX 960M card.

-----
Questions \& suggestions:

* I have no knowledge of CUDA malloc; I just guess the above error is a hardware compatibility problem (correct me if I am wrong) because I am using an old graphic card 960M. Is there anyway ComfyUI can detect such incompatibility during launch and automatically disable this argument, just like enable lowvram when it knows the GPU has smal VRAM?

* If there is little we can do during launch, I suggest that ComfyUI tells user to manually disable it in the terminal when encounters this error at runtime (if possible), or at least warns users in the argument description of `--help`. I am requesting so because when I checked the argument list with the `--help` option, I saw:
```
--cuda-malloc Enable cudaMallocAsync (enabled by default for torch 2.0 and up).
```
The above description really looks like users should enable cuda malloc when torch 2.0 is installed; however, some users like me have to disable it even they have torch 2.0.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.