Comfy-Org / Comfy-Org/ComfyUI

VAEDecode error

Open
#5,828 4 comments 0 reactions 0 assignees View on GitHub
AMD Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Expected Behavior

Generate image

### Actual Behavior

After 15-20 generation an error appears
[enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 5242880 bytes.
reload ComfyUi doesn`t help, only reboot PC

### Steps to Reproduce

Try to generate image after 15-20 generation

### Debug Logs

```powershell
# ComfyUI Error Report
## Error Details
- **Node ID:** 8
- **Node Type:** VAEDecode
- **Exception Type:** RuntimeError
- **Exception Message:** [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 5242880 bytes.
## Stack Trace

File "F:\StableD\Stav\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

File "F:\StableD\Stav\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

File "F:\StableD\Stav\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)

File "F:\StableD\Stav\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))

File "F:\StableD\Stav\ComfyUI\nodes.py", line 284, in decode
images = vae.decode(samples["samples"])

File "F:\StableD\Stav\ComfyUI\comfy\sd.py", line 354, in decode
model_management.load_models_gpu([self.patcher], memory_required=memory_used)

File "F:\StableD\Stav\ComfyUI\comfy\model_management.py", line 525, in load_models_gpu
free_memory(total_memory_required[device] * 1.1 + extra_mem, device, models_already_loaded)

File "F:\StableD\Stav\ComfyUI\comfy\model_management.py", line 441, in free_memory
if current_loaded_models[i].model_unload(memory_to_free):

File "F:\StableD\Stav\ComfyUI\comfy\model_management.py", line 347, in model_unload
self.model.unpatch_model(self.model.offload_device, unpatch_weights=unpatch_weights)

File "F:\StableD\Stav\ComfyUI\comfy\model_patcher.py", line 504, in unpatch_model
self.model.to(device_to)

File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 989, in to
return self._apply(convert)

File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)

File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)

File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)

[Previous line repeated 5 more times]

File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 664, in _apply
param_applied = fn(param)

File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

```
## System Information
- **ComfyUI Version:** v0.3.4-5-gb4526d3
- **Arguments:** main.py
- **OS:** nt
- **Python Version:** 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
- **Embedded Python:** false
- **PyTorch Version:** 1.13.1+cu117
## Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 2060 SUPER :
- **Type:** cuda
- **VRAM Total:** 8589475840
- **VRAM Free:** 7259333632
- **Torch VRAM Total:** 295698432
- **Torch VRAM Free:** 95462400

## Logs
```
2024-11-28T19:47:13.973058 - Total VRAM 8192 MB, total RAM 32694 MB
2024-11-28T19:47:13.973058 - pytorch version: 1.13.1+cu117
2024-11-28T19:47:13.974058 - Set vram state to: NORMAL_VRAM
2024-11-28T19:47:13.974058 - Device: cuda:0 NVIDIA GeForce RTX 2060 SUPER :
2024-11-28T19:47:14.289432 - Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
2024-11-28T19:47:15.505536 - [Prompt Server] web root: F:\StableD\Stav\ComfyUI\web
2024-11-28T19:47:16.711632 - ComfyUI-FreeMemory nodes loaded: Image, Latent, Model, and CLIP versions available.2024-11-28T19:47:16.711632 -
2024-11-28T19:47:16.713633 -
Import times for custom nodes:
2024-11-28T19:47:16.714634 - 0.0 seconds: F:\StableD\Stav\ComfyUI\custom_nodes\websocket_image_save.py
2024-11-28T19:47:16.714634 - 0.0 seconds: F:\StableD\Stav\ComfyUI\custom_nodes\ComfyUI-Unload-Model
2024-11-28T19:47:16.714634 - 0.0 seconds: F:\StableD\Stav\ComfyUI\custom_nodes\ComfyUI-FreeMemory
2024-11-28T19:47:16.714634 -
2024-11-28T19:47:16.800713 - Starting server

2024-11-28T19:47:16.800713 - To see the GUI go to: http://127.0.0.1:8188
2024-11-28T19:47:41.025595 - got prompt
2024-11-28T19:47:41.055621 - Using split attention in VAE
2024-11-28T19:47:41.058625 - Using split attention in VAE
2024-11-28T19:47:41.494980 - model weight dtype torch.float16, manual cast: None
2024-11-28T19:47:41.497266 - model_type V_PREDICTION
2024-11-28T19:47:45.275800 - Using split attention in VAE
2024-11-28T19:47:45.277801 - Using split attention in VAE
2024-11-28T19:47:45.637128 - Requested to load SDXLClipModel
2024-11-28T19:47:45.637128 - Loading 1 new model
2024-11-28T19:47:45.658147 - loaded completely 0.0 1560.802734375 True
2024-11-28T19:47:48.518699 - Attempting to free GPU VRAM and system RAM...2024-11-28T19:47:48.518699 -
2024-11-28T19:47:48.519700 - GPU VRAM: Initial usage: 1.64 GB, Final usage: 1.64 GB, Freed: 0.00 GB2024-11-28T19:47:48.519700 -
2024-11-28T19:47:48.615787 - Garbage collector: collected 224 objects.2024-11-28T19:47:48.615787 -
2024-11-28T19:47:48.624795 - System RAM: Initial usage: 48.70%, Final usage: 48.70%, Freed: 0.00%2024-11-28T19:47:48.624795 -
2024-11-28T19:47:50.312241 - Attempting to free GPU VRAM and system RAM...2024-11-28T19:47:50.312241 -
2024-11-28T19:47:50.315245 - GPU VRAM: Initial usage: 1.64 GB, Final usage: 1.64 GB, Freed: 0.00 GB2024-11-28T19:47:50.315245 -
2024-11-28T19:47:50.414333 - Garbage collector: collected 543 objects.2024-11-28T19:47:50.414333 -
2024-11-28T19:47:50.425343 - System RAM: Initial usage: 52.40%, Final usage: 52.40%, Freed: 0.00%2024-11-28T19:47:50.425343 -
2024-11-28T19:47:50.430347 - Requested to load SDXL
2024-11-28T19:47:50.430347 - Loading 1 new model
2024-11-28T19:47:52.405142 - loaded partially 4590.63951171875 4590.6384353637695 0
2024-11-28T19:48:21.542241 -
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:29<00:00, 1.03s/it]2024-11-28T19:48:21.542241 -
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:29<00:00, 1.04s/it]2024-11-28T19:48:21.542241 -
2024-11-28T19:48:21.544243 - Attempting to free GPU VRAM and system RAM...2024-11-28T19:48:21.544243 -
2024-11-28T19:48:21.549249 - GPU VRAM: Initial usage: 4.88 GB, Final usage: 4.88 GB, Freed: 0.00 GB2024-11-28T19:48:21.549249 -
2024-11-28T19:48:21.645335 - Garbage collector: collected 2560 objects.2024-11-28T19:48:21.645335 -
2024-11-28T19:48:21.654344 - System RAM: Initial usage: 44.30%, Final usage: 44.30%, Freed: 0.00%2024-11-28T19:48:21.654344 -
2024-11-28T19:48:21.654344 - Requested to load AutoencoderKL
2024-11-28T19:48:21.654344 - Loading 1 new model
2024-11-28T19:48:22.789387 - !!! Exception during processing !!! [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 5242880 bytes.
2024-11-28T19:48:22.792390 - Traceback (most recent call last):
File "F:\StableD\Stav\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "F:\StableD\Stav\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "F:\StableD\Stav\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "F:\StableD\Stav\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "F:\StableD\Stav\ComfyUI\nodes.py", line 284, in decode
images = vae.decode(samples["samples"])
File "F:\StableD\Stav\ComfyUI\comfy\sd.py", line 354, in decode
model_management.load_models_gpu([self.patcher], memory_required=memory_used)
File "F:\StableD\Stav\ComfyUI\comfy\model_management.py", line 525, in load_models_gpu
free_memory(total_memory_required[device] * 1.1 + extra_mem, device, models_already_loaded)
File "F:\StableD\Stav\ComfyUI\comfy\model_management.py", line 441, in free_memory
if current_loaded_models[i].model_unload(memory_to_free):
File "F:\StableD\Stav\ComfyUI\comfy\model_management.py", line 347, in model_unload
self.model.unpatch_model(self.model.offload_device, unpatch_weights=unpatch_weights)
File "F:\StableD\Stav\ComfyUI\comfy\model_patcher.py", line 504, in unpatch_model
self.model.to(device_to)
File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 989, in to
return self._apply(convert)
File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
[Previous line repeated 5 more times]
File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 664, in _apply
param_applied = fn(param)
File "C:\Users\Rihnami\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 5242880 bytes.

2024-11-28T19:48:22.794391 - Prompt executed in 41.76 seconds

```
## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.
```
{"last_node_id":30,"last_link_id":39,"nodes":[{"id":22,"type":"FreeMemoryImage","pos":[1385,29],"size":[315,58],"flags":{},"order":11,"mode":0,"inputs":[{"name":"image","type":"IMAGE","link":17}],"outputs":[{"name":"IMAGE","type":"IMAGE","links":[18],"slot_index":0}],"properties":{"Node name for S&R":"FreeMemoryImage"},"widgets_values":[false]},{"id":9,"type":"SaveImage","pos":[1518,322],"size":[210,270],"flags":{},"order":12,"mode":0,"inputs":[{"name":"images","type":"IMAGE","link":18}],"outputs":[],"properties":{},"widgets_values":["ComfyUI"]},{"id":7,"type":"CLIPTextEncode","pos":[253,372],"size":[425.27801513671875,180.6060791015625],"flags":{},"order":7,"mode":0,"inputs":[{"name":"clip","type":"CLIP","link":36}],"outputs":[{"name":"CONDITIONING","type":"CONDITIONING","links":[13],"slot_index":0}],"properties":{"Node name for S&R":"CLIPTextEncode"},"widgets_values":["((furry):1.9),((wattermarks, artist name:1.9, text:1.9)), lowres,bad anatomy,blurry,(worst quality:1.8),low quality,hands bad,(normal quality:1.3),bad hands,mutated hands and fingers,extra legs,extra arms,duplicate,cropped,jpeg,artifacts,long body,multiple breasts,mutated,disfigured,bad proportions,bad feet,ugly,text font ui,missing limb,monochrome,face bad"]},{"id":26,"type":"EmptyLatentImage","pos":[199,740],"size":[315,106],"flags":{},"order":0,"mode":0,"inputs":[],"outputs":[{"name":"LATENT","type":"LATENT","links":[],"slot_index":0}],"properties":{"Node name for S&R":"EmptyLatentImage"},"widgets_values":[768,1344,1]},{"id":28,"type":"FreeMemoryModel","pos":[-115,-3],"size":[315,58],"flags":{},"order":4,"mode":0,"inputs":[{"name":"model","type":"MODEL","link":31}],"outputs":[{"name":"MODEL","type":"MODEL","links":[33],"slot_index":0}],"properties":{"Node name for S&R":"FreeMemoryModel"},"widgets_values":[false]},{"id":4,"type":"CheckpointLoaderSimple","pos":[-192,209],"size":[315,98],"flags":{},"order":1,"mode":0,"inputs":[],"outputs":[{"name":"MODEL","type":"MODEL","links":[31],"slot_index":0},{"name":"CLIP","type":"CLIP","links":[34],"slot_index":1},{"name":"VAE","type":"VAE","links":[],"slot_index":2}],"properties":{"Node name for S&R":"CheckpointLoaderSimple"},"widgets_values":["noobaiXLNAIXL_vPred065SVersion.safetensors"]},{"id":29,"type":"FreeMemoryCLIP","pos":[-157,365],"size":[315,58],"flags":{},"order":5,"mode":0,"inputs":[{"name":"clip","type":"CLIP","link":34}],"outputs":[{"name":"CLIP","type":"CLIP","links":[35,36],"slot_index":0}],"properties":{"Node name for S&R":"FreeMemoryCLIP"},"widgets_values":[false]},{"id":23,"type":"FreeMemoryLatent","pos":[922.8046264648438,82.6084976196289],"size":[315,58],"flags":{},"order":9,"mode":0,"inputs":[{"name":"latent","type":"LATENT","link":38}],"outputs":[{"name":"LATENT","type":"LATENT","links":[23],"slot_index":0}],"properties":{"Node name for S&R":"FreeMemoryLatent"},"widgets_values":[false]},{"id":13,"type":"KSamplerAdvanced","pos":[906,328],"size":[393.9046325683594,468.70849609375],"flags":{},"order":8,"mode":0,"inputs":[{"name":"model","type":"MODEL","link":33},{"name":"positive","type":"CONDITIONING","link":12},{"name":"negative","type":"CONDITIONING","link":13},{"name":"latent_image","type":"LATENT","link":39}],"outputs":[{"name":"LATENT","type":"LATENT","links":[38],"slot_index":0}],"properties":{"Node name for S&R":"KSamplerAdvanced"},"widgets_values":["enable",529913162466544,"randomize",28,5.5,"euler","normal",0,10000,"disable"]},{"id":5,"type":"EmptyLatentImage","pos":[224,591],"size":[315,106],"flags":{},"order":2,"mode":0,"inputs":[],"outputs":[{"name":"LATENT","type":"LATENT","links":[39],"slot_index":0}],"properties":{"Node name for S&R":"EmptyLatentImage"},"widgets_values":[1344,768,1]},{"id":10,"type":"VAELoader","pos":[-183,488],"size":[315,58],"flags":{"collapsed":false},"order":3,"mode":0,"inputs":[],"outputs":[{"name":"VAE","type":"VAE","links":[37],"slot_index":0}],"properties":{"Node name for S&R":"VAELoader"},"widgets_values":["sdxl_vae.safetensors"]},{"id":8,"type":"VAEDecode","pos":[1264,197],"size":[210,46],"flags":{},"order":10,"mode":0,"inputs":[{"name":"samples","type":"LATENT","link":23},{"name":"vae","type":"VAE","link":37}],"outputs":[{"name":"IMAGE","type":"IMAGE","links":[17],"slot_index":0}],"properties":{"Node name for S&R":"VAEDecode"},"widgets_values":[]},{"id":6,"type":"CLIPTextEncode","pos":[230,-9],"size":[493.70452880859375,321.2085266113281],"flags":{},"order":6,"mode":0,"inputs":[{"name":"clip","type":"CLIP","link":35}],"outputs":[{"name":"CONDITIONING","type":"CONDITIONING","links":[12],"slot_index":0}],"properties":{"Node name for S&R":"CLIPTextEncode"},"widgets_values":["masterpiece, ultra detailed, (sketch:0.9), detailed background, ([artist:personal ami]:0.2), ([artist:greenu]:0.9), 1girl, shiro\\(no game no life\\)"]}],"links":[[12,6,0,13,1,"CONDITIONING"],[13,7,0,13,2,"CONDITIONING"],[17,8,0,22,0,"IMAGE"],[18,22,0,9,0,"IMAGE"],[23,23,0,8,0,"LATENT"],[31,4,0,28,0,"MODEL"],[33,28,0,13,0,"MODEL"],[34,4,1,29,0,"CLIP"],[35,29,0,6,0,"CLIP"],[36,29,0,7,0,"CLIP"],[37,10,0,8,1,"VAE"],[38,13,0,23,0,"LATENT"],[39,5,0,13,3,"LATENT"]],"groups":[],"config":{},"extra":{"ds":{"scale":0.9090909090909091,"offset":[253.72065351549537,96.08495530366899]}},"version":0.4}
```

## Additional Context
(Please add any additional context or steps to reproduce the error here)
```

### Other

PC spec:
Ryzen 7 2700x
RTX 2060 Super (8Gb VRAM)
32 Gb RAM

start with arguments --normalvram and --lowvram dosen`t solved this problem

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.