torch.OutOfMemoryError 3 KSampler torch.OutOfMemoryError Allocation on device
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Expected Behavior
Create an image
* Everything is updated to the latest versions
* Even a reboot didn't help
* Didn't help: find . -type d -name '__pycache__' -exec rm -rf '{}' \; &>/dev/null
### Actual Behavior
torch.OutOfMemoryError 3 KSampler torch.OutOfMemoryError Allocation on device
### Steps to Reproduce
Can't attach workflow for privacy reasons.
### Debug Logs
```powershell
torch.OutOfMemoryError 3 KSampler
KSampler
Allocation on device
# ComfyUI Error Report
## Error Details
- **Node ID:** 3
- **Node Type:** KSampler
- **Exception Type:** torch.OutOfMemoryError
- **Exception Message:** Allocation on device
## Stack Trace
File "/srv/www/vhosts/ComfyUI_2024-11-06/execution.py", line 324, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "/srv/www/vhosts/ComfyUI_2024-11-06/execution.py", line 199, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "/srv/www/vhosts/ComfyUI_2024-11-06/execution.py", line 170, in _map_node_over_list
process_inputs(input_dict, i)
File "/srv/www/vhosts/ComfyUI_2024-11-06/execution.py", line 159, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "/srv/www/vhosts/ComfyUI_2024-11-06/nodes.py", line 1467, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/srv/www/vhosts/ComfyUI_2024-11-06/nodes.py", line 1434, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/sample.py", line 43, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 1020, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 918, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 904, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 873, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 857, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 714, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/srv/www/vhosts/ComfyUI_2024-11-06/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/k_diffusion/sampling.py", line 155, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 384, in __call__
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 839, in __call__
return self.predict_noise(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 842, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 364, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 200, in calc_cond_batch
return executor.execute(model, conds, x_in, timestep, model_options)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/samplers.py", line 313, in _calc_cond_batch
output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/model_base.py", line 128, in apply_model
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/model_base.py", line 157, in _apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "/srv/www/vhosts/ComfyUI_2024-11-06/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/ldm/flux/model.py", line 184, in forward
out = self.forward_orig(img, img_ids, context, txt_ids, timestep, y, guidance, control, transformer_options)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/ldm/flux/model.py", line 134, in forward_orig
img, txt = block(img=img, txt=txt, vec=vec, pe=pe)
File "/srv/www/vhosts/ComfyUI_2024-11-06/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/ldm/flux/layers.py", line 164, in forward
attn = attention(torch.cat((txt_q, img_q), dim=2),
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/ldm/flux/math.py", line 8, in attention
q, k = apply_rope(q, k, pe)
File "/srv/www/vhosts/ComfyUI_2024-11-06/comfy/ldm/flux/math.py", line 33, in apply_rope
xq_out = freqs_cis[..., 0] * xq_[..., 0] + freqs_cis[..., 1] * xq_[..., 1]
```
## System Information
- **ComfyUI Version:** v0.3.6-14-g9a616b8
- **Arguments:** main.py --listen 0.0.0.0 --verbose WARNING
- **OS:** posix
- **Python Version:** 3.10.12 (main, Nov 6 2024, 20:22:13) [GCC 11.4.0]
- **Embedded Python:** false
- **PyTorch Version:** 2.5.1+cu124
## Devices
- **Name:** cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
- **Type:** cuda
- **VRAM Total:** 25385500672
- **VRAM Free:** 24446566400
- **Torch VRAM Total:** 67108864
- **Torch VRAM Free:** 58589184
## Logs
```
2024-12-05T05:50:32.905540 -
2024-12-05T05:50:32.908330 -
[1;34m[LoRA Sidebar]:[0m Starting initial cache build...2024-12-05T05:50:32.908395 -
2024-12-05T05:50:32.908827 -
[1m[1;34m[LoRA Sidebar]:[0m Building LoRA cache[0m |[96m[94m▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒[0m| [92m 0.5%[0m (1/210)2024-12-05T05:50:32.908886 -
...
[1m[1;34m[LoRA Sidebar]:[0m Building LoRA cache[0m |[96m██████████████████████████████████████████████████[94m[0m| [92m100.0%[0m (210/210)2024-12-05T05:50:32.917624 - 2024-12-05T05:50:32.917630 -
[1;34m[LoRA Sidebar]:[0m [92mCache build complete![0m2024-12-05T05:50:32.917637 -
2024-12-05T05:50:32.924283 - [1;34m[LoRA Sidebar]:[0m LoRA cache built in[96m 0.02 seconds[0m2024-12-05T05:50:32.924296 -
2024-12-05T05:50:32.924306 - [1;34m[LoRA Sidebar]:[0m Total LoRAs processed:[1m 210[0m2024-12-05T05:50:32.924314 -
2024-12-05T05:50:32.924332 - [1;34m[LoRA Sidebar]:[0m [1m[93mNice collection! You're well on your way to LoRA mastery. 🎓[0m
2024-12-05T05:50:32.924340 -
2024-12-05T05:50:32.924469 -
FlowState Nodes
- System initializing...2024-12-05T05:50:32.924479 -
2024-12-05T05:50:32.924593 - - Loading custom nodes...2024-12-05T05:50:32.924603 -
2024-12-05T05:50:32.924697 - - Loaded Unified Video Sampler node.2024-12-05T05:50:32.924707 -
2024-12-05T05:50:32.925018 - - Setting UNET_LIST_PATH to: at 0x7f0108510f70>2024-12-05T05:50:32.925029 -
2024-12-05T05:50:32.926040 - - Loaded Unified Sampler.2024-12-05T05:50:32.926053 -
2024-12-05T05:50:32.926228 - - Loaded Unified Prompt & Prompt Output nodes.2024-12-05T05:50:32.926239 -
2024-12-05T05:50:32.926319 - - NO LLAMA CUDA2024-12-05T05:50:32.926328 -
2024-12-05T05:50:32.926382 - - NO LLAMA OR CUDA2024-12-05T05:50:32.926390 -
2024-12-05T05:50:32.926568 - - Load Unified Model Loader node.2024-12-05T05:50:32.926579 -
2024-12-05T05:50:32.964438 - - Loaded Latent Chooser node.2024-12-05T05:50:32.964461 -
2024-12-05T05:50:32.964697 - - Loaded Unified Styler node.2024-12-05T05:50:32.964717 -
2024-12-05T05:50:32.965043 - - Nodes Loaded.2024-12-05T05:50:32.965066 -
2024-12-05T05:50:32.965095 - - Loading node name mappings...2024-12-05T05:50:32.965105 -
2024-12-05T05:50:32.965115 - - Mappings Loaded. Available nodes:2024-12-05T05:50:32.965125 -
2024-12-05T05:50:32.965137 - - FlowStateUnifiedVideoSampler: 2024-12-05T05:50:32.965159 -
2024-12-05T05:50:32.965169 - - FlowStateUnifiedSampler: 2024-12-05T05:50:32.965178 -
2024-12-05T05:50:32.965188 - - FlowStateUnifiedModelLoader: 2024-12-05T05:50:32.965197 -
2024-12-05T05:50:32.965207 - - FlowStateUnifiedPrompt: 2024-12-05T05:50:32.965216 -
2024-12-05T05:50:32.965226 - - FlowStatePromptOutput: 2024-12-05T05:50:32.965234 -
2024-12-05T05:50:32.965244 - - FlowStateLatentChooser: 2024-12-05T05:50:32.965253 -
2024-12-05T05:50:32.965262 - - FlowStateUnifiedStyler: 2024-12-05T05:50:32.965271 -
2024-12-05T05:50:33.074927 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json2024-12-05T05:50:33.074959 -
2024-12-05T05:50:33.080831 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json2024-12-05T05:50:33.080862 -
2024-12-05T05:50:33.091616 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json2024-12-05T05:50:33.091649 -
2024-12-05T05:50:33.093389 -
2024-12-05T05:50:33.093418 - [92m[rgthree-comfy] Loaded 42 exciting nodes. 🎉[00m2024-12-05T05:50:33.093429 -
2024-12-05T05:50:33.093437 -
2024-12-05T05:50:33.121877 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json2024-12-05T05:50:33.121906 -
2024-12-05T05:50:33.135267 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json2024-12-05T05:50:33.135281 -
2024-12-05T05:50:53.868629 -
FlowState Unified Model Loader
- Preparing loader
2024-12-05T05:50:53.868671 -
2024-12-05T05:50:58.161458 - clip missing: ['text_projection.weight']
2024-12-05T05:50:58.341246 -
FlowState Unified Model Loader - Loading complete.
- Model Name: acornIsSpinningFLUX_aisf11H8stpChinfx.safetensors
- VAE Name: flux-vae.safetensors
- CLIP Name: t5xxl_fp8_e4m3fn.safetensors & clip_l.safetensors
- Loading Time: 0m 4s
2024-12-05T05:50:58.341279 -
2024-12-05T05:51:01.251668 -
0%| | 0/30 [00:00
Contributor guide
Assessment
This issue has not been assessed yet.