Bug Report: Severe Performance Degradation Over Multiple Runs
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
After running multiple prompts, the performance of the program degrades significantly. Each execution becomes slower than the previous one, eventually reaching unusable speeds.
### Actual Behavior
Execution times increase with every run:
- First run: ~138 seconds
- Later run: ~428 seconds
- Later run: ~543 seconds
Eventually, processing gets interrupted.
### Steps to Reproduce
1. Start the program with the default configuration.
2. Run several prompts in sequence (see attached logs).
3. Observe that execution times increase dramatically with each run.
### Debug Logs
```powershell
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Requested to load QwenImageTEModel_
loaded partially 9588.8 9587.04833984375 0
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Requested to load QwenImageTEModel_
loaded partially 9570.675 9570.57275390625 0
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
Requested to load QwenImage
loaded partially 9214.834965667724 9212.660705566406 645
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [01:58<00:00, 14.75s/it]
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Prompt executed in 00:10:25
got prompt
Requested to load QwenImage
loaded partially 9214.834965667724 9212.660705566406 645
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [01:49<00:00, 13.69s/it]
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Prompt executed in 138.36 seconds
got prompt
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Requested to load QwenImageTEModel_
loaded partially 9570.675 9570.57275390625 0
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Requested to load QwenImageTEModel_
loaded partially 9570.675 9570.57275390625 0
Requested to load QwenImage
loaded partially 9214.834965667724 9212.660705566406 645
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [04:45<00:00, 35.67s/it]
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Prompt executed in 428.50 seconds
got prompt
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Requested to load QwenImageTEModel_
loaded partially 9570.675 9570.57275390625 0
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Requested to load QwenImageTEModel_
loaded partially 9570.675 9570.57275390625 0
Requested to load QwenImage
loaded partially 9214.834965667724 9212.660705566406 645
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [06:30<00:00, 48.84s/it]
Requested to load WanVAE
0 models unloaded.
loaded partially 128.0 127.9998779296875 0
Prompt executed in 543.34 seconds
got prompt
Requested to load QwenImage
loaded partially 9214.834965667724 9212.660705566406 645
0%| | 0/8 [01:02
Contributor guide
Assessment
This issue has not been assessed yet.