[MultiGPU] [WARNING] WARNING, memory leak with model *SOME_TE_MODEL_*
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
- no mem leak
### Actual Behavior
- tons of WARNING memory leak
### Steps to Reproduce
- template ZIT T2I workflow
- Add `SelectModelDevice` / `SelectCLIPDevice` / `SelectVAEDevice` to GPU1
- Run the workflow
- Got warnings in console
The same applies to H3, CLIP to GPU1
### Debug Logs
```powershell
ETCH ComfyRegistry Data: 95/182
[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] Creating deepclone of AutoencodingEngine for cuda:1.
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] Requested to load ZImageTEModel_
[INFO] loaded completely; 7672.25 MB loaded, full load: True
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
[INFO] model weight dtype torch.bfloat16, manual cast: None
[INFO] model_type FLOW
FETCH ComfyRegistry Data: 100/182
[INFO] Creating deepclone of ZImageTEModel_ for cuda:1.
[INFO] Requested to load ZImageTEModel_
[INFO] loaded completely; 7672.25 MB loaded, full load: True
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load ZImageTEModel_
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
FETCH ComfyRegistry Data: 105/182
[INFO] loaded completely; 7672.25 MB loaded, full load: True
[INFO] Creating deepclone of Lumina2 for cuda:1.
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] model weight dtype torch.bfloat16, manual cast: None
[INFO] model_type FLOW
FETCH ComfyRegistry Data: 110/182
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load Lumina2
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] loaded completely; 11739.54 MB loaded, full load: True
25%|████████████████████████████████████▎ | 2/8 [00:01<00:04, 1.31it/s]FETCH ComfyRegistry Data: 115/182
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:06<00:00, 1.32it/s]
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load AutoencodingEngine
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] loaded completely; 159.87 MB loaded, full load: True
[INFO] Prompt executed in 37.20 seconds
FETCH ComfyRegistry Data: 120/182
FETCH ComfyRegistry Data: 125/182
FETCH ComfyRegistry Data: 130/182
[INFO] got prompt
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float16
[INFO] Found quantization metadata version 1
[INFO] Using MixedPrecisionOps for text encoder
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Found quantization metadata version 1
[INFO] Detected mixed precision quantization
[INFO] Using mixed precision operations
[INFO] Native ops: float8_e4m3fn, convrot_w4a4, mxfp8, float8_e5m2, nvfp4, asym_w4a8_int8, int8_tensorwise
[INFO] model weight dtype torch.bfloat16, manual cast: torch.bfloat16
[INFO] model_type FLOW_AV
[INFO] Creating deepclone of MiniMaxH3TEModel_ for cuda:1.
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Found quantization metadata version 1
[INFO] Using MixedPrecisionOps for text encoder
FETCH ComfyRegistry Data: 135/182
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load MiniMaxH3TEModel_
[INFO] Potential memory leak detected with model ZImageTEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] loaded completely; 14960.20 MB loaded, full load: True
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
[INFO] Potential memory leak detected with model MiniMaxH3TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model MiniMaxH3TEModel_. Please make sure it is not being referenced from somewhere.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load MiniMaxH3TEModel_
[INFO] Potential memory leak detected with model MiniMaxH3TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model MiniMaxH3TEModel_. Please make sure it is not being referenced from somewhere.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
FETCH ComfyRegistry Data: 140/182
[INFO] loaded completely; 14960.20 MB loaded, full load: True
FETCH ComfyRegistry Data: 145/182
[INFO] Potential memory leak detected with model MiniMaxH3TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model MiniMaxH3TEModel_. Please make sure it is not being referenced from somewhere.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load MiniMaxH3VideoVAE
[INFO] Potential memory leak detected with model MiniMaxH3TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model MiniMaxH3TEModel_. Please make sure it is not being referenced from somewhere.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] loaded completely; 4966.19 MB loaded, full load: True
[INFO] Potential memory leak detected with model MiniMaxH3TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model MiniMaxH3TEModel_. Please make sure it is not being referenced from somewhere.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
[INFO] Requested to load MiniMaxH3
[INFO] Potential memory leak detected with model MiniMaxH3TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
[WARNING] WARNING, memory leak with model MiniMaxH3TEModel_. Please make sure it is not being referenced from somewhere.
[WARNING] WARNING, memory leak with model ZImageTEModel_. Please make sure it is not being referenced from somewhere.
FETCH ComfyRegistry Data: 150/182
FETCH ComfyRegistry Data: 155/182
[INFO] loaded completely; 32429.07 MB loaded, full load: True
0%| | 0/8 [00:00
Contributor guide
Research direction
Reproduce the template ZIT T2I workflow with SelectModelDevice, SelectCLIPDevice, and SelectVAEDevice assigned to GPU1, then compare the model-loading and deepclone warnings in the debug log. Trace the MultiGPU model-loading path for ZImageTEModel_, MiniMaxH3TEModel_, and related models; done means the workflow runs without repeated memory-leak warnings or unintended retained model references.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100