Comfy-Org / Comfy-Org/ComfyUI

Corruption of in-memory copy of diffusion model when using Dynamic VRAM and loras on Windows

Open
#13,234 14 comments 1 reaction 1 assignee Claimed by @rattus128 View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

The attached workflow does not perform anything useful. It merely runs the high noise and low noise WAN2.2 models in succession many times. It is expected that the models do not get corrupted.

When the models get corrupted it affects the results in random ways.

### Actual Behavior

In practice if we compare the file from which we loaded the model .../app/models/diffusion_models/wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors with a separate backup copy we find after the workflow has run about halfway through or so that the files now differ with usually single bit differences and almost always occuring in the tensors for ffn blocks.

You can do this comparison with "cmp -b" but I have attached a short C program (comparelargefiles.txt as .c is a problem for the upload) which runs a bit faster, reports all byte differences and looks up the offset in the json header to see which tensor is affected. A typical output when there are problems is like this:

```
comparelargefiles wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors /t/safetybackup/wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors
N=180728 N+8=180736
Offset 3293644821 3293464085: file1=0x62 file2=0xE2 set=0x00 cleared=0x80
"blocks.17.ffn.0.weight":{"dtype":"F8_E4M3","shape":[13824,5120],"data_offsets":[3277632816,3348411696]}
Offset 3293644853 3293464117: file1=0x5C file2=0xDC set=0x00 cleared=0x80
"blocks.17.ffn.0.weight":{"dtype":"F8_E4M3","shape":[13824,5120],"data_offsets":[3277632816,3348411696]}
Total differing bytes: 2
```

Typically only a single bit in a byte is affected when they differ but sometimes two bits can be affected. The offset of the affected bytes and the bit within the byte are different for every run but almost always in a ffn block.

This does not occur if --disable-dynamic-vram is used.
This does not occur if _all_ loras are bypassed. It does occur if only some loras are bypassed.
The high noise model is more often affected than the low noise model.
The problem almost always occurs in a ffn block.

The mismatches disappear if you run a big job which consumes RAM and flushes the memory-mapped cache of the file. The files on disk are not affected.

Obviously this requires that the files remain in the memory cache and are not repeatedly fetched back from disk. I have 16GB VRAM and 64GB RAM and for these models they can just fit into available RAM ok between uses.

Is this bad RAM? I have run memtest86 with multiple passes and no fails. I have shortened TrefI (the refresh interval) in case it was due to a refresh issue. I have increased the fans to keep the RAM below 40C and the problem still occurs. I believe the RAM is fine.

Workflow [memorycorruption.json](https://github.com/user-attachments/files/26358919/memorycorruption.json)
Comparison program (or just use cmp -b) [comparelargefiles.txt](https://github.com/user-attachments/files/26358971/comparelargefiles.txt)

### Steps to Reproduce

Make sure the models directory file and the backup file are identical.
Run the attached workflow. Might need to run twice but usually faults start to occur half-way through first run.
During or after the workflow runs, compare the two files again and observe differences if loras are enabled.

### Debug Logs

```powershell
Microsoft Windows [Version 10.0.26200.8037]
(c) Microsoft Corporation. All rights reserved.

M:\pinokio\api\comfy.git\app>conda_hook & conda deactivate & conda deactivate & conda deactivate & conda activate base & M:\pinokio\api\comfy.git\app\env\Scripts\activate M:\pinokio\api\comfy.git\app\env && python main.py --cache-none --use-sage-attention --disable-cuda-malloc --output-directory M:/side/output --input-directory M:/side/input --temp-directory M:/side/temp
Setting output directory to: M:\side\output
Setting input directory to: M:\side\input
[START] Security scan
[ComfyUI-Manager] `python -m pip` not available. Falling back to `uv`.
[ComfyUI-Manager] Using `uv` as Python module for pip operations.
Using Python 3.13.11 environment at: M:\pinokio\api\experiment\app\env
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2026-03-30 19:55:59.638
** Platform: Windows
** Python version: 3.13.11 (main, Dec 17 2025, 21:09:15) [MSC v.1944 64 bit (AMD64)]
** Python executable: M:\pinokio\api\experiment\app\env\Scripts\python.exe
** ComfyUI Path: M:\pinokio\api\comfy.git\app
** ComfyUI Base Folder Path: M:\pinokio\api\comfy.git\app
** User directory: M:\pinokio\api\comfy.git\app\user
** ComfyUI-Manager config path: M:\pinokio\api\comfy.git\app\user\__manager\config.ini
** Log path: M:\pinokio\api\comfy.git\app\user\comfyui.log
Using Python 3.13.11 environment at: M:\pinokio\api\experiment\app\env
Using Python 3.13.11 environment at: M:\pinokio\api\experiment\app\env

Prestartup times for custom nodes:
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui-easy-use
0.7 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI-Manager

Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']}
Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4']}
Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_nvfp4']}
Checkpoint files will always be loaded safely.
Total VRAM 16376 MB, total RAM 65129 MB
pytorch version: 2.10.0+cu130
xformers version: 0.0.35
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4070 Ti SUPER : native
Using async weight offloading with 2 streams
Enabled pinned memory 29308.0
working around nvidia conv3d memory bug.
Using sage attention
aimdo: src-win/cuda-detour.c:77:INFO:aimdo_setup_hooks: found driver at 00007FF95C8A0000, installing 4 hooks
aimdo: src-win/cuda-detour.c:61:DEBUG:install_hook_entrys: hooks successfully installed
aimdo: src/control.c:69:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 4070 Ti SUPER (VRAM: 16375 MB)
DynamicVRAM support detected and enabled
Python version: 3.13.11 (main, Dec 17 2025, 21:09:15) [MSC v.1944 64 bit (AMD64)]
ComfyUI version: 0.18.1
comfy-aimdo version: 0.2.13.dev1
comfy-kitchen version: 0.2.8
Setting temp directory to: M:\side\temp\temp
ComfyUI frontend version: 1.42.8
[Prompt Server] web root: M:\pinokio\api\experiment\app\env\Lib\site-packages\comfyui_frontend_package\static
Asset seeder disabled
[ComfyUI-Easy-Use] server: v1.3.4 Loaded
[ComfyUI-Easy-Use] web root: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui-easy-use\web_version/v2 Loaded
ComfyUI-GGUF: Allowing full torch compile
### Loading: ComfyUI-Manager (V3.39)
[ComfyUI-Manager] network_mode: public
[ComfyUI-Manager] ComfyUI per-queue preview override detected (PR #11261). Manager's preview method feature is disabled. Use ComfyUI's --preview-method CLI option or 'Settings > Execution > Live preview method'.
### ComfyUI Version: v0.18.1-13-gb53b10ea | Released on '2026-03-24'
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
'sox' is not recognized as an internal or external command,
operable program or batch file.
SoX could not be found!

If you do not have SoX, proceed here:
- - - http://sox.sourceforge.net/ - - -

If you do (or think that you should) have SoX, double-check your
path variables.

✅ ComfyUI-Qwen-TTS v1.0.6 loaded

Import times for custom nodes:
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\websocket_image_save.py
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\extrastuff
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI_JPS-Nodes
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\ControlFlowUtils
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI-GGUF
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI-Notifications
0.0 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui-kjnodes
0.1 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI-Manager
0.2 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui_nvidia_rtx_nodes
0.2 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui-easy-dotsocr
0.3 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\qwen3-tts-comfyui
0.4 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui-videohelpersuite
2.7 seconds: M:\pinokio\api\comfy.git\app\custom_nodes\comfyui-easy-use

Context impl SQLiteImpl.
Will assume non-transactional DDL.
Disabling intermediate node cache.
Starting server

To see the GUI go to: http://127.0.0.1:8188

===================================================
# input.event
[
"To see the GUI go to: http://127.0.0.1:8188",
"http://127.0.0.1:8188"
]
===================================================

===================================================
# local variables
{
"url": "http://127.0.0.1:8188"
}
===================================================

[DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[DEPRECATION WARNING] Detected import of deprecated legacy API: /extensions/core/groupNode.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[DEPRECATION WARNING] Detected import of deprecated legacy API: /extensions/core/widgetInputs.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/buttonGroup.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/button.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
FETCH ComfyRegistry Data: 5/135
FETCH ComfyRegistry Data: 10/135
FETCH ComfyRegistry Data: 15/135
FETCH ComfyRegistry Data: 20/135
got prompt
Using xformers attention in VAE
Using xformers attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load WanVAE
Model WanVAE prepared for dynamic VRAM loading. 242MB Staged. 0 patches attached. Force pre-loaded 52 weights: 28 KB.
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load WanTEModel
Model WanTEModel prepared for dynamic VRAM loading. 6419MB Staged. 0 patches attached. Force pre-loaded 49 weights: 392 KB.
FETCH ComfyRegistry Data: 25/135
FETCH ComfyRegistry Data: 30/135
Model WanTEModel prepared for dynamic VRAM loading. 6419MB Staged. 0 patches attached. Force pre-loaded 49 weights: 392 KB.
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.float16, manual cast: torch.float16
model_type FLOW
Using sage attention mode: auto
Requested to load WAN21
Model WAN21 prepared for dynamic VRAM loading. 13631MB Staged. 400 patches attached.
0%| | 0/3 [00:00

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.