Comfy-Org / Comfy-Org/ComfyUI

Possible Cache corruption -- a LORA and checkpoint combo causes data type and VBAR errors in all subsequent (different) generations

Open
#14,952 2 comments 1 reaction 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Standard WAN 2.2 Workflow runs normally without errors

### Actual Behavior

WAN Workflow completes with a good result but spams tons of "bias must be" and vbar exceptions. Even after removing this LORA from the workflow, all subsequent generations will spam the same error. even generations using completely different workflows and checkpoints such as LTX. clearing models/cache seems to cause the errors to stop. untill the LORA is used again.

### Steps to Reproduce

I'm reporting this because using this specific lora with a specific checkpoint seems to cause corruption in the execution cache that stays around and effects subsequent generations even if they are using a completely different LORA, model and workflow.

I was able to reproduce it using a completely bare-bones WAN 2.2 workflow with --disable-all-custom-nodes

It only seems to occur when using this lora https://civitai.red/models/2773174/height-growth?modelVersionId=3122647 together with this model checkpoint https://civitai.red/models/2003153/wan22-remix-t2vandi2v?modelVersionId=2770795

Note I have never had any problems using this checkpoint before. and this LORA works fine with other checkpoints of WAN 2.2 ..the problem only happens when using them together.

I'm not really expecting a fix as it appears to be the fault of the LORA and the checkpoint combination, but I'm hoping someone can tell me what these errors mean and why they cause problems in subsequent generations even if I turn the LORA off.

### Debug Logs

```powershell
G:\ComfyUI_windows_portable_new>"G:\ComfyUI_windows_portable_new\python_embeded\python.exe" -s ComfyUI\main.py --windows-standalone-build --use-sage-attention --fast --disable-all-custom-nodes
[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
[INFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']}
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 24564 MB, total RAM 261991 MB
[INFO] pytorch version: 2.12.1+cu130
[INFO] Enabled fp16 accumulation.
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 4090 : native
[INFO] Using async weight offloading with 2 streams
[INFO] Enabled pinned memory 104796.0
[INFO] Using sage attention
aimdo: src-win/cuda-detour.c:38:INFO:aimdo_setup_hooks: installing 6 hooks
aimdo: src-win/cuda-detour.c:28:DEBUG:install_hook_entries: hooks successfully installed
aimdo: src-win/shmem-detect.c:80:INFO:comfy-aimdo WDDM adapter match: NVIDIA GeForce RTX 4090 runtime_luid=00000000:000126ad dxgi_luid=00000000:000126ad
aimdo: src/control.c:248:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 4090 (VRAM: 24563 MB)
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.13.11 (tags/v3.13.11:6278944, Dec 5 2025, 16:26:58) [MSC v.1944 64 bit (AMD64)]
[INFO] ComfyUI version: 0.27.0
[INFO] comfy-aimdo version: 0.4.10
[INFO] comfy-kitchen version: 0.2.16
[INFO] comfyui-frontend-package version: 1.45.20
[INFO] comfyui-workflow-templates version: 0.11.2
[INFO] comfyui-embedded-docs version: 0.5.7
[INFO] comfy-kitchen version: 0.2.16
[INFO] comfy-aimdo version: 0.4.10
[INFO] [Prompt Server] web root: G:\ComfyUI_windows_portable_new\python_embeded\Lib\site-packages\comfyui_frontend_package\static
[INFO] Asset seeder disabled
G:\ComfyUI_windows_portable_new\python_embeded\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[INFO] No OpenGL_accelerate module loaded: Acceleration disabled
[INFO] Skipping loading of custom nodes
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Using RAM pressure cache.
[INFO] Starting server

[INFO] To see the GUI go to: http://127.0.0.1:8188
[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] Requested to load CLIPVisionModelProjection
[INFO] Model CLIPVisionModelProjection prepared for dynamic VRAM loading. 1205MB Staged. 0 patches attached. Force pre-loaded 132 weights: 330 KB.
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load WanTEModel
[INFO] Model WanTEModel prepared for dynamic VRAM loading. 10835MB Staged. 0 patches attached. Force pre-loaded 73 weights: 488 KB.
[INFO] Model WanTEModel prepared for dynamic VRAM loading. 10835MB Staged. 0 patches attached. Force pre-loaded 73 weights: 488 KB.
[INFO] Requested to load WanVAE
[INFO] Model WanVAE prepared for dynamic VRAM loading. 241MB Staged. 0 patches attached. Force pre-loaded 60 weights: 61 KB.
[INFO] model weight dtype torch.float8_e4m3fn, manual cast: torch.float16
[INFO] model_type FLOW
[INFO] Requested to load WAN21
[INFO] Model WAN21 prepared for dynamic VRAM loading. 13626MB Staged. 813 patches attached. Force pre-loaded 240 weights: 1203 KB.
0%| | 0/2 [00:00

Contributor guide

Open the contributing guide

Research direction

Start at ComfyUI/main.py and reproduce the bare-bones WAN 2.2 workflow with --disable-all-custom-nodes, using the linked LoRA and checkpoint combination. Compare subsequent generations before and after clearing models/cache, and trace the repeated Float8_e4m3fn bias and VBAR errors. Done means determining whether execution or model-loading cache state survives between workflows and documenting a reproducible cause or boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.