[Bug] Disabling async offload breaks Krea2 generation (black or broken images)
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
Using the Krea2 Turbo template workflow with bf16 models
[krea2_turbo_t2i_async_offload-test.json](https://github.com/user-attachments/files/29752693/krea2_turbo_t2i_async_offload-test.json)
Generation with async offload enabled:
Debug logs:
``` powershell
E:\ComfyUI>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --disable-all-custom-nodes
[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
[INFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']}
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 8192 MB, total RAM 65277 MB
[INFO] pytorch version: 2.12.0+cu130
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 3070 Ti Laptop GPU : cudaMallocAsync
[INFO] Using async weight offloading with 2 streams
[INFO] Enabled pinned memory 26110.0
[INFO] Using pytorch attention
aimdo: src-win/cuda-detour.c:38:INFO:aimdo_setup_hooks: installing 6 hooks
aimdo: src-win/cuda-detour.c:28:DEBUG:install_hook_entries: hooks successfully installed
aimdo: src-win/shmem-detect.c:80:INFO:comfy-aimdo WDDM adapter match: NVIDIA GeForce RTX 3070 Ti Laptop GPU runtime_luid=00000000:00011fff dxgi_luid=00000000:00011fff
aimdo: src/control.c:248:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 3070 Ti Laptop GPU (VRAM: 8191 MB)
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.12.7 (tags/v3.12.7:0b05ead, Oct 1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)]
[INFO] ComfyUI version: 0.27.0
[INFO] comfy-aimdo version: 0.4.10
[INFO] comfy-kitchen version: 0.2.16
[INFO] comfyui-frontend-package version: 1.45.20
[INFO] comfyui-workflow-templates version: 0.11.2
[INFO] comfyui-embedded-docs version: 0.5.7
[INFO] comfy-kitchen version: 0.2.16
[INFO] comfy-aimdo version: 0.4.10
[INFO] [Prompt Server] web root: E:\ComfyUI\python_embeded\Lib\site-packages\comfyui_frontend_package\static
[INFO] Asset seeder disabled
[INFO] No OpenGL_accelerate module loaded: Acceleration disabled
[INFO] Skipping loading of custom nodes
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Using RAM pressure cache.
[INFO] Starting server
[INFO] To see the GUI go to: http://127.0.0.1:8188
[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load Krea2TEModel_
[INFO] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[INFO] model weight dtype torch.bfloat16, manual cast: None
[INFO] model_type FLUX
[INFO] Requested to load Krea2
[INFO] 0 models unloaded.
[INFO] Model Krea2 prepared for dynamic VRAM loading. 24449MB Staged. 0 patches attached. Force pre-loaded 160 weights: 2824 KB.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:25<00:00, 3.24s/it]
[INFO] Requested to load WanVAE
[INFO] 0 models unloaded.
[INFO] Model WanVAE prepared for dynamic VRAM loading. 241MB Staged. 0 patches attached. Force pre-loaded 60 weights: 61 KB.
[INFO] Prompt executed in 38.88 seconds
```
### Actual Behavior
If you start ComfyUI with `--disable-async-offload` Krea2 generates black or in some cases broken images.
Same generation as above:
### Steps to Reproduce
- Add `--disable-async-offload` flag
- Run the Krea2 template workflow
- It generates a black or broken image
### Debug Logs
```powershell
E:\ComfyUI>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --disable-all-custom-nodes --disable-async-offload
[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
[INFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 8192 MB, total RAM 65277 MB
[INFO] pytorch version: 2.12.0+cu130
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 3070 Ti Laptop GPU : cudaMallocAsync
[INFO] Enabled pinned memory 26110.0
[INFO] Using pytorch attention
aimdo: src-win/cuda-detour.c:38:INFO:aimdo_setup_hooks: installing 6 hooks
aimdo: src-win/cuda-detour.c:28:DEBUG:install_hook_entries: hooks successfully installed
aimdo: src-win/shmem-detect.c:80:INFO:comfy-aimdo WDDM adapter match: NVIDIA GeForce RTX 3070 Ti Laptop GPU runtime_luid=00000000:00011fff dxgi_luid=00000000:00011fff
aimdo: src/control.c:248:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 3070 Ti Laptop GPU (VRAM: 8191 MB)
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.12.7 (tags/v3.12.7:0b05ead, Oct 1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)]
[INFO] ComfyUI version: 0.27.0
[INFO] comfy-aimdo version: 0.4.10
[INFO] comfy-kitchen version: 0.2.16
[INFO] comfyui-frontend-package version: 1.45.20
[INFO] comfyui-workflow-templates version: 0.11.2
[INFO] comfyui-embedded-docs version: 0.5.7
[INFO] comfy-kitchen version: 0.2.16
[INFO] comfy-aimdo version: 0.4.10
[INFO] [Prompt Server] web root: E:\ComfyUI\python_embeded\Lib\site-packages\comfyui_frontend_package\static
[INFO] Asset seeder disabled
[INFO] No OpenGL_accelerate module loaded: Acceleration disabled
[INFO] Skipping loading of custom nodes
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Using RAM pressure cache.
[INFO] Starting server
[INFO] To see the GUI go to: http://127.0.0.1:8188
[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load Krea2TEModel_
[INFO] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[INFO] model weight dtype torch.bfloat16, manual cast: None
[INFO] model_type FLUX
[INFO] Requested to load Krea2
[INFO] 0 models unloaded.
[INFO] Model Krea2 prepared for dynamic VRAM loading. 24449MB Staged. 0 patches attached. Force pre-loaded 160 weights: 2824 KB.
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:33<00:00, 4.15s/it]
[INFO] Requested to load WanVAE
[INFO] 0 models unloaded.
[INFO] Model WanVAE prepared for dynamic VRAM loading. 241MB Staged. 0 patches attached. Force pre-loaded 60 weights: 61 KB.
E:\ComfyUI\ComfyUI\nodes.py:1682: RuntimeWarning: invalid value encountered in cast
img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))
[INFO] Prompt executed in 47.86 seconds
```
### Other
_No response_
Contributor guide
Research direction
Start with the --disable-async-offload handling in ComfyUI/main.py and the image conversion warning at ComfyUI/nodes.py:1682. Reproduce the supplied Krea2 workflow with and without the flag on the reported bf16 setup, then trace where the disabled-offload run produces invalid values. Done means the workflow generates a valid image with async offload disabled and no invalid-value warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100