Z-Image Qwen3-4B GPU text encoder produces all-NaN conditioning on Blackwell sm_120
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
The Z-Image Qwen3-4B text encoder should produce finite conditioning tensors on every execution, including after previous Z-Image generations in the same ComfyUI process. Subsequent renders should remain valid and should not become black.
### Actual Behavior
On a Blackwell sm_120 GPU with PyTorch 2.9.1+cu130, the first Z-Image render succeeds, but a later GPU execution of the Z-Image Qwen3-4B text encoder can produce an entirely NaN positive-conditioning tensor. The resulting image is black, and subsequent Z-Image renders in the same ComfyUI process can also become black until ComfyUI is restarted.
Impact Pack FaceDetailer is a reliable way to expose the problem, but instrumentation shows that its sampling receives an already all-NaN conditioning context. LoRA loading and the Ultralytics detector were ruled out.
Moving only the Z-Image text encoder to CPU prevents the NaN conditioning and black renders while leaving the UNet and VAE in BF16 on the GPU. Restoring the text encoder to the GPU reproduces the failure.
### Steps to Reproduce
1. Start ComfyUI with the Z-Image Turbo model in BF16 and its Qwen3-4B text encoder loaded on the GPU.
2. Restart ComfyUI to begin with a clean process.
3. Run a normal Z-Image workflow. The initial image renders correctly.
4. In the same process, run a Z-Image face-inpainting workflow containing Impact Pack FaceDetailer with freshly encoded positive and negative prompts.
5. The FaceDetailer refined crop becomes completely black.
6. Run the original normal Z-Image workflow again without restarting ComfyUI.
7. All subsequent images become black. In the fixed test sequence, the clean process produced 0/24 black images, while the same sequence after one FaceDetailer execution produced 24/24 black images, beginning with render 1.
8. Restart ComfyUI and repeat the normal sequence. Rendering works again.
9. Move only the Z-Image Qwen3-4B text encoder to CPU while keeping the UNet and VAE in BF16 on the GPU, then repeat the complete sequence. The conditioning remains finite and none of the outputs become black.
10. Restore the text encoder to the GPU and the failure reproduces.
Temporary finite-value instrumentation showed that the positive-conditioning tensor supplied to sampling was entirely NaN. The FaceDetailer sampling operation received this already-invalid conditioning.
### Debug Logs
```powershell
GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
CUDA capability: sm_120
NVIDIA driver: 595.71.05
CUDA runtime: 13.0
PyTorch: 2.9.1+cu130
Python: 3.12.13
ComfyUI: 0.28.0
ComfyUI commit: 806e092ed42772e4ce7abf44c97c50021cc4bd10
** ComfyUI startup time: 2026-07-28 02:11:13.585
[2026-07-28 02:11:13.585] ** Platform: Linux
[2026-07-28 02:11:13.585] ** Python version: 3.12.13 (main, Jun 23 2026, 15:18:55) [Clang 22.1.3 ]
[2026-07-28 02:11:13.585] ** Python executable: /opt/comfyui/venv/bin/python
[2026-07-28 02:11:13.585] ** ComfyUI Path: /opt/comfyui
[2026-07-28 02:11:13.585] ** ComfyUI Base Folder Path: /opt/comfyui
[2026-07-28 02:11:13.586] ** User directory: /opt/comfyui/user
[2026-07-28 02:11:13.586] ** ComfyUI-Manager config path: /opt/comfyui/user/__manager/config.ini
[2026-07-28 02:11:13.586] ** Log path: /opt/comfyui/user/comfyui.log
[PRE] ComfyUI-Manager
[2026-07-28 02:11:14.758] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'rms_rope', 'rms_rope1', 'rms_rope_split_half', 'rms_rope_split_half1', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[2026-07-28 02:11:14.758] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'rms_rope', 'rms_rope1', 'rms_rope_split_half', 'rms_rope_split_half1', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[2026-07-28 02:11:14.758] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']}
[2026-07-28 02:11:14.760] Checkpoint files will always be loaded safely.
[2026-07-28 02:11:14.853] Total VRAM 97250 MB, total RAM 93928 MB
[2026-07-28 02:11:14.853] pytorch version: 2.9.1+cu130
[2026-07-28 02:11:14.853] Set vram state to: HIGH_VRAM
[2026-07-28 02:11:14.853] Disabling smart memory management
[2026-07-28 02:11:14.853] Device: cuda:0 NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition : native
[2026-07-28 02:11:14.853] Enabled pinned memory 84535.0
[2026-07-28 02:11:14.854] working around nvidia conv3d memory bug.
[2026-07-28 02:11:14.916] Using pytorch attention
[2026-07-28 02:11:15.946] Python version: 3.12.13 (main, Jun 23 2026, 15:18:55) [Clang 22.1.3 ]
[2026-07-28 02:11:15.946] ComfyUI version: 0.28.0
[2026-07-28 02:11:15.947] comfy-aimdo version: 0.4.10
[2026-07-28 02:11:15.947] comfy-kitchen version: 0.2.22
[2026-07-28 02:11:15.948] comfyui-frontend-package version: 1.47.10
[2026-07-28 02:11:15.948] comfyui-workflow-templates version: 0.11.17
[2026-07-28 02:11:15.948] comfyui-embedded-docs version: 0.5.8
[2026-07-28 02:11:15.948] comfy-kitchen version: 0.2.22
[2026-07-28 02:11:15.948] comfy-aimdo version: 0.4.10
[2026-07-28 02:11:15.948] [Prompt Server] web root: /opt/comfyui/venv/lib/python3.12/site-packages/comfyui_frontend_package/static
[2026-07-28 02:11:15.949] Asset seeder disabled
[2026-07-28 02:11:15.949] [START] ComfyUI-Manager
[2026-07-28 02:11:15.965] [ComfyUI-Manager] Using GitPython backend
[2026-07-28 02:11:15.995] [ComfyUI-Manager] network_mode: public
[2026-07-28 02:11:15.996] [ComfyUI-Manager] The matrix sharing feature has been disabled because the `matrix-nio` dependency is not installed.
To use this feature, please run the following command:
/opt/comfyui/venv/bin/python -m pip install matrix-nio
[2026-07-28 02:11:16.146] No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[2026-07-28 02:11:16.586] ### Loading: ComfyUI-Impact-Subpack (V1.3.5)
[2026-07-28 02:11:16.586] [Impact Pack/Subpack] Using folder_paths to determine whitelist path: /opt/comfyui/user/default/ComfyUI-Impact-Subpack/model-whitelist.txt
[2026-07-28 02:11:16.586] [Impact Pack/Subpack] Ensured whitelist directory exists: /opt/comfyui/user/default/ComfyUI-Impact-Subpack
[2026-07-28 02:11:16.586] [Impact Pack/Subpack] Loaded 0 model(s) from whitelist: /opt/comfyui/user/default/ComfyUI-Impact-Subpack/model-whitelist.txt
[2026-07-28 02:11:16.911] [Impact Subpack] ultralytics_bbox: /opt/comfyui/models/ultralytics/bbox
[2026-07-28 02:11:16.911] [Impact Subpack] ultralytics_segm: /opt/comfyui/models/ultralytics/segm
[2026-07-28 02:11:16.914]
[ReActor] - STATUS - Running v0.7.0-a2 in ComfyUI
[2026-07-28 02:11:16.936] Torch version: 2.9.1+cu130
[2026-07-28 02:11:17.057] ### Loading: ComfyUI-Impact-Pack (V8.28.3)
[2026-07-28 02:11:17.102] [Impact Pack] Wildcard total size (0.00 MB) is within cache limit (50.00 MB). Using full cache mode.
[2026-07-28 02:11:17.103] [Impact Pack] Wildcards loading done.
[2026-07-28 02:11:17.104] [/opt/comfyui/custom_nodes/comfyui_controlnet_aux] | INFO -> Using ckpts path: /opt/comfyui/custom_nodes/comfyui_controlnet_aux/ckpts
[2026-07-28 02:11:17.104] [/opt/comfyui/custom_nodes/comfyui_controlnet_aux] | INFO -> Using symlinks: False
[2026-07-28 02:11:17.104] [/opt/comfyui/custom_nodes/comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider']
[2026-07-28 02:11:17.108] DWPose: Onnxruntime with acceleration providers detected
[2026-07-28 02:11:17.109]
Import times for custom nodes:
[2026-07-28 02:11:17.109] 0.0 seconds: /opt/comfyui/custom_nodes/websocket_image_save.py
[2026-07-28 02:11:17.109] 0.0 seconds: /opt/comfyui/custom_nodes/comfyui_controlnet_aux
[2026-07-28 02:11:17.109] 0.0 seconds: /opt/comfyui/custom_nodes/ComfyUI-Impact-Pack
[2026-07-28 02:11:17.109] 0.1 seconds: /opt/comfyui/custom_nodes/ComfyUI-ReActor
[2026-07-28 02:11:17.109] 0.3 seconds: /opt/comfyui/custom_nodes/ComfyUI-Impact-Subpack
[2026-07-28 02:11:17.109]
[2026-07-28 02:11:17.110] Context impl SQLiteImpl.
[2026-07-28 02:11:17.110] Will assume non-transactional DDL.
[2026-07-28 02:11:17.131] Using RAM pressure cache.
[2026-07-28 02:11:17.131] Starting server
[2026-07-28 02:11:17.131] To see the GUI go to: http://0.0.0.0:8188
[2026-07-28 02:11:18.414] got prompt
[2026-07-28 02:11:18.426] Using pytorch attention in VAE
[2026-07-28 02:11:18.427] Using pytorch attention in VAE
[2026-07-28 02:11:18.486] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[2026-07-28 02:11:19.234] Requested to load ZImageTEModel_
[2026-07-28 02:11:19.244] loaded completely; 7672.25 MB loaded, full load: True
[2026-07-28 02:11:19.245] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
[2026-07-28 02:11:19.557] model weight dtype torch.bfloat16, manual cast: None
[2026-07-28 02:11:19.558] model_type FLOW
[2026-07-28 02:11:20.110] Requested to load Lumina2
[2026-07-28 02:11:23.111] loaded completely; 11739.54 MB loaded, full load: True
100%|██████████| 8/8 [00:02<00:00, 3.19it/s]
[2026-07-28 02:11:25.641] Requested to load AutoencodingEngine
[2026-07-28 02:11:25.642] [codex-zimage-weight] before LoadedModel.model_unload Lumina2 finite=True device=cuda:0 dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137175618289664 storage_ptr=137175618289664
[2026-07-28 02:11:28.696] [codex-zimage-weight] after detach Lumina2 finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:28.785] [codex-zimage-weight] before LoadedModel.model_load AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:28.838] [codex-zimage-weight] after model_patches_to AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:28.868] loaded completely; 319.75 MB loaded, full load: True
[2026-07-28 02:11:28.922] [codex-zimage-weight] after model_use_more_vram AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:29.331] [codex-zimage-weight] before LoadedModel.model_unload AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:29.455] [codex-zimage-weight] after detach AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:29.470] Prompt executed in 11.05 seconds
[2026-07-28 02:11:29.737] got prompt
[2026-07-28 02:11:29.760] Requested to load ZImageTEModel_
[2026-07-28 02:11:29.815] [codex-zimage-weight] before LoadedModel.model_load ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:29.870] [codex-zimage-weight] after model_patches_to ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:30.511] loaded completely; 7672.25 MB loaded, full load: True
[2026-07-28 02:11:30.568] [codex-zimage-weight] after model_use_more_vram ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:30.658] [codex-zimage-weight] before LoadedModel.model_load ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:30.714] [codex-zimage-weight] after model_patches_to ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:30.770] [codex-zimage-weight] after model_use_more_vram ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:30.884] [codex-zimage-weight] Impact FaceDetailer before bbox detection finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:30.896]
[2026-07-28 02:11:30.991] 0: 640x448 1 face, 89.2ms
[2026-07-28 02:11:30.991] Speed: 0.9ms preprocess, 89.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)
[2026-07-28 02:11:31.058] [codex-zimage-weight] Impact FaceDetailer after bbox detection finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:31.121] [codex-zimage-weight] Impact DetailerForEach after image clone finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:31.220] [codex-zimage-weight] Impact DetailerForEach after crop and mask blur finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:31.282] [codex-zimage-weight] Impact DetailerForEach before enhance_detail finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:31.307] Detailer: segment upscale for ((np.float32(192.6807), np.float32(273.33606))) | crop region (578, 820) x 1.2489380836486816 -> (721, 1024)
[2026-07-28 02:11:31.380] [codex-zimage-weight] Impact enhance_detail after tensor_resize finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:31.442] [codex-zimage-weight] Impact enhance_detail before to_latent_image finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:31.442] Requested to load AutoencodingEngine
[2026-07-28 02:11:31.504] [codex-zimage-weight] before LoadedModel.model_unload ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:33.497] [codex-zimage-weight] after detach ZImageTEModel_ finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:33.579] [codex-zimage-weight] before LoadedModel.model_load AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:33.637] [codex-zimage-weight] after model_patches_to AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:33.668] loaded completely; 319.75 MB loaded, full load: True
[2026-07-28 02:11:33.727] [codex-zimage-weight] after model_use_more_vram AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:33.804] [Impact Pack] vae encoded in 2.4s
[2026-07-28 02:11:33.807] Requested to load Lumina2
[2026-07-28 02:11:33.866] [codex-zimage-weight] before LoadedModel.model_unload AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:34.000] [codex-zimage-weight] after detach AutoencodingEngine finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:34.065] [codex-zimage-weight] before LoadedModel.model_load Lumina2 finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:34.124] [codex-zimage-weight] after model_patches_to Lumina2 finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137152041709632 storage_ptr=137152041709632
[2026-07-28 02:11:35.136] loaded completely; 11739.54 MB loaded, full load: True
[2026-07-28 02:11:35.139] [codex-zimage-weight] after model_use_more_vram Lumina2 finite=True device=cuda:0 dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137171077955584 storage_ptr=137171077955584
[2026-07-28 02:11:35.141]
[2026-07-28 02:11:35.143] !!! Exception during processing !!! Z-Image became non-finite at patchify input context
[2026-07-28 02:11:35.144] Traceback (most recent call last):
File "/opt/comfyui/execution.py", line 543, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/execution.py", line 342, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/execution.py", line 316, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/opt/comfyui/execution.py", line 304, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/impact_pack.py", line 881, in doit
enhanced_img, cropped_enhanced, cropped_enhanced_alpha, mask, cnet_pil_list = FaceDetailer.enhance_face(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/impact_pack.py", line 835, in enhance_face
DetailerForEach.do_detail(image, segs, model, clip, vae, guide_size, guide_size_for_bbox, max_size, seed, steps, cfg,
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/impact_pack.py", line 365, in do_detail
enhanced_image, cnet_pils = core.enhance_detail(cropped_image, model, clip, vae, guide_size, guide_size_for_bbox, max_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/core.py", line 385, in enhance_detail
refined_latent = impact_sampling.ksampler_wrapper(model2, seed2, steps2, cfg2, sampler_name2, scheduler2, positive2, negative2,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/impact_sampling.py", line 209, in ksampler_wrapper
refined_latent = separated_sample(model, True, seed, advanced_steps, cfg, sampler_name, scheduler,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/impact_sampling.py", line 182, in separated_sample
res = sample_with_custom_noise(model, add_noise, seed, cfg, positive, negative, impact_sampler, sigmas, latent_image, noise=noise, callback=callback)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/custom_nodes/ComfyUI-Impact-Pack/modules/impact/impact_sampling.py", line 126, in sample_with_custom_noise
samples = comfy.sample.sample_custom(model, noise, cfg, sampler, sigmas, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/sample.py", line 79, in sample_custom
samples = comfy.samplers.sample(model, noise, positive, negative, cfg, model.load_device, sampler, sigmas, model_options=model.model_options, latent_image=latent_image, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1334, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1316, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1254, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1229, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 999, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/k_diffusion/sampling.py", line 205, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 639, in __call__
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1202, in __call__
return self.outer_predict_noise(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1209, in outer_predict_noise
).execute(x, timestep, model_options, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 1212, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 619, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 210, in calc_cond_batch
return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 218, in _calc_cond_batch_outer
return executor.execute(model, conds, x_in, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/samplers.py", line 334, in _calc_cond_batch
output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/model_base.py", line 194, in apply_model
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/model_base.py", line 238, in _apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/ldm/lumina/model.py", line 883, in forward
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/ldm/lumina/model.py", line 926, in _forward
img, mask, img_size, cap_size, freqs_cis, timestep_zero_index = self.patchify_and_embed(x, cap_feats, cap_mask, adaln_input, num_tokens, ref_latents=ref_latents, ref_contexts=ref_contexts, siglip_feats=siglip_feats, transformer_options=transformer_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/comfyui/comfy/ldm/lumina/model.py", line 776, in patchify_and_embed
_codex_require_finite(cap_feats, "patchify input context")
File "/opt/comfyui/comfy/ldm/lumina/model.py", line 24, in _codex_require_finite
raise RuntimeError(f"Z-Image became non-finite at {label}")
RuntimeError: Z-Image became non-finite at patchify input context
[2026-07-28 02:11:35.147] [codex-zimage-weight] before LoadedModel.model_unload Lumina2 finite=True device=cuda:0 dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137171077955584 storage_ptr=137171077955584
[2026-07-28 02:11:38.160] [codex-zimage-weight] after detach Lumina2 finite=True device=cpu dtype=torch.bfloat16 nonfinite=0/44236800 parameter_id=137176305861808 tensor_ptr=137142411587648 storage_ptr=137142411587648
[2026-07-28 02:11:38.192] Prompt executed in 8.45 seconds
[2026-07-28 02:13:37.867] FETCH ComfyRegistry Data [DONE]
[2026-07-28 02:13:37.969] [ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
[2026-07-28 02:13:37.988] FETCH DATA from: /opt/comfyui/user/__manager/cache/1514988643_custom-node-list.json [DONE]
[2026-07-28 02:13:38.007] [ComfyUI-Manager] All startup tasks have been completed.
```
### Other
Isolation results:
- Fresh ComfyUI process: 0/24 black images.
- After one Z-Image FaceDetailer execution: 24/24 subsequent renders were black, starting with the first render.
- Restarting ComfyUI restored normal rendering.
- The failure also occurred with comfy-kitchen’s CUDA backend disabled and eager operations enabled.
- PyTorch cross-attention, disabling asynchronous offloading, and changing the CUDA allocator did not prevent the failure.
- LoRA loading and the Ultralytics face detector were ruled out.
- Temporary finite-value checks showed that Z-Image’s positive conditioning was entirely NaN when it reached Lumina2’s patchify_and_embed function.
- The tested Lumina2 model weights remained finite when the invalid conditioning was detected.
A local workaround was added in comfy/sd.py: when the text encoder module is comfy.text_encoders.z_image and the CUDA device capability is (12, 0), only the Z-Image text encoder’s load and offload devices are changed to CPU.
The Z-Image UNet and VAE remain in BF16 on the GPU.
With this workaround, the following sequences passed in the same long-lived ComfyUI process:
- LoRA render, FaceDetailer render, then post-trigger render.
- LoRA render with cache-busted fresh positive and negative conditioning.
- No-LoRA render with cache-busted fresh positive and negative conditioning.
All full images, refined crops, and post-trigger images remained non-black. Whole-prompt execution times with the CPU text encoder were approximately 4.06–7.48 seconds.
This CPU placement is presented as a confirmed workaround, not necessarily the final underlying fix. The issue may ultimately be in PyTorch/CUDA execution of Qwen3-4B on sm_120, but ComfyUI can avoid producing invalid
conditioning or provide a targeted fallback.
The original Impact Pack report is: https://github.com/ltdrdata/ComfyUI-Impact-Pack/issues/1225
Contributor guide
Research direction
No source file or test is named. Start by reproducing the clean-process and FaceDetailer sequence with the ZImageTEModel_ Qwen3-4B text encoder on sm_120, then inspect where the positive conditioning reaches sampling. Done means conditioning remains finite and subsequent Z-Image renders stay valid without moving the encoder to CPU.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100