Z-Image Turbo - VAE crash when using bf16 models on 24GB RX 7900 XTX
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
VAE Decode step executes without crashing
### Actual Behavior
When the VAE Decode step starts, it appears to do nothing for about 5 seconds, and then crashes.
### Steps to Reproduce
Not sure if this issue is an actual bug or if I'm just expecting too much.
But, on https://huggingface.co/Tongyi-MAI/Z-Image-Turbo, I read: fits comfortably within 16G VRAM consumer devices.
I know you have to take these statements with a grain of salt, there usually is an unmentioned condition like "as long as you generate post stamp size images"
But even then, surely this should run correctly on my 24 GB card, not?
And also, in Task Manager, I see I still have plenty of free VRAM and System Ram.
Seemingly confirming that this should indeed be possible.
I am using the "Text to Image (New)" Template from ComfyUI. Unchanged.
The KSampler step executes correctly.
But when the VAE Decode step starts, it appears to do nothing for about 5 seconds, and then crashes.
I double checked if my vae was correct by comparing the hash from my vae file with what is published on:
https://huggingface.co/Comfy-Org/z_image_turbo/blob/main/split_files/vae/ae.safetensors
It has an identical hash:
>certutil -hashfile z_image_turbo_ae.safetensors SHA256
SHA256 hash of z_image_turbo_ae.safetensors:
afc8e28272cd15db3919bacdb6918ce9c1ed22e96cb12c4d5ed0fba823529e38
I tried using the Tiled VAE Decode instead of the regular VAE Decode, but that didn't help.
### Debug Logs
```powershell
## ComfyUI-Manager: installing dependencies done.
[2025-12-28 16:28:18.620] ** ComfyUI startup time: 2025-12-28 16:28:18.620
[2025-12-28 16:28:18.620] ** Platform: Windows
[2025-12-28 16:28:18.620] ** Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
[2025-12-28 16:28:18.620] ** Python executable: C:\Comfy\python_embeded\python.exe
[2025-12-28 16:28:18.620] ** ComfyUI Path: C:\Comfy\ComfyUI
[2025-12-28 16:28:18.620] ** ComfyUI Base Folder Path: C:\Comfy\ComfyUI
[2025-12-28 16:28:18.620] ** User directory: C:\Comfy\ComfyUI\user
[2025-12-28 16:28:18.620] ** ComfyUI-Manager config path: C:\Comfy\ComfyUI\user\__manager\config.ini
[2025-12-28 16:28:18.620] ** Log path: C:\Comfy\ComfyUI\user\comfyui.log
Prestartup times for custom nodes:
[2025-12-28 16:28:19.221] 1.7 seconds: C:\Comfy\ComfyUI\custom_nodes\comfyui-manager
[2025-12-28 16:28:19.221]
[2025-12-28 16:28:20.295] Checkpoint files will always be loaded safely.
[2025-12-28 16:28:20.531] Total VRAM 24560 MB, total RAM 64669 MB
[2025-12-28 16:28:20.531] pytorch version: 2.9.0+rocmsdk20251116
[2025-12-28 16:28:20.531] Set: torch.backends.cudnn.enabled = False for better AMD performance.
[2025-12-28 16:28:20.531] AMD arch: gfx1100
[2025-12-28 16:28:20.531] ROCm version: (7, 1)
[2025-12-28 16:28:20.531] Set vram state to: NORMAL_VRAM
[2025-12-28 16:28:20.531] Device: cuda:0 AMD Radeon RX 7900 XTX : native
[2025-12-28 16:28:20.547] Enabled pinned memory 29101.0
[2025-12-28 16:28:21.259] Using sub quadratic optimization for attention, if you have memory or speed issues try using: --use-split-cross-attention
[2025-12-28 16:28:22.605] Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
[2025-12-28 16:28:22.605] ComfyUI version: 0.6.0
[2025-12-28 16:28:22.637] ComfyUI frontend version: 1.34.9
[2025-12-28 16:28:22.647] [Prompt Server] web root: C:\Comfy\python_embeded\Lib\site-packages\comfyui_frontend_package\static
[2025-12-28 16:28:23.048] Total VRAM 24560 MB, total RAM 64669 MB
[2025-12-28 16:28:23.050] pytorch version: 2.9.0+rocmsdk20251116
[2025-12-28 16:28:23.050] Set: torch.backends.cudnn.enabled = False for better AMD performance.
[2025-12-28 16:28:23.050] AMD arch: gfx1100
[2025-12-28 16:28:23.050] ROCm version: (7, 1)
[2025-12-28 16:28:23.050] Set vram state to: NORMAL_VRAM
[2025-12-28 16:28:23.050] Device: cuda:0 AMD Radeon RX 7900 XTX : native
[2025-12-28 16:28:23.064] Enabled pinned memory 29101.0
[2025-12-28 16:28:23.379] ComfyUI-GGUF: Allowing full torch compile
[2025-12-28 16:28:23.387] ### Loading: ComfyUI-Manager (V3.39)
[2025-12-28 16:28:23.387] [ComfyUI-Manager] network_mode: public
[2025-12-28 16:28:23.387] [ComfyUI-Manager] ComfyUI per-queue preview override detected (PR #11261). Manager's preview method feature is disabled. Use ComfyUI's --preview-method CLI option or 'Settings > Execution > Live preview method'.
[2025-12-28 16:28:23.458] ### ComfyUI Revision: 150 [e4c61d75] *DETACHED | Released on '2025-12-23'
[2025-12-28 16:28:23.458]
Import times for custom nodes:
[2025-12-28 16:28:23.458] 0.0 seconds: C:\Comfy\ComfyUI\custom_nodes\websocket_image_save.py
[2025-12-28 16:28:23.473] 0.0 seconds: C:\Comfy\ComfyUI\custom_nodes\seedvarianceenhancer
[2025-12-28 16:28:23.473] 0.0 seconds: C:\Comfy\ComfyUI\custom_nodes\ComfyUI-GGUF
[2025-12-28 16:28:23.473] 0.1 seconds: C:\Comfy\ComfyUI\custom_nodes\comfyui-manager
[2025-12-28 16:28:23.473]
[2025-12-28 16:28:23.711] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[2025-12-28 16:28:23.727] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[2025-12-28 16:28:23.789] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[2025-12-28 16:28:23.805] Context impl SQLiteImpl.
[2025-12-28 16:28:23.805] Will assume non-transactional DDL.
[2025-12-28 16:28:23.805] No target revision found.
[2025-12-28 16:28:23.837] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[2025-12-28 16:28:23.884] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[2025-12-28 16:28:23.902] Starting server
[2025-12-28 16:28:23.902] To see the GUI go to: http://127.0.0.1:8188
[2025-12-28 16:28:25.956] [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[2025-12-28 16:28:25.961] [DEPRECATION WARNING] Detected import of deprecated legacy API: /extensions/core/groupNode.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[2025-12-28 16:28:26.162] [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/buttonGroup.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[2025-12-28 16:28:26.165] [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/button.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[2025-12-28 16:28:27.199] FETCH ComfyRegistry Data: 5/116
[2025-12-28 16:28:31.567] FETCH ComfyRegistry Data: 10/116
[2025-12-28 16:28:35.319] FETCH ComfyRegistry Data: 15/116
[2025-12-28 16:28:39.105] FETCH ComfyRegistry Data: 20/116
[2025-12-28 16:28:42.859] FETCH ComfyRegistry Data: 25/116
[2025-12-28 16:28:46.588] FETCH ComfyRegistry Data: 30/116
[2025-12-28 16:28:50.314] FETCH ComfyRegistry Data: 35/116
[2025-12-28 16:28:54.973] FETCH ComfyRegistry Data: 40/116
[2025-12-28 16:28:59.026] FETCH ComfyRegistry Data: 45/116
[2025-12-28 16:29:02.756] FETCH ComfyRegistry Data: 50/116
[2025-12-28 16:29:06.847] FETCH ComfyRegistry Data: 55/116
[2025-12-28 16:29:10.625] FETCH ComfyRegistry Data: 60/116
[2025-12-28 16:29:14.273] FETCH ComfyRegistry Data: 65/116
[2025-12-28 16:29:18.299] FETCH ComfyRegistry Data: 70/116
[2025-12-28 16:29:22.031] FETCH ComfyRegistry Data: 75/116
[2025-12-28 16:29:25.755] FETCH ComfyRegistry Data: 80/116
[2025-12-28 16:29:30.075] FETCH ComfyRegistry Data: 85/116
[2025-12-28 16:29:34.128] FETCH ComfyRegistry Data: 90/116
[2025-12-28 16:29:37.900] FETCH ComfyRegistry Data: 95/116
[2025-12-28 16:29:42.039] FETCH ComfyRegistry Data: 100/116
[2025-12-28 16:29:45.754] FETCH ComfyRegistry Data: 105/116
[2025-12-28 16:29:49.524] FETCH ComfyRegistry Data: 110/116
[2025-12-28 16:29:53.225] FETCH ComfyRegistry Data: 115/116
[2025-12-28 16:29:54.486] FETCH ComfyRegistry Data [DONE]
[2025-12-28 16:29:54.594] [ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
[2025-12-28 16:29:54.610] FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[2025-12-28 16:29:54.795] [ComfyUI-Manager] All startup tasks have been completed.
[2025-12-28 16:30:36.225] got prompt
[2025-12-28 16:30:36.241] Using split attention in VAE
[2025-12-28 16:30:36.241] Using split attention in VAE
[2025-12-28 16:30:36.352] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[2025-12-28 16:30:37.447] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[2025-12-28 16:30:38.243] Requested to load ZImageTEModel_
[2025-12-28 16:30:39.555] loaded completely; 22891.42 MB usable, 7672.25 MB loaded, full load: True
[2025-12-28 16:30:41.127] model weight dtype torch.bfloat16, manual cast: None
[2025-12-28 16:30:41.127] model_type FLOW
[2025-12-28 16:30:42.257] unet missing: ['norm_final.weight']
[2025-12-28 16:30:43.382] Requested to load Lumina2
[2025-12-28 16:30:45.759] Unloaded partially: 7672.25 MB freed, 0.00 MB remains loaded, 2225.62 MB buffer reserved, lowvram patches: 0
[2025-12-28 16:30:48.420] loaded completely; 18675.99 MB usable, 11739.55 MB loaded, full load: True
[2025-12-28 16:30:52.975]
100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.04s/it]
100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.11s/it]
[2025-12-28 16:30:52.975] Requested to load AutoencodingEngine
```
### Other
The Debug Log does not contain the error. That is only displayed on stdout:
Requested to load Lumina2
Unloaded partially: 7672.25 MB freed, 0.00 MB remains loaded, 2225.62 MB buffer reserved, lowvram patches: 0
loaded completely; 18675.99 MB usable, 11739.55 MB loaded, full load: True
100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.11s/it]
Requested to load AutoencodingEngine
:0:C:\develop\TheRock\rocm-systems\projects\clr\rocclr\device\device.cpp:379 : 1463596855075 us: Memobj map does not have ptr: 0x37180000
Task Manager Screenshot :
The "bump" in the compute part is the KSampler step.
Then VRAM usage drops to 1.3 GB, and about 5 seconds later it crashes.
I also collected memory usage with Performance Monitor.
You can find that data in this excel:
[issue3_Performance Counter.xlsx](https://github.com/user-attachments/files/24369605/issue3_Performance.Counter.xlsx)
Contributor guide
Assessment
This issue has not been assessed yet.