Comfy-Org / Comfy-Org/ComfyUI

Z-Image Turbo bf16: non-finite output on warm rerun on ROCm gfx1201 (RX 9070 XT), cold run valid

Open
#15,914 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

On AMD RX 9070 XT (gfx1201, RDNA4) with ROCm, Z-Image Turbo bf16 completes a cold run with valid output, but the first warm rerun of the same prompt produces an image containing non-finite values (NaN/Inf).

Environment:
- GPU: AMD Radeon RX 9070 XT, gfx1201, 16 GB
- torch 2.12.0+rocm7.14.0, HIP 7.14.60850, Linux
- ComfyUI at commit b78cec879b9460d5cb25228a83a942fb78d2cd24
- Model: Comfy-Org/z_image_turbo split files at HF revision 08d04455279082882deaabc8d0d09fc914c071e1 (z_image_turbo_bf16.safetensors diffusion, qwen_3_4b.safetensors text encoder, ae.safetensors VAE), all sha256-verified

Reproduction: standard Z-Image Turbo text-to-image graph (CheckpointLoader-equivalent split loaders, 1024x1024, 8 steps, cfg 1.0, fixed seed), run once (cold, output valid), then rerun the identical prompt (warm run 0): the decoded image contains non-finite values. Reproduced 2/2 on this hardware across two fully independent clones and virtual environments, weeks apart. Server ran default settings: NORMAL_VRAM, DynamicVRAM, PyTorch attention, native component staging.

Not reproduced on CUDA (same graph and artifacts are clean on NVIDIA lanes) and not memory pressure: device peak was well under 16 GB and host RAM under half of 62 GB. A separate intermittent single-run non-finite was also observed once on Intel XPU (Arc B570) with SDXL+LoRA, so this may be a non-CUDA backend numerical issue rather than gfx1201-specific, but only the gfx1201 Z-Image case is deterministic for us.

Caveat: measured at the pinned commit above for benchmarking purposes; not yet retested on current master. Happy to rerun a candidate fix on the same hardware.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the standard Z-Image Turbo split-loader graph on current master with the stated RX 9070 XT, fixed seed, and identical cold/warm runs. Compare where non-finite values first appear and validate any candidate fix against the pinned environment; done means the warm rerun produces a finite decoded image without regressing the valid cold run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.