Wan 2.2 14B (I2V) sampler emits NaN latents → every frame one flat colour on AMD gfx1201 (Radeon AI PRO R9700), ROCm 7.2.4, torch 2.14/2.15 — 5B renders correctly through the same graph nodes

## Environment
- GPU: AMD Radeon AI PRO R9700 32 GB, gfx1201 (RDNA4); host Ubuntu 26.04, in-tree amdgpu, 30 GB RAM
- Container: rocm/dev-ubuntu-24.04:7.2.4-complete; python 3.12
- torch 2.14.0+rocm7.2 (HIP 7.2.53211) — and reproduced on torch 2.15.0.dev20260827+rocm7.2 with ComfyUI HEAD
- ComfyUI launched with: `--highvram --use-pytorch-cross-attention --disable-smart-memory --preview-method latent2rgb`
- ComfyUI-GGUF at 6ea2651 (2026-01-12) for the GGUF runs; stock UNETLoader for the fp8 runs
- PYTORCH_HIP_ALLOC_CONF=expandable_segments:True

## Symptom
Graph runs to `execution_success` (no error, no OOM), 88–125 s for 640×640×33–49 frames, ~28 GB peak VRAM.
The decoded video is a single flat colour in every frame (per-frame luminance stddev 0.0–3.4; a real render of
the same start image scores 45–70). Frame 0 of I2V runs sometimes keeps a faint trace of the injected start image;
frames 1..N are uniform. Over ~22 I2V runs, 2 produced a correct video; both good runs were bit-identical to each other, and the same graph/seed run again immediately afterwards was flat — so the computation is deterministic when it works and the corruption is a random event. A second, unrelated graph on the same box shows the same signature: Z-Image Turbo img2img (VAEEncode of a picture → KSampler at denoise 0.3–0.95) comes out flat/mush while its VAE round trip and text-to-image are correct; Wan 2.2 5B image-to-video, which also feeds an encoded start frame, is fine.

## What was eliminated (one variable per run; frame-stddev score on the decoded output)
| axis | tested | result |
|---|---|---|
| weight format | GGUF Q5_K_M (QuantStack) / fp8_scaled safetensors (stock loader) | both flat |
| GGUF dequant dtype | default / float16 / bfloat16 | all flat |
| UNet compute dtype | fp16 (default) / `--bf16-unet` / `--fp32-unet` | all flat (fp32: 220 s/step, still flat) |
| text encoder | umt5-xxl Q4_K_M GGUF / umt5_xxl_fp16 (the GGUF one renders correctly on the 5B) | flat |
| VAE decode | VAEDecode / VAEDecodeTiled (256/64, temporal 16/4) | flat (tiled: 23.8 GB peak, so not memory) |
| attention | `--use-pytorch-cross-attention` / `--use-split-cross-attention` | flat |
| GEMM backend | default / `TORCH_BLAS_PREFER_HIPBLASLT=0 DISABLE_ADDMM_HIP_LT=1 ROCBLAS_USE_HIPBLASLT=0` | flat |
| HIP async | default / `HIP_LAUNCH_BLOCKING=1 AMD_SERIALIZE_KERNEL=3 AMD_SERIALIZE_COPY=3` | flat |
| LoRA | lightx2v 4-step pair (4 steps, cfg 1) / no LoRA (20 steps, cfg 3.5) | flat |
| frame count | 33 / 37 / 41 / 49 | flat |
| torch | 2.14.0+rocm7.2 / 2.15.0.dev20260827+rocm7.2 (+ ComfyUI HEAD) | flat |
| memory mode | default / `--highvram` (and `--gpu-only`) | flat; container never OOMs |
| control | Wan 2.2 TI2V 5B through the SAME container, encoder, VAE, sampler nodes | **correct video every time** |
| control | Z-Image Turbo stills | correct |

## What narrows it (all on the same container, same quant, same sampler split)
| graph | result |
|---|---|
| **T2V** A14B Q5_K_M, EmptyHunyuanLatentVideo, t2v lightx2v v1.1 LoRAs, 640×640×33 | **correct, 3/3 runs, bit-identical scores** (74.7/70.7/73.0) |
| I2V A14B, `WanImageToVideo` **without** `start_image` (zero concat latent + mask, same 36-channel input) | **correct** (55.5/56.0/55.9) |
| I2V A14B with a start image (1024² upload, or a CPU-pre-resized 640² PNG) | **flat ~90%**; when it does come out right it is bit-identical each time (51.1/45.8/49.8) |
| I2V graph instrumented with `SaveLatent` after `WanImageToVideo`, KSampler 1, KSampler 2 | in the good run all three latents are finite and sane; the bad runs are the same graph |
| `wan_2.1_vae` encode→decode round trip: 1 frame; 33 identical frames (temporal path); with GPU `ImageScale` | all correct (51.x) |
| bare torch, cuDNN off as ComfyUI sets it: Conv3d(36→5120,(1,2,2)) fp32/fp16 and the equivalent [14400×144]@[144×5120] GEMM vs CPU | correct 12/12 (both hipBLASLt and rocBLAS) |
| bare torch, cuDNN ON: same Conv3d | MIOpen returns wrong results for in=36/40 fp16, 36 fp32, 48 bf16 depending on solver search (`MIOPEN_DEBUG_CONV_GEMM=0` fixes it) — but ComfyUI does not use MIOpen on AMD, so this is a separate finding |
| `COMFY_KITCHEN_DISABLE_HIP=1` (eager kernels for rope/rmsnorm) | flat |
| HIP serialization, rocBLAS-only GEMM, `--fp32-unet`, torch 2.15 nightly | flat |

So: the same 14B weights produce correct video whenever the concat channels carry no image latent, and
non-deterministically corrupt video (~1 in 10 correct, identical when correct) whenever they do. Every op on the
image-latent path is correct in isolation. The corruption is therefore inside the DiT forward with real
conditioning values, and it is not reproducible run-to-run — consistent with a data-dependent kernel fault or an
uninitialised-memory read on this backend, not with a graph or dtype problem.

## Minimal reproduction
`diag14b.py` (below): builds the API-format graph (UnetLoaderGGUFAdvanced×2 → LoraLoaderModelOnly×2 →
ModelSamplingSD3 shift 8 → CLIPLoaderGGUF umt5 → CLIPTextEncode ±→ WanImageToVideo(start_image) →
KSamplerAdvanced 0→2 (leftover noise) → KSamplerAdvanced 2→end → VAEDecode wan_2.1_vae → SaveWEBM), queues it,
waits, and prints a per-frame luminance stddev for frames 0/mid/last. Flat < 4, real > 40.

## Logs
`model weight dtype torch.float16, manual cast: None` / `loaded completely; 10424.69 MB loaded, full load: True` for
both UNets; no warnings between `Requested to load WAN21` and `Prompt executed`; one GGUF warning per run:
`Dequantizing token_embd.weight to prevent runtime OOM` (also present in the correct 5B+GGUF-encoder runs).

## Related
- #11574 (9070 XT, ROCm 7.1.1: the inverse -- 5B corrupt, 14B T2V fine)
- ROCm/TheRock#1941 (gfx1201 KSampler hang on 7.10 nightlies)

diag14b.py -- reproduction harness (queues the API graph, waits, scores frames 0/mid/last by luminance stddev)

```python
#!/usr/bin/env python3
"""Isolate why the 14B GGUF i2v renders flat colour. One variable per run.
Usage: diag14b.py variants: base | clipfp16 | nolora | deqfp16 | clipfp16_nolora
Prints elapsed, peak VRAM, and a structure score (mean per-frame stddev over 3 sampled frames;
a flat frame scores < 3, a real image scores > 20)."""
import json, subprocess, sys, time, urllib.request, io

COMFY = "http://127.0.0.1:8188"
OUT = ""
variant = sys.argv[1] if len(sys.argv) > 1 else "base"
seed = 424242
prompt = ("the lighthouse beam sweeps slowly across the scene, waves crash against the rocks and send up spray, "
"dramatic clouds drift in the golden light, gentle slow camera push-in, photoreal")
neg = "blurry, low quality, distorted, watermark, text"
image = "start.png"
W, H = 640, 640
L = int(sys.argv[2]) if len(sys.argv) > 2 else 33 # frames: the Studio default is 49
nolora = "nolora" in variant
steps = 20 if nolora else 4
cfg = 3.5 if nolora else 1.0
mid = steps // 2 if nolora else 2
deq = "bfloat16" if "deqbf16" in variant else ("float16" if "deqfp16" in variant else "default")

if "fp8" in variant:
# Same model, NOT via ComfyUI-GGUF: the fp8_scaled safetensors through the
# stock UNETLoader. Separates "the 14B is broken here" from "the GGUF
# dequant kernels are broken here".
# "fp8keep": weight_dtype fp8_e4m3fn keeps the weights fp8 ON THE CARD
# (14 GB each) instead of upcasting to fp16 through host RAM -- the upcast
# is what thrashed the box; compute still upcasts per layer.
wd = "fp8_e4m3fn" if "fp8keep" in variant else "default"
loaders = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors",
"weight_dtype": wd}},
"2": {"class_type": "UNETLoader", "inputs": {"unet_name": "wan2.2_i2v_low_noise_14B_fp8_scaled.safetensors",
"weight_dtype": wd}},
}
else:
loaders = {
"1": {"class_type": "UnetLoaderGGUFAdvanced", "inputs": {"unet_name": "Wan2.2-I2V-A14B-HighNoise-Q5_K_M.gguf",
"dequant_dtype": deq, "patch_dtype": "default", "patch_on_device": True}},
"2": {"class_type": "UnetLoaderGGUFAdvanced", "inputs": {"unet_name": "Wan2.2-I2V-A14B-LowNoise-Q5_K_M.gguf",
"dequant_dtype": deq, "patch_dtype": "default", "patch_on_device": True}},
}
g = {
**loaders,
"8": {"class_type": "VAELoader", "inputs": {"vae_name": "wan_2.1_vae.safetensors"}},
"9": {"class_type": "LoadImage", "inputs": {"image": image}},
"16": {"class_type": "SaveWEBM", "inputs": {"images": ["15", 0], "filename_prefix": f"diag14b-{variant}",
"codec": "vp9", "fps": 16.0, "crf": 32.0}},
"15": ({"class_type": "VAEDecodeTiled", "inputs": {"samples": ["14", 0], "vae": ["8", 0], "tile_size": 256,
"overlap": 64, "temporal_size": 16, "temporal_overlap": 4}}
if "tiled" in variant else
{"class_type": "VAEDecode", "inputs": {"samples": ["14", 0], "vae": ["8", 0]}}),
}
if "clipfp16" in variant:
g["7"] = {"class_type": "CLIPLoader", "inputs": {"clip_name": "umt5_xxl_fp16.safetensors", "type": "wan"}}
else:
g["7"] = {"class_type": "CLIPLoaderGGUF", "inputs": {"clip_name": "umt5-xxl-encoder-Q4_K_M.gguf", "type": "wan"}}
if nolora:
hi, lo = "1", "2"
else:
g["3"] = {"class_type": "LoraLoaderModelOnly", "inputs": {"model": ["1", 0],
"lora_name": "wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors", "strength_model": 1.0}}
g["4"] = {"class_type": "LoraLoaderModelOnly", "inputs": {"model": ["2", 0],
"lora_name": "wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors", "strength_model": 1.0}}
hi, lo = "3", "4"
g["5"] = {"class_type": "ModelSamplingSD3", "inputs": {"model": [hi, 0], "shift": 8.0}}
g["6"] = {"class_type": "ModelSamplingSD3", "inputs": {"model": [lo, 0], "shift": 8.0}}
g["10"] = {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": ["7", 0]}}
g["11"] = {"class_type": "CLIPTextEncode", "inputs": {"text": neg, "clip": ["7", 0]}}
g["12"] = {"class_type": "WanImageToVideo", "inputs": {"positive": ["10", 0], "negative": ["11", 0], "vae": ["8", 0],
"width": W, "height": H, "length": L, "batch_size": 1, "start_image": ["9", 0]}}
g["13"] = {"class_type": "KSamplerAdvanced", "inputs": {"model": ["5", 0], "add_noise": "enable", "noise_seed": seed,
"steps": steps, "cfg": cfg, "sampler_name": "euler", "scheduler": "simple", "positive": ["12", 0],
"negative": ["12", 1], "latent_image": ["12", 2], "start_at_step": 0, "end_at_step": mid,
"return_with_leftover_noise": "enable"}}
g["14"] = {"class_type": "KSamplerAdvanced", "inputs": {"model": ["6", 0], "add_noise": "disable", "noise_seed": seed,
"steps": steps, "cfg": cfg, "sampler_name": "euler", "scheduler": "simple", "positive": ["12", 0],
"negative": ["12", 1], "latent_image": ["13", 0], "start_at_step": mid, "end_at_step": 10000,
"return_with_leftover_noise": "disable"}}

if variant.startswith("5b"):
# The 5B pipeline is known-good with the fp16 text encoder. Swap ONLY the
# encoder for the GGUF one the 14B graph uses: if this goes flat, the
# encoder is the culprit, not the 14B model or its loaders.
clip = ({"class_type": "CLIPLoaderGGUF", "inputs": {"clip_name": "umt5-xxl-encoder-Q4_K_M.gguf", "type": "wan"}}
if "ggufclip" in variant else
{"class_type": "CLIPLoader", "inputs": {"clip_name": "umt5_xxl_fp16.safetensors", "type": "wan"}})
g = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "wan2.2_ti2v_5B_fp16.safetensors", "weight_dtype": "default"}},
"2": clip,
"3": {"class_type": "VAELoader", "inputs": {"vae_name": "wan2.2_vae.safetensors"}},
"4": {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": ["2", 0]}},
"5": {"class_type": "CLIPTextEncode", "inputs": {"text": neg, "clip": ["2", 0]}},
"6": {"class_type": "Wan22ImageToVideoLatent", "inputs": {"vae": ["3", 0], "width": W, "height": H, "length": L, "batch_size": 1}},
"7": {"class_type": "ModelSamplingSD3", "inputs": {"model": ["1", 0], "shift": 8.0}},
"8": {"class_type": "KSampler", "inputs": {"model": ["7", 0], "seed": seed, "steps": 12, "cfg": 5.0,
"sampler_name": "euler", "scheduler": "simple", "positive": ["4", 0], "negative": ["5", 0],
"latent_image": ["6", 0], "denoise": 1.0}},
"9": {"class_type": "VAEDecode", "inputs": {"samples": ["8", 0], "vae": ["3", 0]}},
"16": {"class_type": "SaveWEBM", "inputs": {"images": ["9", 0], "filename_prefix": f"diag-{variant}",
"codec": "vp9", "fps": 16.0, "crf": 32.0}},
}

if variant.startswith("t2v"):
# TEXT-to-video 14B: same quant, same sampler split, same LoRA family
# (the t2v lightx2v v1.1 pair), but NO image conditioning -- the latent
# comes from EmptyHunyuanLatentVideo and the conditioning is the plain
# CLIPTextEncode outputs. If this renders where I2V does not, the fault is
# in WanImageToVideo's concat/mask path, not the 14B blocks.
g["1"]["inputs"]["unet_name"] = "Wan2.2-T2V-A14B-HighNoise-Q5_K_M.gguf"
g["2"]["inputs"]["unet_name"] = "Wan2.2-T2V-A14B-LowNoise-Q5_K_M.gguf"
if not nolora:
g["3"]["inputs"]["lora_name"] = "wan2.2_t2v_lightx2v_4steps_lora_v1.1_high_noise.safetensors"
g["4"]["inputs"]["lora_name"] = "wan2.2_t2v_lightx2v_4steps_lora_v1.1_low_noise.safetensors"
g.pop("9", None); g.pop("12", None)
g["12"] = {"class_type": "EmptyHunyuanLatentVideo", "inputs": {"width": W, "height": H, "length": L, "batch_size": 1}}
for k in ("13", "14"):
g[k]["inputs"]["positive"] = ["10", 0]
g[k]["inputs"]["negative"] = ["11", 0]
g["13"]["inputs"]["latent_image"] = ["12", 0]
prompt = ("cinematic aerial shot slowly gliding forward over a misty bayou at sunrise, cypress trees draped in "
"moss over still dark water, golden light through low fog, a white heron lifting off, photoreal")
g["10"]["inputs"]["text"] = prompt

if variant.startswith("vaeenc"):
# No diffusion at all: LoadImage -> wan_2.1_vae ENCODE -> DECODE -> SaveImage.
# The T2V 14B decodes through this VAE correctly and the I2V graph is the
# only one that ENCODES with it. If the round trip comes back flat, the
# encoder is the broken piece on this backend. "vaeenc22" does the same
# with wan2.2_vae (the 5B's, which works end to end) as the control.
vae = "wan2.2_vae.safetensors" if "22" in variant else "wan_2.1_vae.safetensors"
g = {
"8": {"class_type": "VAELoader", "inputs": {"vae_name": vae}},
"9": {"class_type": "LoadImage", "inputs": {"image": image}},
"20": {"class_type": "VAEEncode", "inputs": {"pixels": ["9", 0], "vae": ["8", 0]}},
"21": {"class_type": "VAEDecode", "inputs": {"samples": ["20", 0], "vae": ["8", 0]}},
"16": {"class_type": "SaveImage", "inputs": {"images": ["21", 0], "filename_prefix": f"diag-{variant}"}},
}
L = 1

if "img640" in variant:
# Start image already 640x640 (resized on the CPU by ffmpeg): WanImageToVideo
# then skips its GPU bilinear resize. Structured => the GPU resize is the
# broken piece; flat => it is the latent placement / model consumption.
g["9"]["inputs"]["image"] = "lighthouse-640.png"

if variant.startswith("vaeenc33"):
# TEMPORAL encode: 33 identical frames of the start image through the 2.1 VAE
# (what WanImageToVideo does, minus the grey padding), decode back, save the
# latent. Any frame after the first that differs from frame 0 is an encoder
# fault -- the input frames are identical.
g = {
"8": {"class_type": "VAELoader", "inputs": {"vae_name": "wan_2.1_vae.safetensors"}},
"9": {"class_type": "LoadImage", "inputs": {"image": "lighthouse-640.png"}},
"18": {"class_type": "RepeatImageBatch", "inputs": {"image": ["9", 0], "amount": 33}},
"20": {"class_type": "VAEEncode", "inputs": {"pixels": ["18", 0], "vae": ["8", 0]}},
"30": {"class_type": "SaveLatent", "inputs": {"samples": ["20", 0], "filename_prefix": "diag-lat-enc33"}},
"21": {"class_type": "VAEDecode", "inputs": {"samples": ["20", 0], "vae": ["8", 0]}},
"16": {"class_type": "SaveWEBM", "inputs": {"images": ["21", 0], "filename_prefix": f"diag-{variant}",
"codec": "vp9", "fps": 16.0, "crf": 32.0}},
}
L = 33

if variant.startswith("vaescale"):
# Round trip WITH the GPU resize in front: LoadImage -> ImageScale 640x640
# (comfy.utils.common_upscale, same call WanImageToVideo makes) -> encode
# -> decode -> SaveImage. Isolates the interpolate kernel.
g = {
"8": {"class_type": "VAELoader", "inputs": {"vae_name": "wan_2.1_vae.safetensors"}},
"9": {"class_type": "LoadImage", "inputs": {"image": image}},
"19": {"class_type": "ImageScale", "inputs": {"image": ["9", 0], "upscale_method": "bilinear",
"width": 640, "height": 640, "crop": "center"}},
"20": {"class_type": "VAEEncode", "inputs": {"pixels": ["19", 0], "vae": ["8", 0]}},
"21": {"class_type": "VAEDecode", "inputs": {"samples": ["20", 0], "vae": ["8", 0]}},
"16": {"class_type": "SaveImage", "inputs": {"images": ["21", 0], "filename_prefix": f"diag-{variant}"}},
}
L = 1

if "savelat" in variant:
# Dump the latent after each sampler stage (and the I2V node's own output)
# so the stage that first produces NaN/inf can be named, not inferred.
g["30"] = {"class_type": "SaveLatent", "inputs": {"samples": ["12", 2], "filename_prefix": "diag-lat-input"}}
g["31"] = {"class_type": "SaveLatent", "inputs": {"samples": ["13", 0], "filename_prefix": "diag-lat-hi"}}
g["32"] = {"class_type": "SaveLatent", "inputs": {"samples": ["14", 0], "filename_prefix": "diag-lat-lo"}}

if "noimage" in variant:
# I2V model, I2V node, but NO start image: WanImageToVideo then builds a
# zero concat latent + mask, so the model's 36-channel input path runs
# with no image content. Structured => the path works and the image
# latent VALUES break it; flat => the 36-channel path itself is broken.
g["12"]["inputs"].pop("start_image", None)
g.pop("9", None)

if "hionly" in variant:
# Single-stage: the high-noise model runs every step. Two 14B UNets do not
# fit the card at once under --highvram (13.6 GB each + encoder), but one
# does -- enough to answer "does this weight path produce NaN or not".
for k in ("2", "4", "6", "14"):
g.pop(k, None)
g["13"]["inputs"].update({"end_at_step": 10000, "return_with_leftover_noise": "disable"})
g["15"]["inputs"]["samples"] = ["13", 0]

def get(url):
return json.loads(urllib.request.urlopen(url, timeout=20).read())

def vram():
best = 0
import glob
for c in glob.glob("/sys/class/drm/card*/device/mem_info_vram_used"):
best = max(best, int(open(c).read()))
return best / 2**30

try:
urllib.request.urlopen("http://127.0.0.1:8080/unload", timeout=20).read()
except Exception:
pass
# Clean card per variant: under --highvram the previous run's UNets stay
# resident, and the fp16 text encoder then OOMs on load -- which is a memory
# artefact, not the variable under test. A container restart is the only
# thing that reliably returns VRAM on this box.
if vram() > 3:
subprocess.run(["docker", "restart", "comfyui"], capture_output=True, timeout=180)
for _ in range(60):
time.sleep(2)
try:
urllib.request.urlopen(f"{COMFY}/system_stats", timeout=3).read(); break
except Exception:
pass
print(f"card cleared: {vram():.1f} GB held")
t0 = time.time()
req = urllib.request.Request(f"{COMFY}/prompt", data=json.dumps({"prompt": g, "client_id": "diag"}).encode(),
headers={"Content-Type": "application/json"})
j = json.loads(urllib.request.urlopen(req, timeout=60).read())
if j.get("node_errors"):
print("REJECTED", json.dumps(j["node_errors"])[:600]); sys.exit(1)
pid = j["prompt_id"]; peak = 0
while time.time() - t0 < 900:
time.sleep(3); peak = max(peak, vram())
try:
h = get(f"{COMFY}/history/{pid}")
except Exception:
h = {}
if pid in h:
break
e = h.get(pid, {}); st = (e.get("status") or {}).get("status_str")
files = [im["filename"] for o in (e.get("outputs") or {}).values() for im in (o.get("images") or [])]
print(f"variant={variant} status={st} elapsed={time.time()-t0:.0f}s peak_vram={peak:.1f}GB files={files}")
if st != "success":
print("ERR", json.dumps((e.get("status") or {}).get("messages"))[:800]); sys.exit(2)
# structure score: stddev of 3 frames
scores = []
for n in (0, L // 2, L - 1):
png = subprocess.run(["ffmpeg", "-v", "error", "-i", f"{OUT}/{files[0]}", "-vf", f"select=eq(n\\,{n})",
"-frames:v", "1", "-f", "image2pipe", "-vcodec", "png", "-"], capture_output=True).stdout
from PIL import Image, ImageStat
im = Image.open(io.BytesIO(png)).convert("L")
scores.append(ImageStat.Stat(im).stddev[0])
print(f"structure score (stddev of frames 0/mid/last): {[round(s,1) for s in scores]} -> "
f"{'FLAT (broken)' if max(scores) < 8 else 'STRUCTURED (real image)'} "
f"(the lighthouse start frame alone scores ~55; a decayed flat frame < 4)")

```

Start by running diag14b.py and compare the T2V, no-start-image I2V, and real-start-image I2V paths through WanImageToVideo and the two KSamplerAdvanced nodes. Inspect the saved latents after WanImageToVideo and each sampler, checking for non-finite values across repeated runs. Done means the 14B I2V path with a start image produces finite latents and non-flat video reliably on the reported AMD environment.

Written by the indexing model from the issue text.