Comfy-Org / Comfy-Org/ComfyUI

[Bug] Comfy Compiler CUDA graphs silently corrupt MiniMax-Music-3 AR text-encoder output on RDNA4 (gfx1201)

Open
#16,222 2 comments 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

## Expected Behavior
Encoding the same prompt+seed with the MiniMax-Music-3 text encoder must
produce identical conditioning whether CUDA graphs are on or off. The framework
itself reports `graph breaks: 0, rogues: 0` on every run, so replay is expected
to be numerically faithful.

### Actual Behavior

With the Comfy Compiler's CUDA graphs enabled (default in v0.35.0), the AR
text-encoder conditioning comes out completely different from the graphs-off
reference, with zero errors or warnings:

| Config (same seed 197122968890040, same prompt, 15s) | cond_0 stats |
|---|---|
| graphs OFF (`--disable-cuda-graphs`) | mean −0.0039, std 0.9555, maxabs 132.0 |
| graphs ON (default) | mean −0.0022, std 1.0364, maxabs 83.0 |

maxabs_diff = 123.4 on tensors with maxabs ~132 (≈93% relative corruption).
The corrupted run logged **13,838 cudagraph replays, breaks=0, rogues=0**.
End to end (official `audio_minimax_music_3` template workflow), generations
complete normally but the audio is incoherent. `--disable-cuda-graphs` alone
fully fixes it (additionally confirmed with `--disable-comfy-compiler`).

Bisected so far:
- Model files verified bit-perfect (sizes + SHA256 vs HF LFS oids for
`minimax_music3_dit_fp16`, `minimax_music3_text_encoder_pruned_int8_convrot`,
`minimax_music3_dav`).
- Workflow verified node-for-node against the official template (flat clone,
no custom nodes — custom nodes also disabled for these runs).
- DynamicVRAM innocent: legacy vs DynamicVRAM-no-graphs conditioning is
bit-identical (diff 0.0); only graph replay diverges.
- GPU healthy: fp16 matmul vs CPU reference matches, no driver crashes/TDRs.
- Suspect scope: per-block `torch.cuda.CUDAGraph` replay in the Qwen AR loop
(`comfy/ldm/minimax_music/ar.py` via `prefetch_queue_pop(enable_graph=True)`).
The AR path does pass `generator=` for RNG registration, so this is not the
naive frozen-noise case. DiT-side graphs not yet isolated (AR fell first).

```powershell
$env:HIP_VISIBLE_DEVICES="1"
python repro_music3_cudagraph.py --mode encode --duration 15 --out cond_on.pt
python repro_music3_cudagraph.py --mode encode --duration 15 --out cond_off.pt -- --disable-cuda-graphs
python repro_music3_cudagraph.py --mode compare --compare cond_on.pt cond_off.pt
# expected: maxabs_diff == 0 ; observed on gfx1201: maxabs_diff ~= 123.4
```

GUI repro (what I hit first): official template
`audio_minimax_music_3` (60s, seed 197122968890040), queue twice — once
default flags (incoherent audio), once with `--disable-cuda-graphs` (correct
audio). Requires AMD RDNA4 for the corruption itself; the tensor-compare
procedure above is the portable check.

### Steps to Reproduce

[repro_music3_cudagraph.py](https://github.com/user-attachments/files/32037583/repro_music3_cudagraph.py)

No custom nodes needed. Headless repro script attached
(`repro_music3_cudagraph.py`, drives ComfyUI core as a library; needs the
MiniMax-Music-3 text-encoder file + the official template prompt):

### Debug Logs

```powershell
Full `comfyui.log` attached. Key excerpts:

ComfyUI version: 0.35.0
Device: cuda:0 AMD Radeon RX 9070 XT : native
Total VRAM 16304 MB, total RAM 31861 MB
Found comfy_kitchen backend hip: {'available': True, ... 'int8_linear', ...}
Found comfy_kitchen backend triton: {'available': False, 'disabled': True, ...}
DynamicVRAM support detected and enabled
Requested to load MiniMaxMusic3TEModel
Model MiniMaxMusic3TEModel prepared for dynamic VRAM loading. 8758MB Staged.
...
Comfy model compiler graph breaks: 0, rogues: 0 # <- on corrupted runs too
```

### Other

[comfyui_music3_bad_runs.log](https://github.com/user-attachments/files/32037575/comfyui_music3_bad_runs.log)

[official_prompt.json](https://github.com/user-attachments/files/32037587/official_prompt.json)

[minimax_music3_baseline_fp16.json](https://github.com/user-attachments/files/32037590/minimax_music3_baseline_fp16.json)

- Environment: Windows, Comfy Desktop, RX 9070 XT (gfx1201) + iGPU present but
isolated via HIP_VISIBLE_DEVICES, ROCm 7.14, torch 2.12.0+rocm7.14.0,
ComfyUI v0.35.0 (40c4fcdf). Regression vs pre-v0.35.0 (Comfy Compiler is new
in v0.35.0). Unknown whether other ROCm arches are affected.
- Model backend note: the corrupted path is the int8-convrot Qwen AR decoder
served through the `hip` comfy-kitchen backend (triton backend unavailable:
no triton module installed).
- Attachments: `repro_music3_cudagraph.py`, full `comfyui.log`.
(I can also provide the 24MB `.pt` conditioning dumps on request.)
- Related (checked, none is this bug):
- PR #16180 regression notes already document graph-capture output corruption
on AMD ROCm (Gemma, R9700, Linux) — same bug class, different model/arch/OS.
This report adds Music3 AR + gfx1201 + Windows with numeric proof.
- #16002 is Music3 + graphs but GGUF loader on NVIDIA with a loud pinned-memory
crash — different failure, different stack.
- #15597 is Music3 graph-lifetime warnings on RTX 5090 with no audible failure.
- #15671 (audio degrades ~3s in, RTX 5090) may or may not share a root cause;
different GPU vendor and symptom shape.
- Disclosure: diagnosis (bisect + repro script) was AI-assisted; every run,
log, and the fix verification are mine on the hardware above.

Contributor guide

Open the contributing guide

Research direction

Start with comfy/ldm/minimax_music/ar.py, especially the Qwen AR path around prefetch_queue_pop(enable_graph=True), and review the attached repro_music3_cudagraph.py. Run the encode and compare modes with CUDA graphs enabled and disabled on the reported setup. Done means the conditioning tensors match, with maxabs_diff == 0, and the Music3 workflow remains coherent with graphs enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.