Comfy-Org / Comfy-Org/comfy-aimdo
[ROCm/Windows] DynamicVRAM: gray image output with quantized models (INT4/GGUF); GGUF hangs before sampling
- Dominant language
- C
- Stars
- 67
- Forks
- 39
- Avg merge
- 1d 25m
- Merged PRs (30d)
- 10
Description
## Environment
- GPU: AMD Radeon RX 7900 XT (20GB), Windows 11
- torch: 2.9.1+rocm7.2.1 / ROCm 7.2.1
- ComfyUI: 0.30.0 (requirements.txt pins comfy-aimdo==0.4.11)
- comfy-aimdo: tested **0.4.11** and **0.4.13** (latest)
## Summary
With `--enable-dynamic-vram` on ROCm/Windows, DynamicVRAM initializes successfully
(log: `comfy-aimdo inited for GPU: AMD Radeon RX 7900 XT (VRAM: 20464 MB)`), but
**quantized models produce wrong output or hang**:
### 1) Quantized safetensors model (MiniMaxH3, INT4 `convrot_w4a4`, MixedPrecisionOps)
→ output is a **completely gray image** (constant latent).
- 0.4.11: sampler 10 steps in <1s (10.35 it/s) — no real compute at all.
- 0.4.13: sampler 10 steps in ~5s (1.74 it/s) — real compute now, but output **still gray**.
- No errors/exceptions in either version (0.4.13 added native log routing; nothing logged).
### 2) GGUF model (same MiniMaxH3 arch, qtypes F32/BF16/Q4_0/Q4_K, sd.cpp compat mode)
→ sampler **never starts step 1**: dedicated VRAM fills to ~19.4GB, GPU compute ~0%,
ComfyUI UI freezes for 5+ minutes (OS itself is responsive).
- This model loads via the **legacy path** (`loaded partially; 16569 MB loaded`),
i.e. it does NOT use ModelPatcherDynamic — but aimdo's CUDA detour hooks
(6 hooks) are still installed at the torch level.
## Diagnostics performed (inside a running ComfyUI process)
Minimal hostbuf → vbar → device roundtrip test with aimdo already initialized:
- HostBuffer write/read: consistent ✅
- `vbar_fault`: OK ✅
- copy pattern bytes to vbar device tensor, read back: **identical** ✅ (`roundtrip_equal: true`)
So the low-level data path (HostBuffer → VBAR → VRAM) is **correct on ROCm**
for plain byte copies.
## Suspected root cause
Issue appears specific to **quantized weights** interacting with dynamic loading:
- In `ModelPatcherDynamic.load()` (`model_patcher.py`), for `QuantizedTensor` weights the
geometry passed to vbar allocation is the weight object itself, whereas non-quantized
weights get a `TensorGeometry` — possible incorrect size/layout accounting.
- GGUF case: aimdo's CUDA detour hooks may interfere with the **legacy** (non-dynamic)
load/compute path even though the model never uses ModelPatcherDynamic.
## Questions
1. Is ROCm officially supported for quantized weights with DynamicVRAM?
2. Any known workaround for quantized models on ROCm?
Happy to provide full logs or run additional diagnostics on request.
Contributor guide
Research direction
Start with ModelPatcherDynamic.load() in model_patcher.py, comparing the QuantizedTensor geometry path with the TensorGeometry path, then reproduce the gray-output case. Also trace the six aimdo CUDA detour hooks against the GGUF legacy loading path. Done means quantized models produce non-gray output and GGUF reaches sampling without freezing on ROCm/Windows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100