Comfy-Org / Comfy-Org/comfy-aimdo

[ROCm/Windows] DynamicVRAM: gray image output with quantized models (INT4/GGUF); GGUF hangs before sampling

Open
#82 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
67
Forks
39
Avg merge
1d 25m
Merged PRs (30d)
10

Description

## Environment
- GPU: AMD Radeon RX 7900 XT (20GB), Windows 11
- torch: 2.9.1+rocm7.2.1 / ROCm 7.2.1
- ComfyUI: 0.30.0 (requirements.txt pins comfy-aimdo==0.4.11)
- comfy-aimdo: tested **0.4.11** and **0.4.13** (latest)

## Summary
With `--enable-dynamic-vram` on ROCm/Windows, DynamicVRAM initializes successfully
(log: `comfy-aimdo inited for GPU: AMD Radeon RX 7900 XT (VRAM: 20464 MB)`), but
**quantized models produce wrong output or hang**:

### 1) Quantized safetensors model (MiniMaxH3, INT4 `convrot_w4a4`, MixedPrecisionOps)
→ output is a **completely gray image** (constant latent).
- 0.4.11: sampler 10 steps in <1s (10.35 it/s) — no real compute at all.
- 0.4.13: sampler 10 steps in ~5s (1.74 it/s) — real compute now, but output **still gray**.
- No errors/exceptions in either version (0.4.13 added native log routing; nothing logged).

### 2) GGUF model (same MiniMaxH3 arch, qtypes F32/BF16/Q4_0/Q4_K, sd.cpp compat mode)
→ sampler **never starts step 1**: dedicated VRAM fills to ~19.4GB, GPU compute ~0%,
ComfyUI UI freezes for 5+ minutes (OS itself is responsive).
- This model loads via the **legacy path** (`loaded partially; 16569 MB loaded`),
i.e. it does NOT use ModelPatcherDynamic — but aimdo's CUDA detour hooks
(6 hooks) are still installed at the torch level.

## Diagnostics performed (inside a running ComfyUI process)
Minimal hostbuf → vbar → device roundtrip test with aimdo already initialized:
- HostBuffer write/read: consistent ✅
- `vbar_fault`: OK ✅
- copy pattern bytes to vbar device tensor, read back: **identical** ✅ (`roundtrip_equal: true`)

So the low-level data path (HostBuffer → VBAR → VRAM) is **correct on ROCm**
for plain byte copies.

## Suspected root cause
Issue appears specific to **quantized weights** interacting with dynamic loading:
- In `ModelPatcherDynamic.load()` (`model_patcher.py`), for `QuantizedTensor` weights the
geometry passed to vbar allocation is the weight object itself, whereas non-quantized
weights get a `TensorGeometry` — possible incorrect size/layout accounting.
- GGUF case: aimdo's CUDA detour hooks may interfere with the **legacy** (non-dynamic)
load/compute path even though the model never uses ModelPatcherDynamic.

## Questions
1. Is ROCm officially supported for quantized weights with DynamicVRAM?
2. Any known workaround for quantized models on ROCm?
Happy to provide full logs or run additional diagnostics on request.

Contributor guide

Open the contributing guide

Research direction

Start with ModelPatcherDynamic.load() in model_patcher.py, comparing the QuantizedTensor geometry path with the TensorGeometry path, then reproduce the gray-output case. Also trace the six aimdo CUDA detour hooks against the GGUF legacy loading path. Done means quantized models produce non-gray output and GGUF reaches sampling without freezing on ROCm/Windows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.