Comfy-Org / Comfy-Org/comfy-kitchen

Segfault in HIP `dequantize_per_tensor_fp8` on gfx1201 (RDNA4)

Open
#125 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
220
Forks
91
Avg merge
1d 7h
Merged PRs (30d)
12

Description

## Environment

| | |
|---|---|
| GPU | AMD Radeon AI PRO R9700, gfx1201 (RDNA4), 32 GB |
| comfy-kitchen | 0.2.31 |
| PyTorch | 2.13.0+rocm7.2 |
| ROCm | 7.2 |
| Python | 3.12.13 |
| OS | Ubuntu 22.04 (Docker, `rocm/dev-ubuntu-22.04:7.2.4-complete`) |
| ComfyUI | 0.21.1 |

## Description

Hard segfault (no Python exception, process dies) in the HIP backend's
`dequantize_per_tensor_fp8` when an fp8-quantized text encoder runs its first
forward pass.

Setting `COMFY_KITCHEN_DISABLE_HIP=1` avoids the crash entirely, and the same
workflow then completes correctly via the eager backend - so the inputs
themselves are fine.

**This is not an architecture mismatch.** comfy-kitchen was rebuilt explicitly
for the local target:

```
COMFY_KITCHEN_BUILD_HIP=1 COMFY_HIP_ARCHS=gfx1201 \
uv pip install --no-build-isolation git+https://github.com/Comfy-Org/comfy-kitchen.git
```

The crash reproduces identically before and after that rebuild.

## Steps to reproduce

1. ComfyUI on ROCm 7.2 / gfx1201, comfy-kitchen with the HIP backend active
2. Load an fp8-scaled text encoder (`qwen3vl_4b_fp8_scaled.safetensors`)
3. Run any workflow that encodes a prompt through it
4. Segfault at the first token generation step
## Backend detection at startup

HIP reports available and enabled, and lists `dequantize_per_tensor_fp8` among
its capabilities - so this is a kernel that claims support, not a fallback path:

```
Found comfy_kitchen backend hip: {'available': True, 'disabled': False,
'unavailable_reason': None, 'capabilities': [..., 'dequantize_per_tensor_fp8', ...]}
```

## Stack trace

```
Generating tokens: 0%| | 0/512 [00:00
File ".../comfy_kitchen/tensor/base.py", line 386 in dequantize_args
File ".../comfy_kitchen/tensor/fp8.py", line 145 in _handle_fp8_linear
File ".../comfy_kitchen/tensor/base.py", line 362 in __torch_dispatch__
File "/ComfyUI/comfy/ops.py", line 1328 in _forward
File "/ComfyUI/comfy/ops.py", line 1345 in forward_comfy_cast_weights
File "/ComfyUI/comfy/ops.py", line 1414 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 554 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 673 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 914 in core
File "/ComfyUI/comfy/model_prefetch.py", line 66 in prefetch_queue_pop
File "/ComfyUI/comfy/text_encoders/llama.py", line 924 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 1052 in generate
File "/ComfyUI/comfy/text_encoders/qwen3vl.py", line 136 in generate
File "/ComfyUI/comfy/sd1_clip.py", line 750 in generate
File "/ComfyUI/comfy/sd.py", line 477 in generate
File "/ComfyUI/comfy_extras/nodes_textgen.py", line 64 in execute
...
```

## Workaround

`COMFY_KITCHEN_DISABLE_HIP=1`

Contributor guide

Open the contributing guide

Research direction

Start at comfy_kitchen/backends/hip/__init__.py line 310, then compare the eager path at comfy_kitchen/backends/eager/quantization.py line 461 and reproduce with COMFY_KITCHEN_DISABLE_HIP unset on gfx1201. Done means the fp8 text-encoder workflow completes without a segmentation fault while the HIP backend remains enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.