Comfy-Org / Comfy-Org/comfy-kitchen
Segfault in HIP `dequantize_per_tensor_fp8` on gfx1201 (RDNA4)
- Dominant language
- Python
- Stars
- 220
- Forks
- 91
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 12
Description
## Environment
| | |
|---|---|
| GPU | AMD Radeon AI PRO R9700, gfx1201 (RDNA4), 32 GB |
| comfy-kitchen | 0.2.31 |
| PyTorch | 2.13.0+rocm7.2 |
| ROCm | 7.2 |
| Python | 3.12.13 |
| OS | Ubuntu 22.04 (Docker, `rocm/dev-ubuntu-22.04:7.2.4-complete`) |
| ComfyUI | 0.21.1 |
## Description
Hard segfault (no Python exception, process dies) in the HIP backend's
`dequantize_per_tensor_fp8` when an fp8-quantized text encoder runs its first
forward pass.
Setting `COMFY_KITCHEN_DISABLE_HIP=1` avoids the crash entirely, and the same
workflow then completes correctly via the eager backend - so the inputs
themselves are fine.
**This is not an architecture mismatch.** comfy-kitchen was rebuilt explicitly
for the local target:
```
COMFY_KITCHEN_BUILD_HIP=1 COMFY_HIP_ARCHS=gfx1201 \
uv pip install --no-build-isolation git+https://github.com/Comfy-Org/comfy-kitchen.git
```
The crash reproduces identically before and after that rebuild.
## Steps to reproduce
1. ComfyUI on ROCm 7.2 / gfx1201, comfy-kitchen with the HIP backend active
2. Load an fp8-scaled text encoder (`qwen3vl_4b_fp8_scaled.safetensors`)
3. Run any workflow that encodes a prompt through it
4. Segfault at the first token generation step
## Backend detection at startup
HIP reports available and enabled, and lists `dequantize_per_tensor_fp8` among
its capabilities - so this is a kernel that claims support, not a fallback path:
```
Found comfy_kitchen backend hip: {'available': True, 'disabled': False,
'unavailable_reason': None, 'capabilities': [..., 'dequantize_per_tensor_fp8', ...]}
```
## Stack trace
```
Generating tokens: 0%| | 0/512 [00:00
File ".../comfy_kitchen/tensor/base.py", line 386 in dequantize_args
File ".../comfy_kitchen/tensor/fp8.py", line 145 in _handle_fp8_linear
File ".../comfy_kitchen/tensor/base.py", line 362 in __torch_dispatch__
File "/ComfyUI/comfy/ops.py", line 1328 in _forward
File "/ComfyUI/comfy/ops.py", line 1345 in forward_comfy_cast_weights
File "/ComfyUI/comfy/ops.py", line 1414 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 554 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 673 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 914 in core
File "/ComfyUI/comfy/model_prefetch.py", line 66 in prefetch_queue_pop
File "/ComfyUI/comfy/text_encoders/llama.py", line 924 in forward
File "/ComfyUI/comfy/text_encoders/llama.py", line 1052 in generate
File "/ComfyUI/comfy/text_encoders/qwen3vl.py", line 136 in generate
File "/ComfyUI/comfy/sd1_clip.py", line 750 in generate
File "/ComfyUI/comfy/sd.py", line 477 in generate
File "/ComfyUI/comfy_extras/nodes_textgen.py", line 64 in execute
...
```
## Workaround
`COMFY_KITCHEN_DISABLE_HIP=1`
Contributor guide
Research direction
Start at comfy_kitchen/backends/hip/__init__.py line 310, then compare the eager path at comfy_kitchen/backends/eager/quantization.py line 461 and reproduce with COMFY_KITCHEN_DISABLE_HIP unset on gfx1201. Done means the fp8 text-encoder workflow completes without a segmentation fault while the HIP backend remains enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100