Comfy-Org / Comfy-Org/comfy-kitchen
Fatal access violation (0xC0000005) running int8_tensorwise model on ROCm Windows (gfx1101 / RDNA3)
- Dominant language
- Python
- Stars
- 220
- Forks
- 91
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 12
Description
## Environment
- **GPU:** AMD Radeon RX 7700 XT (RDNA3, `gfx1101`)
- **ROCm:** `7.2` (`torch 2.9.1+rocm7.2.1`, `torch.version.hip = "7.2"`)
- **Python:** 3.12.12
- **OS:** Windows 11 (ComfyUI Desktop, standalone env)
- **ComfyUI:** nightly (post `int8_tensorwise` merge — `QUANT_ALGOS["int8_tensorwise"]` present)
- **comfy-kitchen:** 0.2.15
- **Model:** Chroma1-HD, `int8mixed_tensorwise` quant (`*_learned_int8mixed_tensorwise.safetensors`)
## Problem
Loading and running an **`int8_tensorwise`** mixed-precision diffusion model on ROCm/Windows (gfx1101) causes a fatal Windows access violation (`0xC0000005`) on the **first sampling step**. The model loads fully into VRAM without error; the crash happens the moment the int8 matmul path executes.
This is **not a catchable Python exception** — the process is killed by the OS before Python regains control, so no traceback reaches ComfyUI. The Windows fault dump shows only CPython interpreter frames (asyncio event loop driving node execution); the faulting frame is in the native int8 GEMM extension and is not symbolized.
Startup confirms the native int8 path is selected (and is the only quant op *not* emulated on this card):
```
Found comfy_kitchen backend cuda: {'available': True, 'disabled': True, ...}
Found comfy_kitchen backend triton: {'available': False, 'unavailable_reason': "ImportError: No module named 'triton'", ...}
Found comfy_kitchen backend eager: {'available': True, 'disabled': False, ...}
...
Detected mixed precision quantization
Native ops: int8_tensorwise , emulated ops: float8_e4m3fn, nvfp4, float8_e5m2, mxfp8
model_type FLUX
...
loaded completely; 10368.17 MB usable, 8767.38 MB loaded, full load: True
```
## Root cause (analysis)
On ROCm, fp8 / nvfp4 / mxfp8 are correctly pushed to the **emulated** path. But the `int8_tensorwise` GEMM (`TensorWiseINT8Layout` → eager backend) runs a native HIP kernel that segfaults on `gfx1101` — the same failure *class* as #32 (fp8 `_scaled_mm_v2` on RDNA4) and #55 (native INT8 on CUDA SM75), but on a path neither of those covers:
- **#32** guarded the **fp8** `scaled_mm_v2` path with `torch.version.hip is None` and was fixed by pytorch PR #179138 — that fix does **not** cover the int8 layout path.
- **#55** is the **int8** native backend, but CUDA SM75 — a different platform from ROCm.
So int8-on-ROCm currently has **no working native kernel, no emulation fallback, and no guard**. Relevant: in ComfyUI core `comfy/ops.py::pick_operations`, the `disabled` set is built from `supports_fp8_compute` / `supports_nvfp4_compute` / `supports_mxfp8_compute`. There is **no `supports_int8_compute` check**, so `int8_tensorwise` is never added to `disabled`, is always dispatched native, and has no CLI/config toggle to force emulation (unlike fp8/fp4).
## Things tried (no effect)
- `TORCH_BLAS_PREFER_HIPBLASLT=0`
- Updating ComfyUI + comfy-kitchen to latest nightly (this is what enabled the format in the first place)
## Works correctly on the same system
- The **fp8mixed** variant of the same model (`Chroma1-HD-fp8mixed.safetensors`) — fp8 runs through the emulated path and generates fine.
## Expected behavior
On hardware where the native int8 kernel is unavailable/broken, `int8_tensorwise` should either:
1. Route through a working path (analogous to the #32 hip-guard / `torch._scaled_mm` fallback), or
2. Fall back to emulation (which would require a `supports_int8_compute`-style gate in ComfyUI core so the format can be added to `disabled`), or
3. At minimum, raise an explicit "int8 unsupported on this device" error instead of a hard `0xC0000005` segfault.
## Actual behavior
Hard native access violation (`0xC0000005`) on the first sampling step; the whole ComfyUI process dies and cannot be caught or recovered.
Contributor guide
Research direction
Start in comfy/ops.py::pick_operations and trace TensorWiseINT8Layout through the eager backend, comparing the existing fp8 guard in #32 and the native INT8 case in #55. Reproduce the first sampling step on ROCm gfx1101, then verify that unsupported int8 avoids the native crash through a fallback or explicit error and that the existing working paths remain unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100