Comfy-Org / Comfy-Org/comfy-kitchen

Memory access fault using Krea 2 INT8 CONVROT with Triton backend on AMD gfx1151

Open
#64 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
220
Forks
91
Avg merge
1d 7h
Merged PRs (30d)
12

Description

Hi there,

I'm testing INT8 models using ROCm and Triton backend. In this scenario, using Krea 2 leads me to a memory fault crash. This only happens when enabling Triton backend.

Tried using Flash Attention, Pytorch Cross Attention and Sub Quad Optimization, same result.

Disabling the Triton backend aparently works fine, just terribly slow, painfully slower than FP8.

Model used: https://huggingface.co/Comfy-Org/Krea-2/blob/main/diffusion_models/krea2_turbo_int8_convrot.safetensors

[ 5603.532594] amdgpu 0000:c4:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:94)
[ 5603.532595] amdgpu 0000:c4:00.0: Process python3 pid 135257 thread python3 pid 135257
[ 5603.532596] amdgpu 0000:c4:00.0: in page starting at address 0x00007f4d8a007000 from client 10
[ 5603.532598] amdgpu 0000:c4:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00801031
[ 5603.532598] amdgpu 0000:c4:00.0: Faulty UTCL2 client ID: TCP (0x8)
[ 5603.532599] amdgpu 0000:c4:00.0: MORE_FAULTS: 0x1
[ 5603.532600] amdgpu 0000:c4:00.0: WALKER_ERROR: 0x0
[ 5603.532601] amdgpu 0000:c4:00.0: PERMISSION_FAULTS: 0x3
[ 5603.532602] amdgpu 0000:c4:00.0: MAPPING_ERROR: 0x0
[ 5603.532603] amdgpu 0000:c4:00.0: RW: 0x0

ComfyUI logs:

[comfyui.log](https://github.com/user-attachments/files/29736227/comfyui.log)

Any extra analysis, let me know, happy to provide information.

ROCM version: 7.2.0
Triton version: 3.7.1
Comfy Kitchen version: 0.2.16
ComfyUI version: 0.27.0

Footnote:
I'm also using INT8 CLIP for text generation, and this seems to work fine. Probably 20% faster than FP8. It doesn't crash.

INT8 Clip used: https://huggingface.co/Winnougan/Comfy-Qwen3-VL-INT8/blob/main/qwen3vl_4b_int8_convrot.safetensors
FP8 Clip used: https://huggingface.co/Comfy-Org/Qwen3-VL/blob/8bb77f7ad334bc594cd68c0c48e868abd1e0f719/text_encoders/qwen3vl_4b_fp8_scaled.safetensors

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the crash with the Krea 2 INT8 CONVROT model, Triton enabled, and the listed ROCm, Triton, Comfy Kitchen, and ComfyUI versions; review the attached ComfyUI log alongside the amdgpu fault. Compare the same model with Triton disabled and confirm that the crash is isolated to the Triton path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.