Comfy-Org / Comfy-Org/comfy-kitchen

[Bug] GPU LOST (driver device lost) with comfy kitchen attention (INT8) on NVIDIA RTX 5060 Ti (sm_120), Windows -- same workflow with native attention is stable

Open
#120 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
220
Forks
91
Avg merge
1d 7h
Merged PRs (30d)
12

Description

### Summary

Using `comfy kitchen attention` (`ModelAttentionBackend` -> `comfy_kitchen_int8`) on an NVIDIA GeForce RTX 5060 Ti (Blackwell, **sm_120**) under Windows causes a **driver-level GPU LOST** (`No devices were found` / `GPU is lost. Reboot the system to recover this GPU`) ~2 seconds after compute starts.

The **exact same workflow** with `pytorch attention` completes successfully. This is a clean single-variable control: only the attention backend differs.

### Environment

- OS: Windows 11 Pro (build 26200)
- GPU: NVIDIA GeForce RTX 5060 Ti 16GB (Blackwell, sm_120)
- Driver / CUDA UMD: 610.88 / 13.3
- torch: 2.12.0+cu130
- triton-windows: 3.7.1.post27
- comfy-kitchen: 0.2.31 (PyPI latest)
- comfy-aimdo: 0.4.13
- ComfyUI: started full-speed (`main.py --listen --enable-manager`, no stable-profile / offload args)

### Reproduction

MiniMax H3 image-to-video workflow, 864x480, **243 frames, 6 steps** (BasicScheduler simple / denoise 1 / euler), 8-step LoRA. Node `ModelAttentionBackend` attention = `comfy kitchen attention` (i.e. `comfy_kitchen_int8`).

The GPU LOST occurs ~2 s into the **first compute burst** (during MiniMaxH3TEModel weight staging / very first attention), before sampling begins.

### Symptom

- `nvidia-smi` then reports `No devices were found` / `Unable to determine the device handle for GPU0: 0000:07:00.0: GPU is lost. Reboot the system to recover this GPU`.
- **Not** a 0x141 BSOD: no new minidump in `C:\Windows\Minidump`.
- No `Kernel-Power 41` / `EventLog 6008`: the recovery reboot was initiated normally by our watchdog (Event 1074), not an unclean power event.
- comfyui.log tail shows `aimdo: src/model-vbar.c:403:ERROR:VRAM Allocation failed` — this appears *after* the device was lost (symptom, not cause).
- At the moment of loss: **79 °C, 140 W / 180 W power limit, 100% GPU util, ~9.4 GB / 16 GB VRAM**. Thermal / power / VRAM-exhaustion explanations are excluded.

### Single-variable control

Same workflow, Node `ModelAttentionBackend` attention = `pytorch attention` (the only change, verified by diff):

- Completes: `Prompt executed in 337.10 seconds`, MP4 produced.
- Peak: **15331 MiB VRAM**, 182.9 W (transient), 79 °C, sustained 100% util.
- **No GPU LOST.**

So native attention is stable under *higher* memory/power load, while INT8 ck-attention loses the device under *lighter* load.

### Questions / notes

- Is NVIDIA / Blackwell (sm_120) a supported/tested target for ck-attention? Issue #110 notes the INT8 precondition is enforced identically in the CUDA backend, but our failure is a **driver device-lost**, not a Python exception.
- Happy to run debug builds or share the full 1-second telemetry CSV, ComfyUI console log, and Windows event log if useful.

### Evidence

- 1-second telemetry CSV capturing the loss moment (79 °C / 140 W / 100% at loss)
- ComfyUI console log
- Windows System event log (Event 1074 = watchdog-initiated reboot)

Contributor guide

Open the contributing guide

Research direction

Start with the ModelAttentionBackend entry point and compare comfy_kitchen_int8 with pytorch attention using the documented MiniMax H3 workflow on the RTX 5060 Ti. Review the CUDA backend context from issue #110 and the reported src/model-vbar.c:403 allocation message, confirming whether it follows device loss. Done means identifying a reproducible compatibility cause or validated fix without GPU LOST.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.