Comfy-Org / Comfy-Org/comfy-kitchen
[Bug] GPU LOST (driver device lost) with comfy kitchen attention (INT8) on NVIDIA RTX 5060 Ti (sm_120), Windows -- same workflow with native attention is stable
- Dominant language
- Python
- Stars
- 220
- Forks
- 91
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 12
Description
### Summary
Using `comfy kitchen attention` (`ModelAttentionBackend` -> `comfy_kitchen_int8`) on an NVIDIA GeForce RTX 5060 Ti (Blackwell, **sm_120**) under Windows causes a **driver-level GPU LOST** (`No devices were found` / `GPU is lost. Reboot the system to recover this GPU`) ~2 seconds after compute starts.
The **exact same workflow** with `pytorch attention` completes successfully. This is a clean single-variable control: only the attention backend differs.
### Environment
- OS: Windows 11 Pro (build 26200)
- GPU: NVIDIA GeForce RTX 5060 Ti 16GB (Blackwell, sm_120)
- Driver / CUDA UMD: 610.88 / 13.3
- torch: 2.12.0+cu130
- triton-windows: 3.7.1.post27
- comfy-kitchen: 0.2.31 (PyPI latest)
- comfy-aimdo: 0.4.13
- ComfyUI: started full-speed (`main.py --listen --enable-manager`, no stable-profile / offload args)
### Reproduction
MiniMax H3 image-to-video workflow, 864x480, **243 frames, 6 steps** (BasicScheduler simple / denoise 1 / euler), 8-step LoRA. Node `ModelAttentionBackend` attention = `comfy kitchen attention` (i.e. `comfy_kitchen_int8`).
The GPU LOST occurs ~2 s into the **first compute burst** (during MiniMaxH3TEModel weight staging / very first attention), before sampling begins.
### Symptom
- `nvidia-smi` then reports `No devices were found` / `Unable to determine the device handle for GPU0: 0000:07:00.0: GPU is lost. Reboot the system to recover this GPU`.
- **Not** a 0x141 BSOD: no new minidump in `C:\Windows\Minidump`.
- No `Kernel-Power 41` / `EventLog 6008`: the recovery reboot was initiated normally by our watchdog (Event 1074), not an unclean power event.
- comfyui.log tail shows `aimdo: src/model-vbar.c:403:ERROR:VRAM Allocation failed` — this appears *after* the device was lost (symptom, not cause).
- At the moment of loss: **79 °C, 140 W / 180 W power limit, 100% GPU util, ~9.4 GB / 16 GB VRAM**. Thermal / power / VRAM-exhaustion explanations are excluded.
### Single-variable control
Same workflow, Node `ModelAttentionBackend` attention = `pytorch attention` (the only change, verified by diff):
- Completes: `Prompt executed in 337.10 seconds`, MP4 produced.
- Peak: **15331 MiB VRAM**, 182.9 W (transient), 79 °C, sustained 100% util.
- **No GPU LOST.**
So native attention is stable under *higher* memory/power load, while INT8 ck-attention loses the device under *lighter* load.
### Questions / notes
- Is NVIDIA / Blackwell (sm_120) a supported/tested target for ck-attention? Issue #110 notes the INT8 precondition is enforced identically in the CUDA backend, but our failure is a **driver device-lost**, not a Python exception.
- Happy to run debug builds or share the full 1-second telemetry CSV, ComfyUI console log, and Windows event log if useful.
### Evidence
- 1-second telemetry CSV capturing the loss moment (79 °C / 140 W / 100% at loss)
- ComfyUI console log
- Windows System event log (Event 1074 = watchdog-initiated reboot)
Contributor guide
Research direction
Start with the ModelAttentionBackend entry point and compare comfy_kitchen_int8 with pytorch attention using the documented MiniMax H3 workflow on the RTX 5060 Ti. Review the CUDA backend context from issue #110 and the reported src/model-vbar.c:403 allocation message, confirming whether it follows device loss. Done means identifying a reproducible compatibility cause or validated fix without GPU LOST.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100