kohya-ss / kohya-ss/sd-scripts

[Feature Request] Add option to disable cuDNN/MIOpen backend (workaround for AMD RDNA 4 GPU crashes)

Open
#2,417 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

### Summary
On AMD RDNA4 GPUs (gfx1200, e.g. Radeon RX 9060 XT), running VAE encoding during latent caching (`cache_latents`) causes severe GPU-level failures: desktop compositor (KDE) rendering glitches, full system hangs, and GPU resets. This is not limited to sd-scripts — the same crash occurs in ComfyUI when its `COMFYUI_ENABLE_MIOPEN=1` option is enabled, and does not occur when disabled.

### Root cause
Potentially related issue (while the Windows issue is reported as fixed in the nightly build, I am still affected by it, as it continues to crash in my Linux testing environment):
- https://github.com/ROCm/TheRock/issues/1542
- https://github.com/ROCm/rocm-libraries/issues/1860

### Proposed solution
The identified workaround is to completely disable the cuDNN/MIOpen backend by inserting the following code immediately after `import torch` in the script:
```python
torch.backends.cudnn.enabled = False
```
(On ROCm, PyTorch routes convolution through the cuDNN-compatible API, which maps to MIOpen — so this flag is MIOpen's effective on/off switch.)

~~I confirmed manually that patching this into the training script eliminates the crash/hang/glitch completely.~~

EDIT 1: Sorry about that. It seemed fine during the first few attempts, but upon further testing, the graphical glitches persisted. At the very least, there is definitely an issue occurring around VAE processing in my environment... However, it is also true that enabling this option has reduced the number of system hangs. (To be honest, the behavior is inconsistent, so I'm not entirely sure what's going on... but I hope this information helps someone.)

EDIT 2: After recreating the venv from scratch and reinstalling the torch-related components, it stopped crashing even with MIOpen enabled...
The issue is resolved in ComfyUI as well... I wonder why? Perhaps a series of unfortunate, temporary hardware glitches coincided?
In any case, since this is considered a useful option for scenarios where crashes occur with MIOpen enabled, I will leave this feature request open.

## Environment
- Arch Linux, kernel 7.1.8 (2026/08/13 updated)
- GPU: AMD Radeon RX 9060 XT (gfx1200, RDNA 4, 16GB VRAM)
- ROCm / PyTorch: 2.13.0+rocm7.2

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the training script and the cache_latents/VAE-processing entry points mentioned in the report, then inspect how backend options are currently configured. Add a user-facing way to disable the cuDNN/MIOpen backend and verify the default behavior remains unchanged, including on ROCm environments where the workaround is needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.