invoke-ai / invoke-ai/InvokeAI

Documentation: amdgpu lowered lockup_timeout in kernel 7.x affects InvokeAI

Open
#9,539 8 comments 0 reactions 1 assignee Claimed by @lstein View on GitHub
bug
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

### Is there an existing issue for this problem?

- [x] I have searched the existing issues

### Install method

Invoke's Launcher

### Operating system

Linux

### GPU vendor

AMD (ROCm)

### GPU model

Radeon R9700 AI PRO

### GPU VRAM

32

### Version number

v6.13.6

### Browser

_No response_

### System Information

Irrelevant

### What happened

I started getting some repeatable GPU timeouts on AMD:

```
ring gfx_0.0.0 timeout
Process Xorg ... Xorg:cs0
Starting gfx_0.0.0 ring reset
Ring gfx_0.0.0 reset succeeded
[drm] device wedged, but recovered through reset
```

The screen would freeze and not recover. InvokeAI was still running in the background until it finished.
I could SSH into the machine and `sudo service restart lightdm` to recover.

But I noticed InvokeAI was fine on Kernel 6.17, though the screen would lock up for roughly 2 seconds while finishing VAEs (which was at the time the system would freeze in 7.0).

The culprit?

[lockup_timeout](https://lists.freedesktop.org/archives/amd-gfx/2026-February/138720.html) was lowered from 10 seconds to 2 seconds:

6.17
```
modinfo amdgpu | grep -E 'lockup_timeout|timeout_period|timeout_fatal_disable|gpu_recovery'
parm: lockup_timeout:GPU lockup timeout in ms (default: 10000 for all jobs. 0: keep default value. negative: infinity timeout), format: for bare metal [Non-Compute] or [GFX,Compute,SDMA,Video]; for passthrough or sriov [all jobs] or [GFX,Compute,SDMA,Video]. (string)
parm: gpu_recovery:Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto) (int)
parm: timeout_fatal_disable:disable watchdog timeout fatal error (false = default) (bool)
parm: timeout_period:watchdog timeout period (0 = timeout disabled, 1 ~ 0x23 = timeout maxcycles = (1 << period) (uint)
```

7.0
```
parm: lockup_timeout:GPU lockup timeout in ms (default: 2000. 0: keep default value. negative: infinity timeout), format: [single value for all] or [GFX,Compute,SDMA,Video]. (string)
parm: gpu_recovery:Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto) (int)
parm: timeout_fatal_disable:disable watchdog timeout fatal error (false = default) (bool)
parm: timeout_period:watchdog timeout period (0 = timeout disabled, 1 ~ 0x23 = timeout maxcycles = (1 << period) (uint)
```

It may be relevant to document that AMDGPU users on Linux should set `amdgpu.lockup_timeout` kernel parameter back to something higher like 10 seconds or else InvokeAI may "crash" the Xorg server.

Technically this is not InvokeAI's fault, but [it is customary on Windows for commercial software](https://experienceleague.adobe.com/en/docs/substance-3d-painter/using/technical-support/technical-issues/gpu-issues/gpu-drivers-crash-with-long-computations-tdr-crash) to explain that TDR setting needs to be increased given that it's relevant to the application.

### What you expected to happen

The system to continue working normally.

### How to reproduce the problem

Just run any workload with SDXL at around 1320x1320 on Kernel 7.0 and the display will freeze.

### Additional context

_No response_

### Discord username

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.