invoke-ai / invoke-ai/InvokeAI
Documentation: amdgpu lowered lockup_timeout in kernel 7.x affects InvokeAI
- Dominant language
- Python
- Stars
- 28.2k
- Forks
- 3k
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 19
Description
### Is there an existing issue for this problem?
- [x] I have searched the existing issues
### Install method
Invoke's Launcher
### Operating system
Linux
### GPU vendor
AMD (ROCm)
### GPU model
Radeon R9700 AI PRO
### GPU VRAM
32
### Version number
v6.13.6
### Browser
_No response_
### System Information
Irrelevant
### What happened
I started getting some repeatable GPU timeouts on AMD:
```
ring gfx_0.0.0 timeout
Process Xorg ... Xorg:cs0
Starting gfx_0.0.0 ring reset
Ring gfx_0.0.0 reset succeeded
[drm] device wedged, but recovered through reset
```
The screen would freeze and not recover. InvokeAI was still running in the background until it finished.
I could SSH into the machine and `sudo service restart lightdm` to recover.
But I noticed InvokeAI was fine on Kernel 6.17, though the screen would lock up for roughly 2 seconds while finishing VAEs (which was at the time the system would freeze in 7.0).
The culprit?
[lockup_timeout](https://lists.freedesktop.org/archives/amd-gfx/2026-February/138720.html) was lowered from 10 seconds to 2 seconds:
6.17
```
modinfo amdgpu | grep -E 'lockup_timeout|timeout_period|timeout_fatal_disable|gpu_recovery'
parm: lockup_timeout:GPU lockup timeout in ms (default: 10000 for all jobs. 0: keep default value. negative: infinity timeout), format: for bare metal [Non-Compute] or [GFX,Compute,SDMA,Video]; for passthrough or sriov [all jobs] or [GFX,Compute,SDMA,Video]. (string)
parm: gpu_recovery:Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto) (int)
parm: timeout_fatal_disable:disable watchdog timeout fatal error (false = default) (bool)
parm: timeout_period:watchdog timeout period (0 = timeout disabled, 1 ~ 0x23 = timeout maxcycles = (1 << period) (uint)
```
7.0
```
parm: lockup_timeout:GPU lockup timeout in ms (default: 2000. 0: keep default value. negative: infinity timeout), format: [single value for all] or [GFX,Compute,SDMA,Video]. (string)
parm: gpu_recovery:Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto) (int)
parm: timeout_fatal_disable:disable watchdog timeout fatal error (false = default) (bool)
parm: timeout_period:watchdog timeout period (0 = timeout disabled, 1 ~ 0x23 = timeout maxcycles = (1 << period) (uint)
```
It may be relevant to document that AMDGPU users on Linux should set `amdgpu.lockup_timeout` kernel parameter back to something higher like 10 seconds or else InvokeAI may "crash" the Xorg server.
Technically this is not InvokeAI's fault, but [it is customary on Windows for commercial software](https://experienceleague.adobe.com/en/docs/substance-3d-painter/using/technical-support/technical-issues/gpu-issues/gpu-drivers-crash-with-long-computations-tdr-crash) to explain that TDR setting needs to be increased given that it's relevant to the application.
### What you expected to happen
The system to continue working normally.
### How to reproduce the problem
Just run any workload with SDXL at around 1320x1320 on Kernel 7.0 and the display will freeze.
### Additional context
_No response_
### Discord username
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.