NVIDIA / NVIDIA/open-gpu-kernel-modules

Xid 109 (GSP CTX SWITCH TIMEOUT) on RTX 5060 Ti (Blackwell) — GPU lockup requiring reboot

Open
#1,243 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
17.4k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

NVIDIA Open GPU Kernel Modules Version

nvidia-open-dkms 610.43.03 (also reproduced on 610.43.02) — Arch Linux package version, not built from a specific commit/SHA.

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
  • I confirm that this does not happen with the proprietary driver package.
Operating System and Version

Arch Linux (rolling release — no fixed version number; lsb_release -d typically reports "Arch Linux" with no version field)

Kernel Release

Linux 7 .0.13-arch1-1 #1 SMP PREEMPT_DYNAMIC Tue, 23 Jun 2026 11:14:21 +0000 x86_64 GNU/Linux (gcc 16.1.1 20260430, GNU ld (GNU Binutils) 2.46.0)

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
  • I am running on a stable kernel release.
Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5060 Ti (UUID: GPU-427e2b9d-b93d-df00-5e64-369175184ed3)

Describe the bug

Motherboard: MSI PRO X870E-P WIFI, BIOS 2A52 (AGESA PI-1.3.0.1b Patch A)
CPU: AMD Ryzen 9 9800X3D
Memory: DDR5-6000, EXPO profile (6000 MT/s, 36-44-44-96, 1.350V), Memory Context Restore disabled
PCIe/BAR config: Resizable BAR and Above 4G Decoding enabled

Describe the bug

The GSP firmware on an RTX 5060 Ti repeatedly fails with a heartbeat/context-switch error (Xid 109), causing the GPU to become unusable until reboot. The two fully-captured occurrences below happened during logged systemd-modules activity, preceded by an Xid 62 (GSP-reported PMU halt) and an Xid 154 (recovery action changed to PF FLR). Per recollection (not independently confirmed via logs), Ollama was running in the background under inference load at the time of these crashes, consistent with a working theory that sustained GPU compute load triggers the failure — the systemd-modules process name in the log likely just reflects whichever process happened to hold a GPU channel when the crash surfaced, rather than indicating the crash is unrelated to inference load.

NVreg_EnableGpuFirmware=0 — the usual workaround for GSP-related issues — is a no-op on the open modules and does not prevent the crash.

Log output (from journalctl -k -b -22, 2025-07-08 boot; this exact sequence reproduced identically in at least two separate boots, ~18:27 and ~18:38 local time):

NVRM: GPU at PCI:0000:01:00: GPU-427e2b9d-b93d-df00-5e64-369175184ed3
NVRM: GPU Board Serial Number: 0
NVRM: Xid (PCI:0000:01:00): 62, 3243ec91 0000c918 00000000 20725e8a 20725040 207251ae 20723604 20723d94
NVRM: GPU0 _kgspRpcGspEventPmuHalted: Received signal from GSP that PMU has halted.
NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_LOG_OOB_XID, &params, sizeof(params)) @ gpu.c:7695
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
NVRM: GPU0 _kgspProcessRpcEvent: Unexpected RPC event from GPU0: 0x4c (GSP_RM_CONTROL), sequence: 1218
NVRM: Xid (PCI:0000:01:00): 109, pid=230, name=systemd-modules, channel 0x00000001, errorString CTX SWITCH TIMEOUT, Info 0x4000
[... "RC watchdog: GPU is probably locked!" and "Xid 109 ... CTX SWITCH TIMEOUT" repeat every ~8-16 seconds for roughly 5 minutes ...]
NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259

Possibly related: #1242 reports a different Xid (175, DUP_OBJECT RPC timeout) on the same GPU (RTX 5060 Ti), also under the open kernel modules driver, also ending in GSP RPC failure requiring reset. The trigger condition and exact Xid sequence differ (that report is a headless secondary GPU failing within 10 seconds of any CUDA call; this one occurred during driver load with inference running in background), so I don't believe they share the same root cause — but both point to GSP RPC-layer instability on this GPU generation under the open driver, which may be useful context for triage.

To Reproduce

Reliable, minimal repro steps are still being narrowed down. What's confirmed so far:

  • Boot the system with the RTX 5060 Ti on nvidia-open-dkms (610.43.02 or 610.43.03)
  • The GPU can fail with the Xid 62 → 154 → 109 sequence above during normal operation
  • Separately, this same GPU has also produced Xid 109 during sustained Ollama inference load (gpt-oss:20b, qwen3:14b via Docker Compose + nvidia-container-toolkit), though a full log capture of that specific case is not yet in hand
  • Per recollection, during the captured crashes, Ollama was running in the background via Docker Compose while the system was in active foreground use — the GPU was likely under inference load at the time, though this isn't confirmed against a live nvidia-smi snapshot
  • [TODO: if/when reproduced again, capture a live nvidia-smi snapshot at time of failure to confirm GPU utilization/active workload]
Bug Incidence

Sometimes

nvidia-bug-report.log.gz

Bug Incidence

Recurring — reproduced identically across at least two separate boots so far (confirmed via journalctl history); exact trigger conditions and frequency not yet fully characterized.

nvidia-bug-report.log.gz

More Info
  • Workaround attempted: NVreg_EnableGpuFirmware=0 confirmed as a no-op on the open kernel modules — does not prevent the crash
  • Driver upgrade from 610.43.02 → 610.43.03 — issue persists on both versions
  • Suspected to be specific to Blackwell-generation hardware (RTX 50-series) with the open kernel modules driver, though I haven't personally confirmed whether other Blackwell cards (RTX 5070/5080/5090) show the same behavior
  • Happy to provide additional logs or reproduce with specific diagnostic commands running, on request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached nvidia-bug-report.log.gz and the repeated journalctl Xid 62 → 154 → 109 sequence. Inspect the reported assertion locations in gpu.c:7695, rs_client.c:844, and rs_server.c:259, then work toward a reliable reproduction that confirms the trigger and distinguishes driver behavior from the suspected inference workload. Done means the failure is reproducible and its cause or a validated resolution is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.