microsoft / microsoft/WSL

[BUG] WSL2 CUDA sequence preceded by dxgkio make_resident -12 ends in host NVIDIA PCIe SRAM ECC/TDR and display loss

Open
#41,224 16 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

emailed-logs GPU
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

### Windows Version

Windows 11 version 25H2, build 26200.8875

### WSL Version

- WSL: 2.7.10.0
- Kernel: 6.18.33.2-2
- WSLg: 1.0.73.2
- Direct3D: 1.611.1-81528511
- DXCore: 10.0.26100.1-240331-1435.ge-release

### Are you using WSL 1 or WSL 2?

WSL 2

### Kernel Version

6.18.33.2-2

### Distro Version

Ubuntu under WSL2

### Other Software

- NVIDIA GeForce RTX 5090, 32 GB
- NVIDIA Windows driver 610.62
- Python/PyTorch CUDA workload through AI Toolkit
- Host-side GPU workload scheduler serializing all model-loading jobs
- Models and datasets on mounted NTFS volumes; AI Toolkit source and venv on the WSL ext4 filesystem

### Repro Steps

This is a high-risk, currently non-deterministic failure that removes the physical display, so I have not intentionally reproduced it after recovery.

Observed sequence on 2026-08-01 KST:

1. Run a large FLUX.2 Klein Base 9B BF16 probe under WSL2 from about 14:06:57 to 15:02:34.
2. WSL kernel log records `dxgkio_make_resident: Ioctl failed: -12` at 14:10:22 and again at 15:02:02.
3. Run a serialized qfloat8 rank-64 LoRA training workload. The first subject completes successfully in 97m59s with peak host-reported GPU usage about 19.3 GB.
4. Start the matching second-subject job with the same model/training contract.
5. During model initialization, at 16:46:24 KST, Windows begins a 153.851-second NVIDIA driver/TDR burst.
6. The physical display reports No Signal, and `nvidia-smi` returns exit code 6: GPU is lost; reboot required.

The earlier `make_resident -12` messages are 1h44m before the host driver crash, so I am reporting them as possible precursor state, not claiming they were simultaneous or the direct trigger. This differs from issue #41176, where every userspace failure coincided with `make_resident -12`.

### Expected Behavior

A failed GPU residency request or resource exhaustion should fail the requesting CUDA process without corrupting the shared Windows display/GPU state. A later WSL CUDA model load should not cause the Windows NVIDIA driver to enter a TDR storm or remove the physical display.

### Actual Behavior

Windows System log contained:

- 60 x `nvlddmkm` Event ID 14
- 709 x Event ID 153
- 4 x `PCIE P2PREQ, Uncorrectable SRAM Error`
- 4 x corresponding `An uncorrectable ECC error has been detected on GPU in the PCIE P2PREQ unit`
- 4 x `PCIE REORDER, Uncorrectable SRAM Error`
- 4 x corresponding REORDER ECC messages
- 708 x `GpuRcReset TDR occurred on GPUID:100`
- 1 x UCode reset

The event burst ran from 16:46:24.836 to 16:48:58.688 KST. Host-reported VRAM use at failure was about 9.3 GB, so this was not an ordinary CUDA device-memory OOM. Windows kernel event recording effectively stopped during the failure, the physical monitor lost signal, and only a Windows reboot restored the GPU.

After reboot:

- Windows and WSL again detected the RTX 5090 at PCI 01:00.0, PCIe Gen4 x16.
- A controlled 30-second BF16 CUDA preflight sustained 100% utilization and approximately 450 W, remained finite, and produced no new `nvlddmkm` or WHEA event.
- The exact failed training configuration passed transformer/text-encoder quantization, latent caching, and the previous failure point.

This recovery makes persistent application configuration failure less likely and suggests accumulated WSL GPU-PV/WDDM/NVIDIA state may be involved. However, the direct PCIe SRAM/ECC payload also means a driver, firmware, power, PCIe, or hardware fault remains possible.

### Diagnostic Logs

A local incident bundle has been preserved with:

- Windows System/Application event exports and normalized event JSON
- raw `nvlddmkm` payload counts
- WSL kernel excerpts
- GPU scheduler job/telemetry records
- post-reboot health and CUDA preflight results
- environment/version inventory

Bundle name: `v277_gpu_lost_incident_evidence_20260801.zip`
SHA-256: `1a8bc18376df25df10f5281adc0fd33739bafec56e452928c6640443df2bc290`

I have not attached the full bundle publicly because Windows/WSL diagnostic exports may contain machine-specific paths. I can provide a sanitized subset or email the full WSL diagnostics to `wsl-gh-logs@microsoft.com` after an issue number is assigned.

Because the event was not intentionally reproduced after recovery, an incident-time `collect-wsl-logs.ps1` ETL was not running. Please advise which trace providers should be armed in a bounded circular capture for a natural recurrence; intentionally reproducing the fault risks another physical display loss.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the preserved WSL kernel excerpts, Windows event exports, and normalized event JSON, then compare the sequence with issue #41176. The issue names no source files or tests and does not define a code change; progress would require a bounded circular capture during a natural recurrence and a determined cause or mitigation.

Written by the indexing model from the issue text.

Assessment

Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.