[BUG] WSL2 CUDA sequence preceded by dxgkio make_resident -12 ends in host NVIDIA PCIe SRAM ECC/TDR and display loss
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 33.7k
- Forks
- 1.8k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 116
Description
### Windows Version
Windows 11 version 25H2, build 26200.8875
### WSL Version
- WSL: 2.7.10.0
- Kernel: 6.18.33.2-2
- WSLg: 1.0.73.2
- Direct3D: 1.611.1-81528511
- DXCore: 10.0.26100.1-240331-1435.ge-release
### Are you using WSL 1 or WSL 2?
WSL 2
### Kernel Version
6.18.33.2-2
### Distro Version
Ubuntu under WSL2
### Other Software
- NVIDIA GeForce RTX 5090, 32 GB
- NVIDIA Windows driver 610.62
- Python/PyTorch CUDA workload through AI Toolkit
- Host-side GPU workload scheduler serializing all model-loading jobs
- Models and datasets on mounted NTFS volumes; AI Toolkit source and venv on the WSL ext4 filesystem
### Repro Steps
This is a high-risk, currently non-deterministic failure that removes the physical display, so I have not intentionally reproduced it after recovery.
Observed sequence on 2026-08-01 KST:
1. Run a large FLUX.2 Klein Base 9B BF16 probe under WSL2 from about 14:06:57 to 15:02:34.
2. WSL kernel log records `dxgkio_make_resident: Ioctl failed: -12` at 14:10:22 and again at 15:02:02.
3. Run a serialized qfloat8 rank-64 LoRA training workload. The first subject completes successfully in 97m59s with peak host-reported GPU usage about 19.3 GB.
4. Start the matching second-subject job with the same model/training contract.
5. During model initialization, at 16:46:24 KST, Windows begins a 153.851-second NVIDIA driver/TDR burst.
6. The physical display reports No Signal, and `nvidia-smi` returns exit code 6: GPU is lost; reboot required.
The earlier `make_resident -12` messages are 1h44m before the host driver crash, so I am reporting them as possible precursor state, not claiming they were simultaneous or the direct trigger. This differs from issue #41176, where every userspace failure coincided with `make_resident -12`.
### Expected Behavior
A failed GPU residency request or resource exhaustion should fail the requesting CUDA process without corrupting the shared Windows display/GPU state. A later WSL CUDA model load should not cause the Windows NVIDIA driver to enter a TDR storm or remove the physical display.
### Actual Behavior
Windows System log contained:
- 60 x `nvlddmkm` Event ID 14
- 709 x Event ID 153
- 4 x `PCIE P2PREQ, Uncorrectable SRAM Error`
- 4 x corresponding `An uncorrectable ECC error has been detected on GPU in the PCIE P2PREQ unit`
- 4 x `PCIE REORDER, Uncorrectable SRAM Error`
- 4 x corresponding REORDER ECC messages
- 708 x `GpuRcReset TDR occurred on GPUID:100`
- 1 x UCode reset
The event burst ran from 16:46:24.836 to 16:48:58.688 KST. Host-reported VRAM use at failure was about 9.3 GB, so this was not an ordinary CUDA device-memory OOM. Windows kernel event recording effectively stopped during the failure, the physical monitor lost signal, and only a Windows reboot restored the GPU.
After reboot:
- Windows and WSL again detected the RTX 5090 at PCI 01:00.0, PCIe Gen4 x16.
- A controlled 30-second BF16 CUDA preflight sustained 100% utilization and approximately 450 W, remained finite, and produced no new `nvlddmkm` or WHEA event.
- The exact failed training configuration passed transformer/text-encoder quantization, latent caching, and the previous failure point.
This recovery makes persistent application configuration failure less likely and suggests accumulated WSL GPU-PV/WDDM/NVIDIA state may be involved. However, the direct PCIe SRAM/ECC payload also means a driver, firmware, power, PCIe, or hardware fault remains possible.
### Diagnostic Logs
A local incident bundle has been preserved with:
- Windows System/Application event exports and normalized event JSON
- raw `nvlddmkm` payload counts
- WSL kernel excerpts
- GPU scheduler job/telemetry records
- post-reboot health and CUDA preflight results
- environment/version inventory
Bundle name: `v277_gpu_lost_incident_evidence_20260801.zip`
SHA-256: `1a8bc18376df25df10f5281adc0fd33739bafec56e452928c6640443df2bc290`
I have not attached the full bundle publicly because Windows/WSL diagnostic exports may contain machine-specific paths. I can provide a sanitized subset or email the full WSL diagnostics to `wsl-gh-logs@microsoft.com` after an issue number is assigned.
Because the event was not intentionally reproduced after recovery, an incident-time `collect-wsl-logs.ps1` ETL was not running. Please advise which trace providers should be armed in a bounded circular capture for a natural recurrence; intentionally reproducing the fault risks another physical display loss.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the preserved WSL kernel excerpts, Windows event exports, and normalized event JSON, then compare the sequence with issue #41176. The issue names no source files or tests and does not define a code change; progress would require a bounded circular capture during a natural recurrence and a determined cause or mitigation.
Written by the indexing model from the issue text.
Assessment
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100