dxgkrnl VMBus permanently broken (-75) after forced termination of nvcc processes during CUDA compilation
- Dominant language
- C++
- Stars
- 33.7k
- Forks
- 1.8k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 116
Description
## WSL2 dxgkrnl VMBus failure after forced termination of CUDA compilation processes
### System Info
- Windows: Windows 11 24H2
- WSL: 2.x (Ubuntu)
- GPU: NVIDIA GeForce RTX 5090 Laptop GPU
- Driver: 610.74 (WDDM mode)
- Kernel: 6.6.87.2-microsoft-standard-WSL2
### Symptom
`nvidia-smi` returns:
```
Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system
```
Windows-side `nvidia-smi` works normally. Device `/dev/dxg` exists.
### dmesg Evidence
```
[386702.754405] misc dxg: dxgk: dxgvmb_send_create_process: create_process failed -75
[386706.316347] misc dxg: dxgk: dxgvmb_send_create_process: create_process failed -75
...
```
Error code `-75` = `EOVERFLOW` — repeated every few seconds, indicating the VMBus channel between WSL and the Windows GPU driver is broken and never recovers.
### Trigger (what caused it in our case)
We were building [vLLM](https://github.com/vllm-project/vllm) from source inside WSL2, which spawns many long-running `nvcc` (CUDA compiler) processes via `ninja`. During development, these build processes were forcibly terminated multiple times with SIGTERM (killed mid-compilation because they were taking too long).
After several rounds of `kill -SIGTERM` on active `nvcc`/`ninja` processes (which hold GPU contexts via NVML/CUDA during compilation), the WSL GPU virtualization layer permanently entered a state where `dxgvmb_send_create_process` always fails with -75.
### Root Cause Hypothesis
The WSL2 `dxgkrnl` kernel module manages GPU-PV (GPU Paravirtualization) across the VMBus between the WSL VM and the Windows host. Each CUDA process creates a GPU context via the VMBus. When a process is **forcibly terminated** (SIGTERM/SIGKILL), the VMBus context may not be properly cleaned up. After accumulated corruption, the VMBus channel enters an unrecoverable state, and `dxgvmb_send_create_process` returns `-75` (EOVERFLOW) for every subsequent GPU request.
### Workaround
The only way to recover is from Windows PowerShell (admin):
```powershell
wsl --shutdown
```
Then restart WSL. This fully terminates the WSL VM and resets the dxgkrnl state.
### Expected Behavior
A proper fix should either:
1. Gracefully handle process termination during CUDA compilation to prevent VMBus corruption
2. Or auto-recover from this state (e.g., reset the VMBus channel) instead of permanently blocking GPU access
3. Or at minimum, release GPU contexts when a process is terminated rather than leaving orphaned VMBus state
### Additional Context
This appears to be a long-standing issue affecting multiple GPU types (RTX 2060 in #13769, RTX 5090 in this report) across different WSL versions. Past reports include #9938, #9962, #12859, #12835, #13769, #14339.
Contributor guide
Research direction
Start by reproducing the failure with the documented CUDA compilation and termination sequence, then inspect the dmesg entry point dxgvmb_send_create_process and the related dxgkrnl/VMBus cleanup path. Done means GPU access recovers after process termination without requiring wsl --shutdown, with the behavior verified across the reported scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100