microsoft / microsoft/WSL

dxgkrnl VMBus permanently broken (-75) after forced termination of nvcc processes during CUDA compilation

Open
#41,062 2 comments 1 reaction 0 assignees View on GitHub
GPU
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

## WSL2 dxgkrnl VMBus failure after forced termination of CUDA compilation processes

### System Info
- Windows: Windows 11 24H2
- WSL: 2.x (Ubuntu)
- GPU: NVIDIA GeForce RTX 5090 Laptop GPU
- Driver: 610.74 (WDDM mode)
- Kernel: 6.6.87.2-microsoft-standard-WSL2

### Symptom
`nvidia-smi` returns:
```
Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system
```

Windows-side `nvidia-smi` works normally. Device `/dev/dxg` exists.

### dmesg Evidence
```
[386702.754405] misc dxg: dxgk: dxgvmb_send_create_process: create_process failed -75
[386706.316347] misc dxg: dxgk: dxgvmb_send_create_process: create_process failed -75
...
```

Error code `-75` = `EOVERFLOW` — repeated every few seconds, indicating the VMBus channel between WSL and the Windows GPU driver is broken and never recovers.

### Trigger (what caused it in our case)
We were building [vLLM](https://github.com/vllm-project/vllm) from source inside WSL2, which spawns many long-running `nvcc` (CUDA compiler) processes via `ninja`. During development, these build processes were forcibly terminated multiple times with SIGTERM (killed mid-compilation because they were taking too long).

After several rounds of `kill -SIGTERM` on active `nvcc`/`ninja` processes (which hold GPU contexts via NVML/CUDA during compilation), the WSL GPU virtualization layer permanently entered a state where `dxgvmb_send_create_process` always fails with -75.

### Root Cause Hypothesis
The WSL2 `dxgkrnl` kernel module manages GPU-PV (GPU Paravirtualization) across the VMBus between the WSL VM and the Windows host. Each CUDA process creates a GPU context via the VMBus. When a process is **forcibly terminated** (SIGTERM/SIGKILL), the VMBus context may not be properly cleaned up. After accumulated corruption, the VMBus channel enters an unrecoverable state, and `dxgvmb_send_create_process` returns `-75` (EOVERFLOW) for every subsequent GPU request.

### Workaround
The only way to recover is from Windows PowerShell (admin):
```powershell
wsl --shutdown
```
Then restart WSL. This fully terminates the WSL VM and resets the dxgkrnl state.

### Expected Behavior
A proper fix should either:
1. Gracefully handle process termination during CUDA compilation to prevent VMBus corruption
2. Or auto-recover from this state (e.g., reset the VMBus channel) instead of permanently blocking GPU access
3. Or at minimum, release GPU contexts when a process is terminated rather than leaving orphaned VMBus state

### Additional Context
This appears to be a long-standing issue affecting multiple GPU types (RTX 2060 in #13769, RTX 5090 in this report) across different WSL versions. Past reports include #9938, #9962, #12859, #12835, #13769, #14339.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the documented CUDA compilation and termination sequence, then inspect the dmesg entry point dxgvmb_send_create_process and the related dxgkrnl/VMBus cleanup path. Done means GPU access recovers after process termination without requiring wsl --shutdown, with the behavior verified across the reported scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.