microsoft / microsoft/WSL

⚠️ WSL2 GPU OOM leads to Hyper-V kernel panic and Windows BLUE SCREEN OF DEATH — total system crash, not a recoverable error

Open
#40,732 4 comments 0 reactions 0 assignees View on GitHub
GPU
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

## Bug Description

WSL2 GPU OOM (Out of Memory) due to repeated CUDA allocation failures leads to a **Hyper-V VM kernel panic** followed by a **Windows host bugcheck (BSOD)** — the entire system crashes and reboots, not just a recoverable CUDA error.

## Steps to Reproduce

1. Load a large model (>18GB VRAM) in WSL2 via PyTorch or llama.cpp — e.g., Qwen3.6-35B on an RTX 5090 (24GB)
2. While it is loaded, attempt to load a second model (e.g., LocateAnything-3B, ~14GB)
3. PyTorch CUDA caching allocator / CUDA driver API returns OOM
4. Retry without cleaning up VRAM fragments 3-5 times
5. ⚠️ **Instead of returning a clean CUDA error, the WSL2 VM kernel panics, and Windows host bugchecks (BSOD)**
6. Full system restart required

## Expected Behavior

CUDA OOM should return a clean error code (`cudaErrorMemoryAllocation`). The process should fail gracefully. The WSL VM should never crash the Windows host.

## Actual Behavior

Consecutive GPU OOM events corrupt the GPU-PV (GPU ParaVirtualization) translation layer state, causing:
1. Hyper-V VM kernel panic (WSL crashes internally)
2. Windows host unable to recover → bugcheck (BLUE SCREEN)
3. Full system reboot — all unsaved work lost

## Environment

- **Windows 11** 23H2
- **GPU:** NVIDIA GeForce RTX 5090 Laptop GPU (Blackwell sm_120)
- **NVIDIA Driver:** 610.47
- **WSL2 Kernel:** 6.18.26.1-microsoft-standard-WSL2
- **CUDA:** 13.3
- **VRAM:** 24 GB total (only ~8 GB practically usable in WSL2 GPU-PV)

## Notes

- Single large models work fine in WSL2 via both PyTorch and llama.cpp
- The crash ONLY happens with **consecutive OOM failures** from repeated allocation attempts
- This is a **WSL2 GPU-PV layer stability issue** — the virtualized GPU driver should never be able to crash the entire Hyper-V VM and host

## Related

- #40401 (16GB GPU-PV driver overhead on Blackwell)
- #4166 (WSL2 memory management)

## Impact

Data loss from unsaved work, productivity disruption, and erodes trust in WSL2 for GPU workloads. This makes WSL2 unusable for any development involving large GPU models where OOM is a risk.

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the consecutive CUDA allocation failures in the stated Windows 11, WSL2, NVIDIA, and CUDA environment, then compare findings with #40401 and #4166. Done means repeated GPU OOM attempts return clean allocation errors without a WSL kernel panic, Windows bugcheck, or host reboot.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.