microsoft / microsoft/WSL

[Bug] WSL2 GPU passthrough has 16 GiB CUDA driver overhead on Blackwell sm_120 — blocks PyTorch large allocations

Open
#40,401 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

GPU
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

### Windows Version

Microsoft Windows [Version 10.0.26100.1742]

### WSL Version

2.6.3.0

### Are you using WSL 1 or WSL 2?

- [x] WSL 2
- [ ] WSL 1

### Kernel Version

6.6.87.2-microsoft-standard-WSL2

### Distro Version

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

### Other Software

- vLLM 0.20.0 (also tested 0.17.1) — PyTorch-based LLM inference engine
- SGLang 0.5.10 — alternative PyTorch-based inference engine
- PyTorch 2.10.0+cu128 (also tested 2.11.0+cu130)
- NVIDIA driver 596.36 on Windows host (also tested 581.80 RTX Enterprise)
- CUDA 12.8 / 13.0 (via pip nvidia-cuda-* wheels in venv)
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB VRAM, sm_120)

### Repro Steps

1. Fresh WSL2 Ubuntu 22.04 install on Windows 11 LTSC build 26100
2. Install NVIDIA driver on Windows host (596.36 or 581.80)
3. Install Python 3.10 + PyTorch + vLLM in WSL2:
```bash
python3.10 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install vllm==0.17.1 --extra-index-url https://download.pytorch.org/whl/cu128
4. Try to load any hybrid Mamba model (e.g. Qwen3.6-27B-FP8):
5. Observe the error in journal/logs
6. Note that torch.cuda.mem_get_info() reports free=80+ GiB at fresh init, BUT actual usable contiguous space is ~24 GiB due to hidden CUDA context overhead

### Expected Behavior

WSL2 GPU passthrough should have CUDA driver context overhead **comparable to native Linux** (~1-2 GiB).

The same hardware on native Linux (any distro with appropriate NVIDIA driver) shows 1-2 GiB CUDA context overhead at idle, allowing PyTorch to allocate large contiguous tensors (3-18 GiB) for hybrid Mamba models without OOM.

### Actual Behavior

WSL2 GPU passthrough on **NVIDIA RTX PRO 6000 Blackwell (sm_120)** has **~16 GiB CUDA driver context overhead** at PyTorch init — about **10x higher** than native Linux on the same hardware.

This invisible overhead is consumed by:
- DXGI shim layer (`/usr/lib/wsl/drivers/...`)
- Hyper-V virtualization layer
- CUDA driver context for sm_120 architecture

The overhead is NOT visible to `torch.cuda.mem_get_info()` so PyTorch-based frameworks (vLLM, SGLang) plan KV cache allocations based on incorrect free memory estimates → OOM at allocation time even though `nvidia-smi` shows 50+ GiB free.

Concrete error from vLLM:
torch.OutOfMemoryError: CUDA out of memory.
Tried to allocate 3.48 GiB.
GPU 0 has a total capacity of 95.59 GiB of which 50.40 GiB is free.
Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use.
↑ display bug, actually = 16 GiB
Of the allocated memory 42.49 GiB is allocated by PyTorch

**Critical:** Tested with **two NVIDIA driver versions** (596.36 from Dec 2025 and 581.80 RTX Enterprise from Nov 2025) — **identical 16 GiB overhead**. Same drivers on **native Linux show 1-2 GiB**. So this is **NOT a NVIDIA driver bug** — it's WSL2 virtualization layer behavior.

## Impact

- All hybrid Mamba/SSM models (Qwen3.6 family, Mamba-2, Jamba, etc.) cannot be used
- All PyTorch-based inference engines affected (vLLM, SGLang, TGI, LMDeploy)
- Workstation Blackwell cards ($5000-10000 hardware) are impacted
- WSL2 is the standard Windows AI dev environment

## Cross-references

- vllm-project/vllm#41619 (main bug report with full debugging context)
- sgl-project/sglang issue (TODO link if filed)
- pytorch/pytorch issue (TODO link if filed)

### Diagnostic Logs

- Full system info dump: see attached `system-info.txt`
- collect_env.py output: see attached `collect_env_output.txt`
- vLLM full error log: see attached `vllm-error-full.log`
- Reproduction script: see attached `reproduce.sh`
- WSL config: `.wslconfig` contents:
[wsl2]
memory=24GB
processors=14
swap=16GB
swapFile=F:\wsl-swap.vhdx
nestedVirtualization=false
networkingMode=mirrored
vmIdleTimeout=-1

[collect_env_output.txt](https://github.com/user-attachments/files/27362251/collect_env_output.txt)
[reproduce.sh](https://github.com/user-attachments/files/27362253/reproduce.sh)
[system-info.txt](https://github.com/user-attachments/files/27362254/system-info.txt)
[vllm-error-full.log](https://github.com/user-attachments/files/27362252/vllm-error-full.log)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached reproduce.sh, collect_env_output.txt, system-info.txt, and vllm-error-full.log to verify the reported allocation failure and compare the WSL2 environment details. The issue does not identify a WSL source entry point or a proposed implementation path; done would require identifying the virtualization-layer cause and demonstrating corrected GPU memory behavior without regressing the reported setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, pytorch, ubuntu
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.