vmmem growth starves Windows-side GPU/NPU allocations (Level Zero `ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY`, D3D12 OOM) — page cache is never returned, and neither side reports the cause
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 33.7k
- Forks
- 1.8k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 116
Description
### Summary
When vmmem grows toward the `.wslconfig` `memory=` cap, Windows-side device allocations
start failing: Level Zero cannot create an NPU command queue
(`ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY`), and D3D12 shared-memory allocations for the iGPU
fail far below their nominal budgets. The cause is that WSL2 returns memory to Windows only
through free-page reporting, which covers memory the Linux kernel has freed — the page
cache and all anonymous memory count as in-use and stay in vmmem indefinitely. The failure
is invisible from both sides: Linux shows tens of GB "available" (reclaimable cache),
Windows shows almost nothing free, and the driver errors name neither. No Windows setting
fixes this; only releasing memory inside the guest does.
### Environment
- Windows 11, 128 GB RAM, WSL2 Ubuntu 26.04, `networkingMode=mirrored`
- `.wslconfig`: `memory=124GB`, `autoMemoryReclaim=disabled`
- Host-side consumers: Intel NPU via OpenVINO GenAI (Level Zero), Intel iGPU via D3D12
(Core Ultra 7 270K)
### Reproduction
1. Set `memory=` close to the machine total (the default 50%–8GB rule avoids this, which
is why the failure mostly hits tuned setups).
2. Run a memory-heavy Linux workload — in my case an LLM inference server with large
mmap-loaded model files, which fills the page cache by design, plus process RSS in the
tens of GB.
3. On the Windows side, start anything that needs device memory: an OpenVINO GenAI
pipeline on the NPU, or a D3D12 app using iGPU shared memory.
Observed once vmmem neared the cap:
```
# OpenVINO GenAI, NPU pipeline creation:
Exception from src\plugins\intel_npu\...\zero_wrappers.cpp:323:
L0 zeCommandQueueCreate result: ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY, code 0x70000002
# D3D12 (iGPU shared memory), allocations of 745 MB and 997 MB on a device
# reporting 67 GB visible: ErrorOutOfDeviceMemory
```
### Diagnosis
The two sides disagree about the same physical memory:
```
# WSL — looks healthy, 74 GB "available":
$ free -g
total used free buff/cache available
Mem: 121 46 6 78 74
# Windows — nearly exhausted:
PS> Get-CimInstance Win32_OperatingSystem | select FreePhysicalMemory
11.9 GB free of 127.3
```
Linux's 74 GB "available" is page cache it would drop under *its own* memory pressure —
but the host's pressure generates no signal inside the guest, so it never drops, and page
reporting has nothing to return.
Confirming the mechanism: releasing ~19 GB of anonymous memory inside the guest (stopping
the inference server) returned it to Windows within seconds, and the same
`zeCommandQueueCreate` then succeeded. No reboot, no `wsl --shutdown`.
Two things do *not* help, worth stating because they are the documented knobs:
- `echo 1 > /proc/sys/vm/drop_caches` (the workaround in the memory-reclaim blog post)
cannot evict pages actively mmapped by a running process — a resident model file is
pinned regardless.
- `autoMemoryReclaim` trades this problem for another on this workload class: dropping the
cache means re-reading tens of GB of model files on the next inference, so mmap-heavy
setups run with it disabled by design.
### Suggestions
1. **Docs** (cheap, high value): a paragraph in the `.wslconfig` reference stating that
`memory=` should be sized to leave the host a working floor when the host itself runs
GPU/NPU/driver workloads, because page cache and process memory are not returned while
in use — and that host-side symptoms are device-allocation failures that never mention
WSL. Today nothing connects those dots for the person debugging the driver error.
2. **Diagnosability**: a `wsl` command (or perf counter) exposing "guest memory returnable
vs pinned" would turn a multi-hour investigation into one look. The driver errors can't
name the cause; WSL can.
3. **Longer term**: a host-floor semantic — some notion of host reserve that vmmem growth
respects under host memory pressure — would make mixed WSL+host-compute machines safe
without hand-tuning `memory=` against the worst day.
### Related
#10011 (memory reclaim discussion) covers returning freed memory; this issue is about
memory that is *never* freed guest-side and the host-side failure mode that results. #12634
(Intel iGPU in WSL2) is adjacent hardware but a different problem. I did not find an
existing issue describing the host-side GPU/NPU allocation failures.
Happy to provide full logs and the before/after memory counters from both sides.
---
*Disclosure: investigation assisted by AI (Claude, Anthropic). I directed and verified;
all numbers were measured on my hardware as described.*
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the .wslconfig memory= and autoMemoryReclaim behavior described here, then review the provided Linux free and Windows FreePhysicalMemory observations alongside related issue #10011. No source file or test is named; the scope must first be narrowed between documentation, diagnostics, and host-floor behavior before completion can be verified.
Written by the indexing model from the issue text.
Assessment
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100