microsoft / microsoft/WSL

[Kernel Proposal] Prevent VMBus Control-Plane Starvation & High-Order Ring Allocation Deadlocks Under Memory Pressure

Open
#41,634 1 comment 0 reactions 0 assignees View on GitHub
feature kernel
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

### Problem Statement & Architectural Context

Under heavy memory allocation (dense container compilation, heavy memory tiering, or active swap pressure), Microsoft WSL2 instances frequently encounter severe control-plane timeouts:
1. **HCS Watchdog Timeout:** When guest available memory drops into direct reclaim, atomic page allocations (`GFP_ATOMIC`) required by synthetic VMBus packet rings and host heartbeat channels fail. The Windows Host Compute System infers a hard lock and terminates the instance with `Wsl/Service/E_UNEXPECTED (0x8000ffff)` or drops the virtual switch (`Hyper-V-VmSwitch Event 102/291`).
2. **Order-7 Allocation Deadlock:** Hyper-V synthetic channel rings (`vmbus_alloc_ring()`) require contiguous physical 512 KiB allocations (`order:7, mode:0xdc0(GFP_KERNEL|__GFP_ZERO)`). Under normal session fragmentation, Order-7 is completely exhausted even when gigabytes of physical memory remain free across orders 0–3, completely wedging session creation.

### Root Cause Forensic Evidence

#### Buddy Allocator State at Failure (/proc/buddyinfo):
```text
Node 0, zone Normal 815 420 120 40 12 8 3 0 0 0 0
```
*(Orders 7–10 are 0; allocator cannot service 512 KiB contiguous blocks).*

#### Kernel Allocation Trace:
```text
kswapd0: page allocation failure: order:0, mode:0x800(GFP_ATOMIC)
Call Trace:
dump_stack_lvl+0x48/0x70
netvsc_alloc_recv_comp+0x28/0x60 [hv_netvsc]
vmbus_onoffer+0x110/0x240 [hv_vmbus]
hv_balloon: balloon inflation requested: 131072 pages (512 MB) from host
hv_balloon: page allocation failure in alloc_balloon_pages (competing with direct reclaim)
```

---

### The Proposed Fixes (Validated on Kernel 6.18.40.1)

#### 1. Dynamic VMBus Atomic Headroom & Balloon Backpressure (`drivers/hv/hv_common.c`, `drivers/hv/hv_balloon.c`)
- Calibrates `vm.min_free_kbytes` dynamically during `late_initcall` (clamped up to 512 MiB) to ensure dedicated reservation for synthetic VMBus channels.
- Defers host balloon inflation (`hv_balloon`) when guest memory is constrained (`si_mem_available() < totalram_pages() / 32`), preventing balloon thrashing during direct reclaim.

#### 2. High-Order Virtual Memory Fallback (`drivers/hv/ring_buffer.c`, `drivers/hv/channel.c`)
- Falls back to `vzalloc()` when physical contiguous order-7 allocation fails.
- Translates virtually allocated pages into PFNs for the GPA descriptor table via `vmalloc_to_page()`, providing 100% transparent operation to the Windows Hyper-V host.
- Fully preserves Confidential VM (Azure CVM / AMD SEV-SNP) memory encryption lifecycle checks (`vfree()` on decrypted pages).

---

### Empirical Hardware Validation

Tested on physical hardware under WSL2 2.7.14.0 (Kernel `6.18.40.1-microsoft-standard-WSL2+`, NVIDIA GeForce RTX 2060):
- **99% RAM Pressure (14.7 GB allocation):** Sustained 100% hold with `PASS_ZERO_PANIC` (Stock WSL2 times out within 45s).
- **Synthetic Channel Latency Under Zero Order-7 Blocks:** Established cleanly in ≤ 0.15 ms via virtual ring buffer.
- **Teardown & Recovery:** 10+ GB clean restored RAM to host upon workload completion.

### Reference Implementation
Complete reference code and build instructions are live on:
👉 **[emersonbusson/WSL2-Linux-Kernel](https://github.com/emersonbusson/WSL2-Linux-Kernel)**

Contributor guide

Open the contributing guide

Research direction

Start by reviewing drivers/hv/hv_common.c, drivers/hv/hv_balloon.c, drivers/hv/ring_buffer.c, and drivers/hv/channel.c, then compare the reference implementation in emersonbusson/WSL2-Linux-Kernel. Done means preventing control-plane allocation failures and balloon thrashing under memory pressure, providing the high-order ring fallback, and preserving teardown and confidential-VM checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.