microsoft / microsoft/WSL

ProcessInteropMessages spins at 100% CPU forever when an interop control socket reaches EOF

Open
#41,173 6 comments 1 reaction 1 assignee Claimed by @OneBlue View on GitHub
bug emailed-logs
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

### Windows Version

```
Microsoft Windows [Version 10.0.26200.8037]
```

### WSL Version

```
2.7.10.0
```

### Are you using WSL 1 or WSL 2?

- [x] WSL 2
- [ ] WSL 1

### Kernel Version

```
6.18.33.2-microsoft-standard-WSL2
```

### Distro Version

```
Ubuntu 24.04.1 LTS
```

### Other Software

Windows Terminal (the affected `wsl.exe` was the relay for a single long-lived
tab). No third-party software is implicated — the defect is in WSL's own
interop path and is reachable from any terminal.

### Related issues

**#40651** reports the same symptom — `wsl.exe` spinning at 100% because a
`ReadFile` loop has no `bytesRead == 0` branch on the synchronous-success path.
It is closed, but not resolved: the `microsoft-github-policy-service` bot
auto-closed it after seven days of author inactivity, with the note *"If you're
still experiencing this issue please re-file it as a new issue."* This report is
that re-file, and it differs in three ways that I think matter:

1. **Different file.** #40651 points at `src/windows/common/relay.cpp`. A
maintainer [replied](https://github.com/microsoft/WSL/issues/40651#issuecomment-4561130265)
that `ScopedMultiRelay` *"is indeed missing the read length check. But it's
only used in set version. So it's likely not the cause of the busy loop you
are seeing."* I agree — the loop in this report is
`ProcessInteropMessages` in `src/windows/common/interop.cpp`, which is a
different function with a different exit structure.
2. **The function is pinned.** The original reporter noted *"We cannot pin the
spin to one specific function."* Here the stack, the thread name, and the
handle all resolve to a single loop, detailed below.
3. **The requested dumps exist.** The same comment asked for *"several dumps
seconds apart with `procdump -ma `"*. I captured three, 20 seconds
apart, and the zero-byte read is measured in them rather than inferred.

### Repro Steps

**I do not have a deterministic repro.** The trigger is a narrow race, but the
race itself is specific and reviewable, so I've described it precisely rather
than guessing at steps.

**What produced it in practice:**

1. Open a WSL2 session in a terminal and leave it open for days
2. Use it normally — in particular, run Windows binaries from inside Linux
(`explorer.exe .`, `cmd.exe /c …`, `git.exe`, `code`, `clip.exe`, anything
routed through the `WSLInterop` binfmt handler)
3. Watch `wsl.exe` thread count and CPU over time

Each interop invocation spawns one `"Interop"` thread. The overwhelming
majority exit normally. Roughly **once per 20 hours of ordinary use**, one
enters an unterminating loop and never exits. They accumulate; each burns
~100% of one core permanently.

**The race that has to be lost.** Normal shutdown does *not* trip this:

1. The launched Windows process exits → `WaitForMultipleObjects` wakes on the
process handle (`interop.cpp:467`) → loop breaks
2. Windows sends `LX_INIT_PROCESS_EXIT_STATUS` (`interop.cpp:338`)
3. Linux receives it (`binfmt.cpp:320`), returns at `:398`, and its
`wil::unique_fd` destructors close all four sockets

Step 1 runs on the **async** branch, so the happy path works. The bug needs
that order inverted — the Linux side closing `Sockets[3]` first, in the window
*between* one Windows-side read completing and the next being issued:

- Windows already parked in the async wait → EOF arrives via
`GetOverlappedResult` → `BytesRead == 0` → **handled correctly**
- Windows issues `ReadFile` *after* the FIN has already landed → completes
**synchronously**, success, 0 bytes → **spins forever**

The losing window is a handful of instructions wide, which is why the rate is
~1 per 20 h rather than one per invocation. `SIGINT` shares the signal mask
with `SIGWINCH` (`binfmt.cpp:206`), so a Ctrl-C'd interop command looks like a
plausible way to get the Linux side closing first.

**Forcing it deterministically** should be possible by delaying the
Windows-side re-read, or closing the Linux-side control socket on a timer, so
the FIN reliably lands before the next `ReadFile`.

### Expected Behavior

When the interop control socket reaches EOF, `ProcessInteropMessages` should
break out of its read loop, the `"Interop"` thread should exit, and `wsl.exe`
should return to ~0% CPU.

### Actual Behavior

The thread never exits. It busy-reads a closed socket at ~100% of one core for
the entire remaining lifetime of the `wsl.exe` process. Leaked threads
accumulate with no upper bound.

On my machine a single `wsl.exe` reached **68.5% of a 12-core system (821% of
one core) across 10 leaked threads**, having consumed **849 CPU-hours in 9
days**. Left running it would have saturated all 12 cores. Killing the process
was the only remedy.

#### The defect

`ProcessInteropMessages` (`src/windows/common/interop.cpp:422`) has an
infinite read loop whose body handles *only* the cases where `ReadFile`
returns FALSE. When `ReadFile` completes **synchronously and returns TRUE**,
the loop takes no action and immediately re-issues the read:

```c
// src/windows/common/interop.cpp:422
DWORD ExitCode = 1;
for (;;)
{
DWORD BytesRead;
LX_INIT_WINDOW_SIZE_CHANGED WindowSizeMessage;
bool Success = ReadFile(MessageHandle, &WindowSizeMessage, sizeof(WindowSizeMessage), &BytesRead, &Overlapped);
if (!Success)
{
const auto LastError = GetLastError();
if ((LastError == ERROR_BROKEN_PIPE) || (LastError == ERROR_HANDLE_EOF))
{
/* ... break ... */
}

THROW_LAST_ERROR_IF(LastError != ERROR_IO_PENDING);

/* async path: waits, then correctly checks (!Success) || (BytesRead == 0) -> break */
}
// <- line 480: no `else`. Synchronous success falls through and re-reads.
}
```

The asymmetry is the bug. The **asynchronous** completion path correctly
treats `BytesRead == 0` as EOF and breaks:

```c
Success = GetOverlappedResult(MessageHandle, &Overlapped, &BytesRead, FALSE);
CancelIo.release();
if ((!Success) || (BytesRead == 0)) // <-- EOF handled here
{
...
break;
}
```

The **synchronous** completion path performs no equivalent check.

**All three loop exits sit inside `if (!Success)`**, so once the socket starts
reporting EOF-as-success the loop is not merely leaky but *provably*
non-terminating:

| Exit | Line | Requires |
|---|---|---|
| `ERROR_BROKEN_PIPE` / `ERROR_HANDLE_EOF` | 430 | `Success == FALSE` — a socket produces neither |
| `BytesRead == 0` | 452 | `Success == FALSE` first (async branch) |
| child process exited | 467 | `Success == FALSE` first (async branch) |

The third is worth calling out: the `WaitForMultipleObjects` on
`Result->Process` is also unreachable, so **not even the launched Windows
process exiting can break the loop.**

The comment above the loop (line 416) suggests how this happened:

> *"Read messages from the message handle. Break out of the loop if the pipe
> is connection is closed or the process exits."*

That is accurate for the WSL1 path, which does pass a real named pipe. The
loop appears to have been reused for the socket path without revisiting EOF
semantics.

#### Why WSL2 hits this and WSL1 does not

The two call sites pass different kinds of handle:

- **WSL1 / LxBus path** — `interop.cpp:559` passes `SignalPipe.first.get()`,
a real named pipe (`OpenAnonymousPipe` → `NtCreateNamedPipeFile`). A closed
pipe yields `ERROR_BROKEN_PIPE`, which *is* handled. No spin.
- **WSL2 / VM path** — `interop.cpp:335` passes a socket cast to a handle:

```c
ExitStatus.ExitCode = ProcessInteropMessages(reinterpret_cast(Sockets[3].get()), &Result);
```

A graceful socket close is EOF-as-success, so neither handled status is
ever produced, and the loop spins.

**Which socket:** a Hyper-V socket (`AF_HYPERV` on Windows, `AF_VSOCK` on
Linux), created overlapped at `hvsocket.cpp:102`. Linux opens it first —
`binfmt.cpp:153` allocates a fresh listening vsock on an ephemeral port per
interop invocation and passes the port number up in the create-process message
(`Message->Port = SocketAddress.svm_port`, `:161`). Windows connects four
times (`interop.cpp:283`), Linux accepts four (`binfmt.cpp:184`) — stdin,
stdout, stderr, and the control channel. It is the control channel,
`Sockets[3]`, that `ProcessInteropMessages` reads, and its peer is the `/init`
binfmt interpreter inside the utility VM.

This is distinct from the long-lived interop server channel
(`UtilConnectToInteropServer()`, `binfmt.cpp:167`), which only carries the
launch request.

#### Secondary defect (same missing branch)

A window-resize message that completes synchronously is silently discarded —
`ResizePseudoConsole` is only reached on the asynchronous path. So a resize
delivered inline is dropped rather than applied.

#### Suggested fix

Handle synchronous completion symmetrically with asynchronous completion:

```c
bool Success = ReadFile(MessageHandle, &WindowSizeMessage, sizeof(WindowSizeMessage), &BytesRead, &Overlapped);
if (!Success)
{
/* ... unchanged ... */
}
else if (BytesRead == 0)
{
// Peer closed gracefully. On a socket this surfaces as success with zero
// bytes rather than ERROR_BROKEN_PIPE.
if (WI_IsFlagClear(Result->Flags, LX_INIT_CREATE_PROCESS_RESULT_FLAG_GUI_APPLICATION))
{
THROW_IF_WIN32_BOOL_FALSE(TerminateProcess(Result->Process.get(), 1));
}

break;
}
else
{
// Synchronous completion with data: currently dropped.
WI_ASSERT((BytesRead == sizeof(WindowSizeMessage)) && (WindowSizeMessage.Header.MessageType == LxInitMessageWindowSizeChanged));
const COORD Size{static_cast(WindowSizeMessage.Columns), static_cast(WindowSizeMessage.Rows)};
THROW_IF_FAILED(ResizePseudoConsole(Result->PseudoConsole.get(), Size));
}
```

The EOF-handling and resize-handling blocks are now duplicated between the
synchronous and asynchronous paths; factoring each into a small lambda would
be cleaner. Happy to send this as a PR if the approach looks right — I have
not built WSL locally, so it is untested beyond review.

A defensive backstop worth considering separately: these interop threads have
no upper bound and no liveness check, so any future non-terminating condition
in this loop degrades the whole machine silently. A leaked thread is permanent
for the life of the process.

#### Note on current sources

`ProcessInteropMessages` is byte-identical between tag `2.7.10` and `master`
at the time of writing, so this is not fixed in current sources. Release
2.7.11 contains only a MoveDistribution VHD fix and security backports.

### Diagnostic Logs

I did not collect `collect-wsl-logs.ps1` output — the failure is a CPU spin
with no error path, so nothing is logged. Instead I captured three
full-memory dumps of the affected `wsl.exe`, 20 seconds apart, and analysed
them in WinDbg.

**Identical stack signature** (matching hash across all three dumps) on all
ten runaway threads:

```
ntdll!NtReadFile+0x14
KERNELBASE!ReadFile+0x12a
wsl+0x113c51
wsl+0x111fcf
wsl+0x110bc2
wsl+0x1b35df
kernel32!BaseThreadInitThunk+0x17
ntdll!RtlUserThreadStart+0x2c
```

All ten threads are named **`"Interop"`** — set at `interop.cpp:271`, inside
the very `std::thread` that calls `ProcessInteropMessages`. The four `wsl`
frames map one-to-one onto the source:

| Frame | Source |
|---|---|
| `wsl+0x1b35df` | `std::thread` trampoline (interop.cpp:268) |
| `wsl+0x110bc2` | lambda body (interop.cpp:271–340) |
| `wsl+0x111fcf` | `ProcessInteropMessages` (interop.cpp:409) |
| `wsl+0x113c51` | the `ReadFile` call (interop.cpp:426) |

Each thread holds a distinct handle, consistent with one leaked thread per
interop session. Windows reports every one as **`Running` with no wait
reason** — they never block, which is the signature of a busy loop rather
than a stalled wait.

**The zero-byte read, observed.** `wsl.exe` ships without public symbols, so
`BytesRead` is not recoverable by name — but `ReadFile` deposits the byte
count in the `OVERLAPPED` on the thread stack, which is reachable:

- `ntdll!NtReadFile` begins `mov r10,rcx` and **`r10` survives the syscall**,
giving the `HANDLE` directly (`0x428`, `0x748`, `0x710`)
- `rdx` holds the `Event` parameter (`0x66c`); `r9` holds `ApcContext`, which
`KERNELBASE!ReadFile` sets to the `OVERLAPPED` pointer (`0xb5c8fff858`)
- reading 32 bytes at that address returns an `OVERLAPPED` whose `hEvent` is
`0x66c`, matching `rdx` — confirming the right structure

`Internal` is the `NTSTATUS`, `InternalHigh` the byte count. Across the three
captures of the same thread:

| Dump | `Internal` | `InternalHigh` |
|---|---|---|
| 1 | `0x103` `STATUS_PENDING` | 0 |
| 2 | `0x103` `STATUS_PENDING` | 0 |
| **3** | **`0x0` `STATUS_SUCCESS`** | **0** |

Two samples caught the IRP in flight; the third caught it completed —
**success with zero bytes.** That is the EOF-as-success case the synchronous
path has no branch for, measured rather than inferred.

One further oddity, offered as an observation rather than a claim:
`PointerCount` on these three handles ranges **33,764–65,423** and fluctuates
between captures, where ordinary file objects sit in the low tens. I could not
determine the mechanism from a user-mode dump, but it is a consistent
fingerprint of extreme I/O churn on those handles.

**Per-thread CPU**, showing each thread burning ~85–92% of a core
continuously from the moment it was created, and never exiting:

| Thread created | CPU hours | % of one core since creation |
|---|---|---|
| day 0 | 189.6 | ~92% |
| day 3 | 125.8 | ~90% |
| day 3 | 124.0 | ~89% |
| day 4 | 103.3 | ~88% |
| day 5 | 85.8 | ~89% |
| day 6 | 65.9 | ~86% |
| day 6 | 60.5 | ~83% |
| day 6 | 59.1 | ~83% |
| day 8 | 18.7 | ~87% |
| day 8 | 17.0 | ~79% |

These sum to the process's entire CPU total, so the loop accounts for 100% of
the consumption.

I still have the three full-memory dumps. I have not attached them because
they are ~72 MB each and, being full-memory captures of a process that relayed
terminal I/O for nine days, may contain environment variables, command lines,
and buffered stdio. **Happy to send them to wsl-gh-logs@microsoft.com** if
that would help — with private PDBs they should resolve `wsl+0x113c51` to an
exact line and let you read `BytesRead` directly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.