Init interop channel never re-established after it closes: `accept4 failed 110` on `/run/WSL/1_interop` for the VM's lifetime
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 33.7k
- Forks
- 1.8k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 116
Description
### Windows Version
Microsoft Windows [Version 10.0.26200.8875]
### WSL Version
WSL version: 2.7.8.0
Kernel version: 6.18.33.1-1
WSLg version: 1.0.73.2
MSRDC version: 1.2.6676
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26200.8875
### Are you using WSL 1 or WSL 2?
- [x] WSL 2
- [ ] WSL 1
### Kernel Version
6.18.33.1-microsoft-standard-WSL2
### Distro Version
Ubuntu 24.04.4 LTS
### Other Software
No `%USERPROFILE%\.wslconfig` (all defaults).
```
# /etc/wsl.conf
[boot]
systemd=true
```
Also running in the same utility VM: Docker Desktop WSL integration, VS Code
remote server, WSLg. None of them is required to observe the failure.
### Repro Steps
I have no deterministic reproduction yet, and I explicitly could not reproduce
it on demand (see "What I could not reproduce" below). What follows is a fully
characterised occurrence plus the detection method, in the hope the guest-side
half is actionable on its own — the recovery defect is independent of whatever
triggers the initial channel loss.
**Symptom.** About 4h52m into the VM's life, every attempt to launch a Windows
binary that goes through the init interop fallback started failing, and kept
failing for the remaining ~19 hours of VM uptime:
```
WSL (1283985 - ) ERROR: UtilAcceptVsock:273: accept4 failed 110
```
Each attempt takes ~10s and returns errno 110 (ETIMEDOUT). Sessions with a
valid, live `$WSL_INTEROP` are unaffected, which is why this stays invisible
until some process's `$WSL_INTEROP` goes stale.
**How a process ends up on the fallback.** `$WSL_INTEROP` points at
`/run/WSL/_interop`. A long-lived process (in my case a `tmux`
server started at boot, but any daemon reparented away from its session
qualifies) keeps the value it inherited. When the session that spawned it
exits, that socket is removed and the inherited value dangles. `/init` then
falls back to its hardcoded `/run/WSL/1_interop`, which on a systemd
distribution is a symlink to `/run/WSL/2_interop`.
**What happened around the moment of failure.** The distribution was started at
utility VM bring-up, and that first session hosted a `tmux` server. About 4h52m
later the Windows Terminal window hosting that session closed abruptly, on its
own, while work was in progress in it — plausibly including interop calls. I
reattached to the surviving `tmux` server from a new terminal and a new
`wsl.exe`, and from that point on interop was broken. The first
`SendMessage`/`Broken pipe` pair in `dmesg` is at that time, so the channel died
then rather than silently earlier.
I want to be explicit that I have not established the direction of causality
here. A healthy `wsl.exe` does not terminate spontaneously, so the terminal's
death may well be a second symptom of the same host-side fault rather than the
cause of the channel loss. See "What I could not reproduce" — deliberately
killing that process tree does not reproduce the failure.
**Guest-side chain.** `/run/WSL/2_interop` is owned by the in-distribution WSL
init (pid 2). It accepts the connection, then forwards
`LxInitMessageCreateProcessUtilityVm` over its Interop channel — and that
channel is dead:
```
WSL (2 - Interop) ERROR: SendMessage:182: Failed to write message LxInitMessageCreateProcessUtilityVm. Channel: Interop
WSL (2) ERROR: Broken pipe @C:/__w/1/s/src/shared/inc\SocketChannel.h:183 (SendMessage)
```
The message never reaches the host, so the host never connects back, so the
caller-side helper times out in `accept4`. Every failing launch produces exactly
one such `SendMessage`/`Broken pipe` pair in `dmesg`, at the same timestamp — 30
pairs over the period I was measuring.
**The dead channel.** fd table of the in-distribution init (pid 2 inside the
distribution, 203 in the utility VM's root PID namespace):
```
100 -> socket:[12301] AF_VSOCK ESTAB *:481174001 -> 2:50000 control channel, healthy
6 -> socket:[9237]
8 -> socket:[9243] AF_VSOCK CLOSING *:481173381 -> 2:3406191018 Interop channel, dead
9 -> socket:[9244] AF_UNIX LISTEN /run/WSL/2_interop
```
So this is not a general loss of host communication: the control channel on fd
100 is still established. Exactly one channel died, and it is the Interop one.
fd 8 stayed in `CLOSING` for the whole remaining VM lifetime — 19+ hours — and
was inherited by every session created afterwards. `ss -a -e -p --vsock`, from
`wsl --debug-shell` (pids are root-namespace pids):
```
v_str CLOSING 0 0 *:481173381 2:3406191018 users:(("Relay(848062",pid=14964,fd=8),("SessionLeader",pid=14963,fd=8),("Relay(138607",pid=9782,fd=8),("SessionLeader",pid=9781,fd=8),("Relay(138100",pid=9273,fd=8),("SessionLeader",pid=9269,fd=8),("Relay(138041",pid=9211,fd=8),("SessionLeader",pid=9210,fd=8),("Relay(137547",pid=8719,fd=8),("SessionLeader",pid=8718,fd=8),("init-systemd(My",pid=203,fd=8))
```
Every `SessionLeader` and `Relay` created after the channel died holds fd 8 on
the same dead socket.
**A healthy sibling channel, created at the same moment.** The same
distribution's outer `/init` — the one that runs in the VM rootfs before
pivoting into the distribution, and that holds a `/run/WSL/1_interop` listener
in its own mount namespace — still has a working interop channel after 19 hours:
```
v_str ESTAB 0 0 *:481173378 2:3406191014 users:(("init(",pid=198,fd=7))
v_str CLOSING 0 0 *:481173381 2:3406191018 users:(…,("init-systemd(My",pid=203,fd=8))
```
Host ports `3406191014` and `3406191018` are 4 apart, and the socket inodes
(12313 vs 9243) are both in the earliest allocation range, so both channels were
established at VM bring-up. One survived, the other did not. That rules out
host-wide causes and narrows this to a per-channel loss.
**No recovery path.** `/init` never closes the dead fd and never rebuilds the
server. The binary contains the guard strings:
```
Could not create init interop server
Interop server already created
/run/WSL/1_interop
```
Meanwhile `/run/WSL/2_interop` keeps accepting connections happily, so callers
have no way to detect that the socket behind the hardcoded fallback path is
useless. They just block for 10s and get a misleading `accept4` error.
### What I could not reproduce
On a freshly created test distribution (Debian, also `systemd=true`), started
while the utility VM was already running, the in-distribution init's interop
channel stayed `ESTAB` through every teardown path I could construct. In each
case the fallback still worked afterwards. The session whose launch started the
distribution was the one destroyed, since that matches the affected channel's
ownership:
1. That session exiting cleanly, with a second session still open.
2. The same, but leaving a daemonized child behind (`setsid sleep infinity &`),
so its `Relay` was orphaned and reparented to init while still holding fd 8 on
the channel:
```
before logout: v_str ESTAB *:481174008 2:3031882125 users:(("Relay(194",pid=28982,fd=8),("SessionLeader",pid=28981,fd=8),("init-systemd(sc",pid=28438,fd=8))
after logout: v_str ESTAB *:481174008 2:3031882125 users:(("Relay(194",pid=28982,fd=8),("init-systemd(sc",pid=28438,fd=8))
```
3. Abrupt destruction of the whole host-side process tree —
`taskkill /F /T` on the terminal host, taking down the outer `wsl.exe`, the
inner `wsl.exe` and the session's `wslhost.exe` at once with no graceful
shutdown — with nothing in flight.
4. The same abrupt destruction while interop requests were in flight (a loop of
short `cmd.exe` invocations), both with and without a second session open.
5. The same, but with a single long-running `cmd.exe` rather than a loop of short
ones, to rule out landing between two iterations. Again with and without a
second session.
I also checked that failing fallback launches do not themselves leak anything:
two failed attempts left the orphaned-relay count, the total `AF_VSOCK`
connection count and the `CLOSING` count all unchanged.
So no session-teardown path visible from the guest reproduces this, including
the abrupt one that matches what I actually observed. Three factors remain that
a test distribution cannot vary:
- **The channel was established at utility VM bring-up.** Its socket inodes
(9243/9244) and host ports (`3406191014`–`3406191039`) are all in the earliest
allocation cluster, whereas a distribution started later gets a channel in a
completely different range (`3031882125`, inode 16558018). If the host endpoint
for a bring-up channel is wired differently from one created on a later
distribution launch, no test on a later distribution can reach that code path.
Varying this requires `wsl --shutdown`.
- **Elapsed time.** The affected channel lived 4h52m before dying; the
reproduction attempts last minutes.
- **A host-side fault upstream of both symptoms**, which by definition cannot be
provoked from the guest.
I am now capturing a persistent circular `netsh trace scenario=netconnection`
ring on the host plus a guest-side watcher that timestamps any new non-`ESTAB`
`AF_VSOCK` socket, so the next occurrence is documented from both sides. Happy to
attach both when it recurs.
### Host-side evidence, searched retroactively
The utility VM has not been restarted since, so I went back through everything
Windows persists. None of it contains the trigger, which seems worth stating
explicitly so it does not get re-checked:
| Checked | Result |
|---|---|
| WER / Application Error / Hang events, 16:05–16:25 | none at all — the terminal window's disappearance produced no crash record |
| WER report directories created in the window | none |
| `wslservice.exe` | pid unchanged, created at 11:12:46, i.e. before VM bring-up; never restarted |
| Service Control Manager events for WSL / Lxss / Hyper-V / vmcompute | none that day |
| `Win32_ReliabilityRecords`, 15:30–17:00 | nothing before 16:26:51, a Windows Update after the fact |
| Kernel-Power (sleep / resume) | none |
| `Microsoft-Windows-Hyper-V-VmSwitch-Operational` | only the recurring slow-OID notices on the NAT vNIC, same cadence before and after |
| NDIS, NetworkProfile, DHCP, DNS, SMB, VPN, WLAN, Kernel-PnP, Winlogon | no events in the window |
| Guest `systemd` journal, 16:14:30–16:15:30 | only the two error pairs and a routine cron job — no session teardown, no `logind` activity, no process death |
A `NETLOGON` 5719 lands 3 seconds before the first failure, but that event fires
4–7 times a day on this domain-joined machine (547 occurrences over five months,
seven of them that day), so it is background noise rather than a lead.
Note that the failure timestamps only bound the channel's death loosely. Baseline
interop traffic in this distribution is about one launch every 5 minutes, so the
last known-good moment is somewhere in 16:10–16:14:52; the dense retries
immediately after 16:14:52 are me reacting to the breakage, not a poller.
The reason nothing else is available is structural: WSL's own diagnostics are
ETW-only and not persisted, and this machine has no populated
`Microsoft-Windows-Subsystem-Linux` channel. The channel teardown is invisible
after the fact by design — hence the forward-looking capture above.
### Impact
The failure has been continuous since it started. Over the following two days the
guest journal records **587 failed forwarding attempts** — roughly 99 in the first
two hours, then a steady one every ~5 minutes from background activity in the
distribution — each one a 10 s stall for whichever process triggered it.
### How to detect it
From inside the distribution:
```bash
# is this process on a dangling socket?
[ -S "$WSL_INTEROP" ] || echo "stale WSL_INTEROP: $WSL_INTEROP"
# is the fallback dead?
WSL_INTEROP=/run/WSL/1_interop cmd.exe /c ver # ~10s, then accept4 failed 110
```
From `wsl --debug-shell` (needed for socket-to-process attribution: a shell
inside the distribution is in a child PID namespace but the root *network*
namespace, so `ss` lists every vsock in the VM while resolving owners only for
the distribution's own processes):
```sh
ss -a -e -p --vsock | grep CLOSING
```
### Expected Behavior
An init interop channel that closes should either be re-established on the next
request, or the socket that depends on it should stop accepting connections so
callers fail immediately and visibly. Either way the dead fd should be closed
rather than left in `CLOSING` for the lifetime of the VM and inherited by every
subsequent session.
Secondarily, the surfaced error is misleading: it names `accept4` in the caller,
whereas the actual failure is an `EPIPE` on the forwarding write one layer away.
This was already noted in #40650; this report is a concrete case where the real
cause is not where the message points.
### Actual Behavior
The channel is never rebuilt and the fd is never closed. Every Windows binary
launch that reaches the init interop fallback fails after a 10s timeout, for the
rest of the utility VM's lifetime, and `wsl --shutdown` is the only documented
way out.
### Workaround
Interop can be restored without `wsl --shutdown` by pointing `$WSL_INTEROP` at
the socket of a session that is actually alive:
```bash
export WSL_INTEROP=/run/WSL/_interop
```
Two caveats found the hard way:
- A leaked relay keeps its `/run/WSL/_interop` listener, so the presence of
the socket does not mean it is usable — borrowing an orphan's socket
reproduces `accept4 failed 110`.
- `comm` is not a reliable liveness test either. A relay serving a live session
is a child of its `SessionLeader`; an orphan has been reparented to init. That
parent check is the criterion that held up:
```sh
ppid=$(awk '{print $4}' "/proc/$pid/stat")
[ "$(cat "/proc/$ppid/comm")" = SessionLeader ] # true => serving a live session
```
### Related
#40650 reports the same `UtilAcceptVsock:273: accept4 failed 110` string from a
different cause (vsock connection accumulation / leaked relays). When this
occurrence started, none of that issue's markers were present: 0 leaked relays,
49 `AF_VSOCK` connections, order-7 blocks plentiful in both zones, and zero
`Waiting for abnormally long accept` lines in `dmesg`. Filing separately so the
two are not conflated.
### Diagnostic Logs
Two captures are available. Because the fallback is permanently broken on this
VM, the failure could be reproduced on demand throughout both trace windows:
- default `WSL` profile — 19 failed forwarding attempts inside the window
- `-LogProfile hvsocket` — 13 failed forwarding attempts inside the window
Both contain an account name and a corporate domain name, so rather than
attaching them here I am sending them to wsl-gh-logs@microsoft.com from
cschieli@gmail.com with this issue number in the subject, as described in
CONTRIBUTING, and will follow up with `/emailed-logs`.
Worth noting for whoever reads them: the `hvsocket` capture is the cleaner of
the two, and it is also the one that covers the layer the dead channel lives on
— the default profile does not include the HvSocket provider, and the
`WSL-Networking` profile includes it but none of the `lxcore`/`wsl*` providers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the `/init` interop server and the `UtilAcceptVsock` path, using `SocketChannel.h` and the `SendMessage` errors as entry points. Reproduce or instrument the `/run/WSL/1_interop` and `/run/WSL/2_interop` channels to understand the CLOSING state. Done means a lost in-distribution interop channel is detected and re-established instead of leaving fallback launches blocked for 10 seconds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100