microsoft / microsoft/WSL

Blocking I/O behavior in localhost relay hangs with full duplex traffic due to half duplex forwarding logic.

Open
#10,688 10 comments 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

network
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

## Problem Description

The implementation of localhost relaying on the Linux portion is using blocking I/O in a single thread to copy both sides of a connection. While this will work with a half-duplex request-reply style system (e.g. http/1.1), it will lead to connection hangs when there is a multiplexed protocol and bidirectional traffic (e.g. ssh, http/2, etc).

As an example using the reproducer linked later in this report, the following strace output shows a thread in the localhost forwarding process (init ran under the name localhost in the host CBL OS) hung while writing to its destination VSOCK connection (FD=6). This blocks forever since the other end of the connection (wslrelay.exe + the client win application) has a full read buffer, and the protocol can not advance until the WSL init process reads from the VSOCK fd (opening up more buffer space between the two ends).

```
[pid 778] ppoll([{fd=6, events=POLLIN}, {fd=9, events=POLLIN}], 2, NULL, NULL, 8) = 2 ([{fd=6, revents=POLLIN}, {fd=9, revents=POLLIN}])
[pid 778] read(6, "table language.\n\nThe most import"..., 131072) = 16384
[pid 778] write(9, "table language.\n\nThe most import"..., 16384) = 16384
[pid 778] read(9, "suitable language.\n\nThe most imp"..., 131072) = 131072
[pid 778] write(6, "suitable language.\n\nThe most imp"..., 131072 <----- Hang
```

While the Windows side (wslrelay.exe) is using Overlapped I/O, it simulates a similar write-blocking pattern. Relay.cpp polls reads on both the VSOCK and TCP win app connection using WaitForMultipleObjects; however, the writing portion on both ends calls an Overlapped WriteFile immediately followed by a WaitForMultipleObjects on that descriptor, blocking until the write fully completes.

## Suggested Fix

Oh the Linux side, I would recommend switching the socket to non-blocking, and introducing a buffer for each direction of the connection, which is managed by a simple state machine. The limit for each direction would need to be reduced by the leftover amount of an incomplete write (essentially pre-filling the read buffer) to establish back-pressure to and allow SOCK_STREAM flow-control to throttle an abusive peer, as well as limit resource consumption.

Alternatively, if you prefer to stay with blocking strategies, you could go with a two-thread per-connection approach, where each thread copies in opposite directions of the connection. In this model, you would rely on cooperative write shutdown triggering graceful close of the connection.

On the Windows side, in addition to mirroring the above strategies, you could alter the existing Overlapped I/O model to have a singular event handler that covers writes as well as reads, although you would still want to ensure some form of back-pressure is established (limiting buffer / pending i/o) to prevent runaway event queuing.

## Symptoms

This problem can often be spotted by observing persistent non-zero Send/Receive socket queues on the Linux host (indicates the application is not consuming buffer):

```
wsl --debug-shell
# ss -n --tcp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
ESTAB 418552 0 127.0.0.1:57366 127.0.0.1:9191
ESTAB 0 0 127.0.0.1:9191 127.0.0.1:57366
```

## Reproducer

Since the existing localhost relay implementation is all local with low latency and generous buffers, hang scenarios are timing-sensitive. To make it easier to recreate, I created a stress scenario of a simple synthetic multiplexed protocol here:
https://github.com/n1hility/duplex-stress/ (Under AL2 license, but let me know if you prefer under any other terms)

In my testing it triggers a hang condition pretty quickly (tested on multiple environments including 10, 11, Windows ARM and x64). It's written in Golang for Win/Linux portability and to avoid any runtime requirements.

To recreate:

1. Install Golang 1.21+, Build it:
```powershell
PS> .\build.ps1
```

2. Run the server in WSL:
```powershell
PS> wsl
$ ./duplex-stress server 0.0.0.0 9191
```

3. Run the client from Windows
```powershell
PS> .\duplex-stress client 127.0.0.1 9191
```

5. Observe hang (text will stop printing, socket will be stuck as described above)

6. Ctrl-C both and restart using direct TCP

7. Start new server on Linux with a different port (9191 is now dead from hang)
```bash
$ ./duplex-stress server 0.0.0.0 9192
```

9. Connect client on Windows using IP addr of WSL dist
```powershell
PS> .\duplex-stress client $(wsl hostname -I).Trim() 9192
```

10. Observe works

## Versions

Observed reproducer with 2.0.5 and 1.2.5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Windows relay implementation in Relay.cpp and reproduce the hang using the linked duplex-stress server and client commands. Compare localhost relay behavior with direct TCP, then verify that sustained bidirectional traffic completes without a stuck connection or persistent socket queues.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
39/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.