Blocking I/O behavior in localhost relay hangs with full duplex traffic due to half duplex forwarding logic.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 33.7k
- Forks
- 1.8k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 116
Description
## Problem Description
The implementation of localhost relaying on the Linux portion is using blocking I/O in a single thread to copy both sides of a connection. While this will work with a half-duplex request-reply style system (e.g. http/1.1), it will lead to connection hangs when there is a multiplexed protocol and bidirectional traffic (e.g. ssh, http/2, etc).
As an example using the reproducer linked later in this report, the following strace output shows a thread in the localhost forwarding process (init ran under the name localhost in the host CBL OS) hung while writing to its destination VSOCK connection (FD=6). This blocks forever since the other end of the connection (wslrelay.exe + the client win application) has a full read buffer, and the protocol can not advance until the WSL init process reads from the VSOCK fd (opening up more buffer space between the two ends).
```
[pid 778] ppoll([{fd=6, events=POLLIN}, {fd=9, events=POLLIN}], 2, NULL, NULL, 8) = 2 ([{fd=6, revents=POLLIN}, {fd=9, revents=POLLIN}])
[pid 778] read(6, "table language.\n\nThe most import"..., 131072) = 16384
[pid 778] write(9, "table language.\n\nThe most import"..., 16384) = 16384
[pid 778] read(9, "suitable language.\n\nThe most imp"..., 131072) = 131072
[pid 778] write(6, "suitable language.\n\nThe most imp"..., 131072 <----- Hang
```
While the Windows side (wslrelay.exe) is using Overlapped I/O, it simulates a similar write-blocking pattern. Relay.cpp polls reads on both the VSOCK and TCP win app connection using WaitForMultipleObjects; however, the writing portion on both ends calls an Overlapped WriteFile immediately followed by a WaitForMultipleObjects on that descriptor, blocking until the write fully completes.
## Suggested Fix
Oh the Linux side, I would recommend switching the socket to non-blocking, and introducing a buffer for each direction of the connection, which is managed by a simple state machine. The limit for each direction would need to be reduced by the leftover amount of an incomplete write (essentially pre-filling the read buffer) to establish back-pressure to and allow SOCK_STREAM flow-control to throttle an abusive peer, as well as limit resource consumption.
Alternatively, if you prefer to stay with blocking strategies, you could go with a two-thread per-connection approach, where each thread copies in opposite directions of the connection. In this model, you would rely on cooperative write shutdown triggering graceful close of the connection.
On the Windows side, in addition to mirroring the above strategies, you could alter the existing Overlapped I/O model to have a singular event handler that covers writes as well as reads, although you would still want to ensure some form of back-pressure is established (limiting buffer / pending i/o) to prevent runaway event queuing.
## Symptoms
This problem can often be spotted by observing persistent non-zero Send/Receive socket queues on the Linux host (indicates the application is not consuming buffer):
```
wsl --debug-shell
# ss -n --tcp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
ESTAB 418552 0 127.0.0.1:57366 127.0.0.1:9191
ESTAB 0 0 127.0.0.1:9191 127.0.0.1:57366
```
## Reproducer
Since the existing localhost relay implementation is all local with low latency and generous buffers, hang scenarios are timing-sensitive. To make it easier to recreate, I created a stress scenario of a simple synthetic multiplexed protocol here:
https://github.com/n1hility/duplex-stress/ (Under AL2 license, but let me know if you prefer under any other terms)
In my testing it triggers a hang condition pretty quickly (tested on multiple environments including 10, 11, Windows ARM and x64). It's written in Golang for Win/Linux portability and to avoid any runtime requirements.
To recreate:
1. Install Golang 1.21+, Build it:
```powershell
PS> .\build.ps1
```
2. Run the server in WSL:
```powershell
PS> wsl
$ ./duplex-stress server 0.0.0.0 9191
```
3. Run the client from Windows
```powershell
PS> .\duplex-stress client 127.0.0.1 9191
```
5. Observe hang (text will stop printing, socket will be stuck as described above)
6. Ctrl-C both and restart using direct TCP
7. Start new server on Linux with a different port (9191 is now dead from hang)
```bash
$ ./duplex-stress server 0.0.0.0 9192
```
9. Connect client on Windows using IP addr of WSL dist
```powershell
PS> .\duplex-stress client $(wsl hostname -I).Trim() 9192
```
10. Observe works
## Versions
Observed reproducer with 2.0.5 and 1.2.5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Windows relay implementation in Relay.cpp and reproduce the hang using the linked duplex-stress server and client commands. Compare localhost relay behavior with direct TCP, then verify that sustained bidirectional traffic completes without a stuck connection or persistent socket queues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 39/100