microsoft / microsoft/WSL

TCP connections stall/reset in Docker containers & network namespaces connected to bridge interface when using mirrored networking mode

Open
#11,819 20 comments 0 reactions 0 assignees View on GitHub
network
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

### Windows Version

Microsoft Windows [Version 10.0.22631.3880]

### WSL Version

2.0.14.0 & 2.2.4.0

### Are you using WSL 1 or WSL 2?

- [X] WSL 2
- [ ] WSL 1

### Kernel Version

5.15.133.1-1 & 5.15.153.1-microsoft-standard-WSL2

### Distro Version

Ubuntu 24.04

### Other Software

```
$ sudo dpkg -l | grep -i docker
ii docker-buildx-plugin 0.15.1-1~ubuntu.24.04~noble amd64 Docker Buildx cli plugin.
ii docker-ce 5:27.0.3-1~ubuntu.24.04~noble amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:27.0.3-1~ubuntu.24.04~noble amd64 Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:27.0.3-1~ubuntu.24.04~noble amd64 Rootless support for Docker.
ii docker-compose 1.29.2-6ubuntu1 all define and run multi-container Docker applications with YAML
ii docker-compose-plugin 2.28.1-1~ubuntu.24.04~noble amd64 Docker Compose (V2) plugin for the Docker CLI.
rc docker.io 24.0.7-0ubuntu4 amd64 Linux container runtime
ii python3-compose 1.29.2-6ubuntu1 all Python implementation of docker-compose file specification
ii python3-docker 5.0.3-1ubuntu1 all Python 3 wrapper to access docker.io's control socket
ii python3-dockerpty 0.4.1-5 all Pseudo-tty handler for docker Python client (Python 3.x)
```

### Repro Steps

```
docker run -it ubuntu:20.04 /bin/bash
```

Install wget:

```
apt -y update && apt -y install wget
```

Then do download tests:

```
while true; do wget --max-redirect 0 -O/dev/null http://angry.lastninja.net/test.dat; done
```

At some point the connection will stall with a progress like this, note the `--.-KB/s` which indicates the connection has stalled:

```
/dev/null 5%[===> ] 5.94M --.-KB/s eta 5m 22s
```

### Expected Behavior

It should download without issue, here's the same wget on the wsl2 host (Ubuntu 24.04):

```
$ wget -O /dev/null --max-redirect 0 http://angry.lastninja.net/test.dat
--2024-07-21 21:50:58-- http://angry.lastninja.net/test.dat
Resolving angry.lastninja.net (angry.lastninja.net)... 149.248.1.168
Connecting to angry.lastninja.net (angry.lastninja.net)|149.248.1.168|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104858624 (100M)
Saving to: ‘/dev/null’

/dev/null 100%[====================================================================>] 100.00M 4.92MB/s in 24s

2024-07-21 21:51:23 (4.14 MB/s) - ‘/dev/null’ saved [104858624/104858624]
```

### Actual Behavior

This is the wget stalling in the container:

```
/dev/null 5%[===> ] 5.94M --.-KB/s eta 5m 22s
```

Strace output of `strace -s4 wget --quiet -O/dev/null --max-redirect 0 http://angry.lastninja.net/test.dat 2>&1 | tee /tmp/wget-strace.txt` this:

```
select(5, [4], NULL, NULL, {tv_sec=900, tv_usec=0}) = 1 (in [4], left {tv_sec=899, tv_usec=999421})
read(4, "\0\0\0\0"..., 8192) = 1440
write(3, "\0\0\0\0"..., 1440) = 1440
select(5, [4], NULL, NULL, {tv_sec=900, tv_usec=0}) = 1 (in [4], left {tv_sec=899, tv_usec=999998})
read(4, "\0\0\0\0"..., 8192) = 1440
write(3, "\0\0\0\0"..., 1440) = 1440
select(5, [4], NULL, NULL, {tv_sec=900, tv_usec=0}) = 1 (in [4], left {tv_sec=899, tv_usec=999643})
read(4, "\0\0\0\0"..., 8192) = 1440
write(3, "\0\0\0\0"..., 1440) = 1440
select(5, [4], NULL, NULL, {tv_sec=900, tv_usec=0}) = 1 (in [4], left {tv_sec=899, tv_usec=999998})
read(4, "\0\0\0\0"..., 8192) = 2880
write(3, "\0\0\0\0"..., 2880) = 2880
select(5, [4], NULL, NULL, {tv_sec=900, tv_usec=0} (frozen/stalls/blocks forever here)
```

In the `eth1` pcap after the http download stalls there's a TCP reset sent from the client to the server:

```
25678 40.503410 149.248.1.168 192.168.0.161 HTTP 1506 Continuation
25680 40.504042 192.168.0.161 149.248.1.168 TCP 54 60489 → 80 [RST] Seq=155 Win=0 Len=0
25681 40.504044 192.168.0.161 149.248.1.168 TCP 54 60489 → 80 [RST] Seq=155 Win=0 Len=0
.
.
.
```

In the capture of the docker container interface `vethcdca12b.docker0` after the connection stalls I see:

```
25582 34.784331 149.248.1.168 172.17.0.2 HTTP 2946 Continuation
25583 34.784468 172.17.0.2 149.248.1.168 TCP 66 54612 → 80 [ACK] Seq=155 Ack=85918330 Win=3015808 Len=0 TSval=92428168 TSecr=1890299880
25584 34.784485 172.17.0.1 172.17.0.2 ICMP 94 Destination unreachable (Network unreachable)
25585 34.785821 149.248.1.168 172.17.0.2 HTTP 2946 Continuation
25586 34.785876 149.248.1.168 172.17.0.2 HTTP 1506 Continuation
25587 34.785960 172.17.0.2 149.248.1.168 TCP 66 54612 → 80 [ACK] Seq=155 Ack=85922650 Win=3015808 Len=0 TSval=92428170 TSecr=1890299881
25588 34.785970 172.17.0.1 172.17.0.2 ICMP 94 Destination unreachable (Network unreachable)
25589 34.786762 149.248.1.168 172.17.0.2 HTTP 10146 Continuation
.
.
.
```

The destination unreachable packet (25584) looks like this:

```
Internet Control Message Protocol
Type: 3 (Destination unreachable)
Code: 0 (Network unreachable)
Checksum: 0x3717 [correct]
[Checksum Status: Good]
Unused: 00000000
Internet Protocol Version 4, Src: 172.17.0.2, Dst: 149.248.1.168
Transmission Control Protocol, Src Port: 54612, Dst Port: 80, Seq: 1879835401, Ack: 1155371443
Source Port: 54612
Destination Port: 80
Sequence Number: 1879835401
[Stream index: 0]
[Conversation completeness: Incomplete, DATA (15)]
Acknowledgment Number: 1155371443 (relative ack number)
Acknowledgment number (raw): 1155371443
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window: 23561
[Calculated window size: 23561]
[Window size scaling factor: 128]
Checksum: 0x43da [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
```

The Ack packet (25583) that is sent by the client just prior to receiving that connection reset has the following in the TCP headers:

```
Transmission Control Protocol, Src Port: 54612, Dst Port: 80, Seq: 155, Ack: 85918330, Len: 0
Source Port: 54612
Destination Port: 80
[Stream index: 0]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 0]
Sequence Number: 155 (relative sequence number)
Sequence Number (raw): 1879835401
[Next Sequence Number: 155 (relative sequence number)]
Acknowledgment Number: 85918330 (relative ack number)
Acknowledgment number (raw): 1155371443
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window: 23561
[Calculated window size: 3015808]
[Window size scaling factor: 128]
Checksum: 0x43da [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
```

The discrepency I see is that the raw acknowledgement number is correct, while there seems to be a discrepency with the relative acknowledgement number. I'm not sure if this is the root cause, and I'm not sure why an ICMP message is even being sent by the docker gateway IP `172.17.0.1`.

### Diagnostic Logs

[pcaps.zip](https://github.com/user-attachments/files/16324291/pcaps.zip)
[WslNetworkingLogs-2024-07-21_21-56-30.zip](https://github.com/user-attachments/files/16324295/WslNetworkingLogs-2024-07-21_21-56-30.zip)
[wget-strace.txt](https://github.com/user-attachments/files/16324298/wget-strace.txt)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the supplied docker run command and repeated wget download, then compare the container, eth1, and vethcdca12b.docker0 captures. Review the wget strace and diagnostic archives alongside the ICMP destination-unreachable and TCP reset packets. Done means mirrored networking no longer stalls or resets downloads from a bridged container.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.