Performance Issue: Poor Bandwidth on TCP

## Summary

I am experiencing poor bandwidth performance and high latency when using Tailscale for a media server use case. Despite following Tailscale's [[performance best practices](https://tailscale.com/kb/1320/performance-best-practices#linux-optimizations-for-subnet-routers-and-exit-nodes)](https://tailscale.com/kb/1320/performance-best-practices#linux-optimizations-for-subnet-routers-and-exit-nodes), the observed speeds are significantly lower than expected. This issue persists across multiple tests, with varying tools such as `iperf3` and `speedtest-cli`. Below, I have detailed my setup, use case, and extensive testing results.

---

## Environment

### Devices:
1. **Server**: A PC in India running on a Ryzen 3900X (average CPU usage: 1-2%)
- **Tailscale version**: 1.78.1 (latest)
- **Operating system**: Ubuntu 22.04 LTS (Linux kernel 6.2)
2. **Client**: A MacBook Pro M1 in the US
- **Tailscale version**: 1.78.1 (latest)

### Network Speeds:
- **Server Internet Speed**: 200 Mbps symmetrical
- **Client Internet Speed**: Gigabit download (with Wi-Fi slowdown), ~20 Mbps upload

### Use Case:
- Hosting a **media server** (Jellyfin) on the server, with the client streaming data from it.
- Data flow: **Server -> Client**
- Expected bottleneck: Server's upload speed at 200 Mbps.

---

## Steps Followed

1. Configured Tailscale following the official [[performance best practices](https://tailscale.com/kb/1320/performance-best-practices#linux-optimizations-for-subnet-routers-and-exit-nodes)](https://tailscale.com/kb/1320/performance-best-practices#linux-optimizations-for-subnet-routers-and-exit-nodes):
- Verified kernel version (Linux 6.2)
- Applied `ethtool` optimizations for UDP throughput.
2. Tested direct connectivity with `mtr` and confirmed no relays are involved.
3. Conducted bandwidth tests using `iperf3` (default flags and with UDP).
4. Compared speed test results with and without Tailscale enabled.

---

## Observations

### Direct Connectivity (MTR Results)
#### Server -> Client:
```
My traceroute [v0.95]
server (100.73.251.111) -> 100.104.170.76 (100.104.170.76) 2025-01-25T12:15:43+0530
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 100.104.170.76 0.2% 466 291.9 278.0 265.8 361.9 11.0
```

#### Client -> Server:
```
My traceroute [v0.95]
Ashays-MacBook-Pro.local (100.104.170.76) -> server (100.73.251.111) 2025-01-24T22:46:21-0800
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. server.tail82e7b.ts.net 0.0% 512 284.2 277.6 266.5 362.1 8.9
```

---

### Speed Tests

#### Server (Native):
```
Server: Tata Play Fiber - Mumbai (id: 23647)
ISP: Oneott Intertainment
Idle Latency: 1.92 ms (jitter: 0.62ms, low: 1.82ms, high: 3.04ms)
Download: 216.79 Mbps (data used: 113.7 MB)
89.05 ms (jitter: 26.71ms, low: 3.64ms, high: 231.05ms)
Upload: 219.71 Mbps (data used: 159.7 MB)
20.04 ms (jitter: 2.61ms, low: 5.41ms, high: 35.33ms)
Packet Loss: 0.0%
```

#### Client (Native):
```
Testing from Comcast Cable ()...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by SiteHUB Agency (Fremont, CA) [80.75 km]: 23.996 ms
Testing download speed................................................................................
Download: 315.24 Mbit/s
Testing upload speed......................................................................................................
Upload: 14.42 Mbit/s
```

### Results with Tailscale

#### Without Tailscale:
```
Download: 240.67 Mbit/s
Upload: 15.36 Mbit/s

Download: 481.46 Mbit/s
Upload: 22.25 Mbit/s
```

#### With Tailscale (No Exit Node):
```
Download: 316.54 Mbit/s
Upload: 15.38 Mbit/s

Download: 324.80 Mbit/s
Upload: 15.77 Mbit/s
```

#### With Exit Node Pointing to Server:
```
Hosted by Logon Broadband (Mumbai) [12.46 km]: 992.592 ms
Download: 1.68 Mbit/s
Upload: 2.19 Mbit/s

Hosted by Bharti Airtel Limited (Chandivali) [4.01 km]: 992.241 ms
Download: 1.61 Mbit/s
Upload: 2.42 Mbit/s
```

**Key Observations:**
1. **High Latency:** ~992ms with the exit node enabled, even though direct connection latency is ~250ms.
2. **Severely Reduced Bandwidth:** Exit node download speed drops to ~1.6 Mbps, significantly below expectations.

---

### Bandwidth Testing with iPerf3

#### Default Flags:
```
[ 5] 0.00-10.34 sec 5.38 MBytes 4.36 Mbits/sec 2 sender
[ 5] 0.00-10.27 sec 5.12 MBytes 4.18 Mbits/sec 2 sender
[ 5] 0.00-10.27 sec 14.5 MBytes 11.8 Mbits/sec 10 sender
[ 5] 0.00-10.28 sec 17.1 MBytes 14.0 Mbits/sec 3 sender
...
```

#### With UDP and Bitrate Set to 200 Mbps:
```
iperf3-darwin --udp --bitrate 200M -c server -R
Connecting to host server, port 5201
Reverse mode, remote host server is sending
[ 7] local 100.104.170.76 port 62738 connected to 100.73.251.111 port 5201
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 7] 0.00-1.00 sec 23.8 MBytes 200 Mbits/sec 0.089 ms 6/20356 (0.029%)
[ 7] 1.00-2.00 sec 23.8 MBytes 199 Mbits/sec 0.054 ms 38/20323 (0.19%)
...
```

**Key Observations:**
- **UDP Results:** Suggest the ability to maintain 200 Mbps with minimal packet loss (~0.05%).
- **TCP Results:** Show poor performance, with speeds ranging from 4 Mbps to 20 Mbps.

---

## Questions and Concerns

1. **Why are the speeds so poor (~10% of the actual internet speed) when using Tailscale, especially with the exit node?**
- Is this a limitation of Tailscale's architecture, or is there an underlying misconfiguration in my setup?

2. **Why does `iperf3` with default TCP flags show such bad results, even though UDP results suggest the connection is capable of 200 Mbps?**

3. **Why does enabling the exit node result in such high latency (~992ms) and drastically reduced speeds?**

4. **Could there be specific optimizations or configurations I am missing for a long-distance connection (India -> US)?**

---

## Additional Notes

1. This issue is not related to Jellyfin itself, as the performance bottleneck persists across all network tests.
2. All tests were conducted using the latest versions of Tailscale on both devices, with no relays involved.
3. I am happy to provide more logs or perform additional tests if needed to assist in debugging.

---

Thank you for your support and for creating such a fantastic product! I look forward to hearing from the Tailscale team regarding potential solutions or insights into this issue.

### Steps to reproduce

_No response_

### Are there any recent changes that introduced the issue?

_No response_

### OS

Linux, macOS

### OS version

Ubuntu 24.04.1 LTS

### Tailscale version

latest, on both machines

### Other software

_No response_

### Bug report

BUG-e724cb237ecea1a9ef2c21601b135ad7fe056b91cf37f2ca4d7cba4a8b0e8c50-20250127090438Z-18a5888d9418573b