tailscale / tailscale/tailscale

Discovery retries permanently pin a locally masqueraded UDP source-port rewrite (OpenWrt fw4, subnet router)

Open
#20,910 0 comments 0 reactions 0 assignees View on GitHub
bug OS-linux
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What is the issue?

When `tailscaled` runs **on a Linux router that masquerades its own outgoing traffic** (OpenWrt firewall 4), a one-time kernel connection-tracking collision can rewrite the wire source port of **one peer flow** (here: 41641 → 61303) while Tailscale continues to advertise 41641.

Discovery retries then fire every few seconds. Each retry refreshes the 60-second UDP timeout on that broken `[UNREPLIED]` record, so the mapping **never expires**. The pair stays on DERP (public relays) permanently: “direct connection not established”. Other peers of the **same** router still go direct. Deleting that single kernel record restores a direct path on the next retry.

This is **not** an ISP mapping problem and **not** `randomizeClientPort`. Both listen ports were already stable. The rewrite is local (OpenWrt firewall 4 masquerade also processing router-originated packets). The part Tailscale owns, and that an end user should not have to do by hand, is: **do not refresh a dead mapping forever**.

What I am asking Tailscale to change in the client (any one of these would let this self-heal):

1. Back off discovery retries past the kernel’s default 60-second UDP connection-tracking timeout, or
2. Detect that the advertised endpoint port and the port actually leaving the box disagree, and rebind / start a new flow, or
3. Randomize the client port **on re-establish** (the request already open as #18328) — that would also escape this class of stale record.

I am **not** asking anyone to open inbound firewall ports, add UDP 41641 forwards, change Tailscale settings on the remote peer, or flash the router.

### Full setup

Two sites. Neither has inbound holes or port forwards. Direct path is expected from hole punching only.

**Piko site**

- LAN `192.168.101.0/24`
- Router: GL.iNet BE3600, hostname `charon-piko`, firmware 4.9.0 (OpenWrt-based)
- LAN address `192.168.101.254`
- WAN `192.168.21.241/24`, default via a further upstream translator `192.168.21.254` (double network address translation)
- Public address at the time: `31.146.92.121`
- `tailscaled` **1.92.5** runs **on the router** and advertises `192.168.101.0/24` as a subnet route, because Crestron control hardware and other equipment cannot run a Tailscale client
- That router uses the London host as its **exit node** (all Piko LAN internet leaves via London)

**London site**

- LAN `192.168.55.0/24`
- Ubuntu 24.04 host, Tailscale name `lucid-tailsacle`, LAN `192.168.55.1`
- `tailscaled` **1.102.2**
- Public address `195.224.240.201`
- Offers itself as an exit node

**Path conditions**

- Both sides easy address translation: `tailscale netcheck` reported `UDP: true` and `MappingVariesByDestIP: false`
- A Mac on the **same Piko upstream network** (not behind the BE3600’s masquerade of its own sockets) held a working **direct** path to the same London host — so a direct path between the sites was possible; only the router’s pairing was stuck

### Symptom

- `tailscale ping` printed `via DERP(lhr)` or `via DERP(nue)` and `direct connection not established` for days
- Site-to-site transfers ~9–22 Mbit/s; 23% loss at an offered 20 Mbit/s
- London line ~49 Mbit/s up / 31+ down; Piko line 178/110
- Eight parallel TCP streams barely increased total throughput (one shared relay choke)
- Other peers of the same BE3600 connected **direct**

### Evidence (on the BE3600)

Broken kernel connection-tracking record (stable across days until deleted):

```
src= dst= sport=41641 dport=13147 [UNREPLIED]
reply: src=195.224.240.201 sport=13147 dst=192.168.21.241 dport=61303
```

- Tailscale advertised 41641; the wire used 61303 toward this one peer
- Fresh flows to every other destination kept source port 41641
- The masquerade rule had no port-randomization flags
- A test peer punched direct instantly
- Deleting **that one record** (`conntrack -D -p udp -s 192.168.21.241 -d 195.224.240.201 --sport 41641 --dport 13147`) made the next punch keep 41641 on the wire
- Both sides then reported direct: `via 195.224.240.201:13147` and `via 31.146.92.121:41641`, 74–77 ms

Before / after (single TCP stream, 30 s):

| Path | Relayed (before) | Direct (after) |
|---|---|---|
| London → Piko LAN device | 14–16 Mbit/s | 37.1 Mbit/s |
| Piko LAN device → London | 10–18 Mbit/s | 40.5 Mbit/s |
| Packet loss, UDP 20 Mbit/s London → Piko | 23% | 0% |
| Piko LAN internet via London exit node | ~14 Mbit/s | 42.4 Mbit/s |

### Steps to reproduce

This is a timing race at hole-punch time, then a permanent pin. It does **not** reproduce on every reboot; when it happens it does not self-heal.

1. Run `tailscaled` on an OpenWrt (or OpenWrt-based) router that masquerades WAN, including the router’s own packets (firewall 4 default).
2. Advertise a LAN subnet; use a remote peer as exit node; do **not** open inbound UDP 41641.
3. Confirm `netcheck` is easy NAT (`UDP: true`, `MappingVariesByDestIP: false`).
4. After a punch-time collision, inspect `grep "sport=41641" /proc/net/nf_conntrack` for an `[UNREPLIED]` record whose reply `dport` is **not** 41641.
5. Observe that `tailscale ping` stays on DERP while that record’s expiry timer stays near 60 s (retries refresh it).
6. `conntrack -D` of **that one record** restores a direct path on the next retry. No Tailscale config change.

### Are there any recent changes that introduced the issue?

No firmware flash and no Tailscale settings change on the London host. The pair had been stuck for days on the BE3600’s built-in Tailscale 1.92.5. Version skew (1.92.5 vs 1.102.2) was ruled out: both connect directly once the record is cleared.

### OS

Linux (GL.iNet BE3600 / OpenWrt-based firmware 4.9.0) and Linux (Ubuntu 24.04)

### OS version

GL.iNet firmware 4.9.0 (OpenWrt-based, `ipq53xx`); Ubuntu 24.04

### Tailscale version

Router: 1.92.5. London host: 1.102.2

### Other software

- OpenWrt firewall 4 masquerade on the router WAN (also processes router-originated packets — see OpenWrt issue below)
- Double NAT on the Piko WAN (`192.168.21.241` behind `192.168.21.254`)
- No inbound port forwards
- Ethernet offload (TSO/GSO/GRO) on the BE3600 LAN port was confirmed **off** and is unrelated

### Related posts already filed (please treat these as part of this report)

These describe the same incident from the three vendors that share the chain:

1. **This repository**, comment on #18328 (related, not the same as the original ISP/PPPoE report): https://github.com/tailscale/tailscale/issues/18328#issuecomment-5321032724
2. **OpenWrt** — firewall 4 masquerade rewriting the source port of router-originated traffic: https://github.com/openwrt/openwrt/issues/22765#issuecomment-5321032858
3. **GL.iNet forum** — same write-up against their firmware image: https://forum.gl-inet.com/t/built-in-tailscale-can-get-permanently-stuck-on-slow-relays-firewall-rewrites-the-routers-own-udp-source-port-be3600-firmware-4-9-0/70649

OpenWrt owns the masquerade rewrite. I am filing **this** issue so Tailscale can address the client behaviour that turns a one-time collision into a permanent relay: discovery retries never back off past the kernel UDP timeout, so an end user is left deleting a kernel record by hand.

Happy to provide the full measurement log.

-Peter Scream

Contributor guide

Open the contributing guide

Research direction

Start by tracing the discovery retry behavior in tailscaled and reproduce the scenario with `tailscale netcheck`, `tailscale ping`, and the documented `[UNREPLIED]` conntrack record. Done means a one-time local UDP source-port collision no longer keeps the peer on DERP indefinitely, without requiring manual `conntrack -D` intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.