tailscale / tailscale/tailscale
Probable race condition in event bus port update mechanism causes direct connections to be impossible and endpoints not reported to the control plane
- Dominant language
- Go
- Stars
- 36.5k
- Forks
- 3.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 123
Description
### What is the issue?
Server provider: Hetzner
OS: Ubuntu 22.04/24.04
Firewall: UFW
We've updated running systems to the 1.92.3 version of Tailscale and lost all direct connections between servers.
Rebooting the servers restored direct connection. Rolling the version back to 1.88.4 fixed direct connections without reboots. Tried flushing conntrack, tried tailscale down & up, tried restarting the tailscaled service, but those things didn't help
We've updated around 70 machines in the span of 10 minutes and all of them experienced this issue and it kept being a problem for an hour until I've rolled back the update.
```
tailscale ping MACHINE-2
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
direct connection not established
```
```
Report:
* Time: 2025-12-17T23:48:21.266329363Z
* UDP: true
* IPv4: yes, PUBLIC.IP.MACHINE.1:59002
* IPv6: no, unavailable in OS
* MappingVariesByDestIP: false
* PortMapping:
* Nearest DERP: Nuremberg
* DERP latency:
- nue: 2.7ms (Nuremberg)
- fra: 5.6ms (Frankfurt)
- ams: 10.9ms (Amsterdam)
- par: 14.4ms (Paris)
- lhr: 15.9ms (London)
- waw: 21.3ms (Warsaw)
- hel: 25.1ms (Helsinki)
```
```
Report:
* Time: 2025-12-17T23:55:38.496953714Z
* UDP: true
* IPv4: yes, PUBLIC.IP.MACHINE.2:34778
* IPv6: no, unavailable in OS
* MappingVariesByDestIP: false
* PortMapping:
* Nearest DERP: Nuremberg
* DERP latency:
- nue: 2.7ms (Nuremberg)
- fra: 5.6ms (Frankfurt)
- ams: 10.9ms (Amsterdam)
- par: 14.1ms (Paris)
- lhr: 15.8ms (London)
- waw: 21.3ms (Warsaw)
- hel: 25ms (Helsinki)
- mad: 33ms (Madrid)
```
### Steps to reproduce
Update tailscale via apt to the 1.92.3 version on servers already experiencing a pretty constant connections from other nodes in the Tailnet.
### Are there any recent changes that introduced the issue?
_No response_
### OS
Linux
### OS version
Ubuntu 22.04, Ubuntu 24.04
### Tailscale version
1.92.3
### Other software
UFW, HAProxy, JVM apps, Datadog
### Bug report
BUG-a5226304a670b4f115a9db0a1aa230e2f116c4d43f0050c4fe07015f3f960b0d-20251217235845Z-6bed08e43c44a027
Contributor guide
Assessment
This issue has not been assessed yet.