tailscale / tailscale/tailscale

Probable race condition in event bus port update mechanism causes direct connections to be impossible and endpoints not reported to the control plane

Open
#18,240 6 comments 0 reactions 0 assignees View on GitHub
bug OS-linux
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What is the issue?

Server provider: Hetzner
OS: Ubuntu 22.04/24.04
Firewall: UFW

We've updated running systems to the 1.92.3 version of Tailscale and lost all direct connections between servers.

Rebooting the servers restored direct connection. Rolling the version back to 1.88.4 fixed direct connections without reboots. Tried flushing conntrack, tried tailscale down & up, tried restarting the tailscaled service, but those things didn't help

We've updated around 70 machines in the span of 10 minutes and all of them experienced this issue and it kept being a problem for an hour until I've rolled back the update.

```
tailscale ping MACHINE-2
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
pong from MACHINE-2 (TAILNET.IP.MACHINE.2) via DERP(nue) in 6ms
direct connection not established
```

```
Report:
* Time: 2025-12-17T23:48:21.266329363Z
* UDP: true
* IPv4: yes, PUBLIC.IP.MACHINE.1:59002
* IPv6: no, unavailable in OS
* MappingVariesByDestIP: false
* PortMapping:
* Nearest DERP: Nuremberg
* DERP latency:
- nue: 2.7ms (Nuremberg)
- fra: 5.6ms (Frankfurt)
- ams: 10.9ms (Amsterdam)
- par: 14.4ms (Paris)
- lhr: 15.9ms (London)
- waw: 21.3ms (Warsaw)
- hel: 25.1ms (Helsinki)

```

```
Report:
* Time: 2025-12-17T23:55:38.496953714Z
* UDP: true
* IPv4: yes, PUBLIC.IP.MACHINE.2:34778
* IPv6: no, unavailable in OS
* MappingVariesByDestIP: false
* PortMapping:
* Nearest DERP: Nuremberg
* DERP latency:
- nue: 2.7ms (Nuremberg)
- fra: 5.6ms (Frankfurt)
- ams: 10.9ms (Amsterdam)
- par: 14.1ms (Paris)
- lhr: 15.8ms (London)
- waw: 21.3ms (Warsaw)
- hel: 25ms (Helsinki)
- mad: 33ms (Madrid)
```

### Steps to reproduce

Update tailscale via apt to the 1.92.3 version on servers already experiencing a pretty constant connections from other nodes in the Tailnet.

### Are there any recent changes that introduced the issue?

_No response_

### OS

Linux

### OS version

Ubuntu 22.04, Ubuntu 24.04

### Tailscale version

1.92.3

### Other software

UFW, HAProxy, JVM apps, Datadog

### Bug report

BUG-a5226304a670b4f115a9db0a1aa230e2f116c4d43f0050c4fe07015f3f960b0d-20251217235845Z-6bed08e43c44a027

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.