tailscale / tailscale/tailscale-rs
control/netmap: reconnect if a keepalive hasn't been received recently
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.2k
- Forks
- 61
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 22
Description
The control plane purposely disables layer 4 TCP keepalives^1 on control connections in favor of layer 7 keepalives via the MapResponse::keep_alive field[^2]. As a result, the control client can end up in a situation where the TCP stream is still ESTABLISHED as far as it knows, even though the control plane has effectively dropped the connection. This happens most often when the machine running a tailscale-rs device goes to sleep/suspends, then wakes up; ss/netstat show the TCP control plane connection as established, but we're receiving no netmap messages, because the control plane has written us off as dead. This means any sleep/suspend that lasts longer than the control plane's layer 7 keepalive interval can break a tailscale-rs device, requiring a process restart.
To fix this, the control client needs to track the time since the last MapResponse message was received[^3]. If the control client hasn't received a message in ~120 seconds, it should tear down the existing TCP connection and start a new one.
[^2]: This was done for battery life reasons; in ~2021, some mobile devices would wake the main processor to handle receiving/sending TCP keepalives, which depletes battery life. By disabling layer 4 keepalives for control connections and moving the keepalive concept to layer 7, the control plane helps preserve battery life on those devices, but requires all clients to manually track connection liveness and reconnect after extended idle time. There's been some internal discussion about re-introducing layer 4 keepalives to the control connection, as most mobile devices have a low-power processor/DSP that can handle TCP keepalives without waking the whole device/depleting battery life excessively. However, it still may not be possible if system APIs don't (sanely) expose TCP keepalive state to the application, and requires some up-front research/design.
[^3]: Per the tailscaled code, any valid message received resets the watchdog timer, not just ones with MapResponse.KeepAlive set.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Rust control client and its handling of MapResponse messages and connection teardown. Track when any valid message is received, then verify that an idle connection is closed and re-established after roughly 120 seconds without messages. Done means a suspended or otherwise abandoned control connection reconnects without requiring a process restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100