tailscale / tailscale/tailscale-rs

control/netmap: reconnect if a keepalive hasn't been received recently

Open
#210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement tech debt
Dominant language
Rust
Stars
1.2k
Forks
61
Avg merge
2d 13h
Merged PRs (30d)
22

Description

The control plane purposely disables layer 4 TCP keepalives^1 on control connections in favor of layer 7 keepalives via the MapResponse::keep_alive field[^2]. As a result, the control client can end up in a situation where the TCP stream is still ESTABLISHED as far as it knows, even though the control plane has effectively dropped the connection. This happens most often when the machine running a tailscale-rs device goes to sleep/suspends, then wakes up; ss/netstat show the TCP control plane connection as established, but we're receiving no netmap messages, because the control plane has written us off as dead. This means any sleep/suspend that lasts longer than the control plane's layer 7 keepalive interval can break a tailscale-rs device, requiring a process restart.

To fix this, the control client needs to track the time since the last MapResponse message was received[^3]. If the control client hasn't received a message in ~120 seconds, it should tear down the existing TCP connection and start a new one.

[^2]: This was done for battery life reasons; in ~2021, some mobile devices would wake the main processor to handle receiving/sending TCP keepalives, which depletes battery life. By disabling layer 4 keepalives for control connections and moving the keepalive concept to layer 7, the control plane helps preserve battery life on those devices, but requires all clients to manually track connection liveness and reconnect after extended idle time. There's been some internal discussion about re-introducing layer 4 keepalives to the control connection, as most mobile devices have a low-power processor/DSP that can handle TCP keepalives without waking the whole device/depleting battery life excessively. However, it still may not be possible if system APIs don't (sanely) expose TCP keepalive state to the application, and requires some up-front research/design.

[^3]: Per the tailscaled code, any valid message received resets the watchdog timer, not just ones with MapResponse.KeepAlive set.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Rust control client and its handling of MapResponse messages and connection teardown. Track when any valid message is received, then verify that an idle connection is closed and re-established after roughly 120 seconds without messages. Done means a suspended or otherwise abandoned control connection reconnects without requiring a process restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.