tailscale / tailscale/tailscale

tsnet instances sometimes don't recover after an Extend Key operation

Open
#19,326 1 comment 0 reactions 0 assignees View on GitHub
bug tsnet
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What is the issue?

When a tsnet instance restarts or we do StartInteractiveLogin, it cancels (or doesn't start) any running map request, and has only an auth request poll. If our node key has expired, we have the option in the control server to extend it, but this extension triggers only an updated map response; it doesn't cancel an outstanding auth request, so a client can't see that its old key is once again valid. It recovers if you restart the client... but the reason we're using the "extend key" function is that we are likely not able to do that.

This happens in tsnet apps on every app restart, because tsnet seems to try to login in .Up() whenever its key is expired. The main tailscale client only tries to login when you explicitly start the interactive login process, which bypasses the bug (it leaves a maprequest running which then eventually updates the expired key). But "tailscale up" will trigger it.

I suggest the following fixes:

1) In the client, if we have anything that looks like a valid key, even an expired one, always run a maprequest even if we are also going to try an authrequest in parallel. This lets us continue the session immediately if the admin extends our key.

2) In the control server, if a node doesn't have an active maprequest poll (or maybe regardless of that?), and an admin disables key expiry or extends a key, immediately complete any authrequest for that same node. This might awkwardly interrupt any interactive login flow already in progress -- but the fact that the admin is extending the key right now is a pretty strong indicator that the interactive login isn't working. I think we should adopt "first valid key extension flow wins". If they don't like the 30-second extension, worst case they can then start a new interactive login.
--- this will make key extension work for nodes that haven't been updated with fix 1

3) In tsnet apps, don't immediately unnecessarily restart tsnet upon transition from Running->NeedsLogin state. Doing so aggravates the bug and doesn't provide much value (wait until the user explicitly requests an interactive login, like the tailscale client does). (This was a bug in Aperture that triggered the discovery of this problem.)

### Steps to reproduce

1. Create a tailnet with minimal key expiry (in the UI, that's 24 hours)
2. Register a simple tsnet app on the tailnet
3. Wait for the node key to expire
4. Notice that the tsnet app transitions to NeedsLogin state and data plane doesn't work (blocked by expired key)
5. Extend the key by 30 minutes
6. Notice that the tsnet app immediately transitions to Running state an data plane does work
7. Wait for the node key to expire again
8. Notice that the tsnet app transitions to NeedsLogin state and data plane doesn't work (blocked by expired key)
9. Restart the tsnet app
10. Notice that the tsnet app is in NeedsLogin still
11. Extend the key by 30 minutes
12. Notice that the tsnet app does *not* transition to Running state or start working
13. Restart the tsnet app again
14. Notice that the app successfully gets to the Running state

(Step 12 is the wrong behaviour here)

### Are there any recent changes that introduced the issue?

No, I think this is a very long standing bug that explains anecdotal problems users have experience with the Extend Key operation over the years.

### OS

_No response_

### OS version

_No response_

### Tailscale version

1.97.129

### Other software

_No response_

### Bug report

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.