`ExistsViaCoderConnect` false positives when Coder Desktop has stale DNS but no working tunnel, causing `coder ssh --stdio` to hang
- Dominant language
- Go
- Stars
- 14.5k
- Forks
- 1.5k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 601
Description
## Summary
When Coder Desktop (Coder Connect) is in a broken-but-not-cleaned-up state on the
user's machine — i.e. the NRPT rule and embedded DNS responder are still in place
and answering with addresses inside `tailnet.CoderServicePrefix` (`fd60:627a:a42b::/48`),
but the tailnet itself is not actually carrying traffic — `coder ssh --stdio` is
fooled into taking the Coder Connect direct-dial path and hangs until timeout.
End-user symptom: `ssh coder.myworkspace` (via the `coder config-ssh`-generated
ProxyCommand) fails with:
```
error: dial coder connect host:
- dial tcp [fd60:627a:a42b:....]:22: connect: operation timed out
```
…while `coder ssh myworkspace` (which builds its own tailnet client in-process and
does not consult `ExistsViaCoderConnect`) works fine against the same workspace
from the same machine.
A reboot (which clears Coder Desktop's residual NRPT rule / Wintun adapter / DNS
state) restores `ssh coder.*`.
## Root cause
[`ExistsViaCoderConnect`]()
decides whether Coder Connect is "available" purely by doing an IPv6 DNS lookup
and checking whether any returned address falls inside `CoderServicePrefix`:
```go
ips, err := resolver.LookupIP(ctx, "ip6", hostname)
// ...
for _, ip := range ips {
if tailnet.CoderServicePrefix.AsNetip().Contains(addr) {
return true, nil
}
}
```
There is no liveness probe of the tunnel itself. As a result, any environment
where the OS resolver still returns a `fd60::/48` address — even though the
underlying tunnel is dead — causes `coder ssh --stdio` (called from the
ProxyCommand `coder ssh --stdio --ssh-host-prefix coder. %h`) to enter
`runCoderConnectStdio`, which then tries `dial tcp [fd60:…]:22` and blocks
until the OS-level connect timeout.
The function's own doc comment acknowledges the gap:
> This doesn't guarantee the workspace is actually reachable, if, for example,
its agent is unhealthy, but rather that Coder Connect knows about the workspace
and advertises the hostname via DNS.
…but the caller in `cli/ssh.go` treats a `true` return as sufficient to commit
to the Connect-only path.
## Reproduction
1. On Windows (or macOS) with Coder Desktop installed and Coder Connect previously
enabled, get the host into a state where the NRPT rule for `.coder` (or
equivalent on macOS) and the embedded DNS responder are still present, but the
tunnel is no longer functional. In practice this seems to happen after:
* network/VPN flap
* waking from sleep
* control-plane DNS briefly failing to resolve, then recovering
2. `Resolve-DnsName myworkspace.coder` still returns an `fd60:…` address.
3. Run `ssh coder.myworkspace` via a `coder config-ssh`-generated config (or an
equivalent hand-written one using `coder ssh --stdio --ssh-host-prefix coder.`).
4. Observe the hang and the `dial tcp [fd60:...]:22: connect: operation timed out`.
5. `coder ssh myworkspace` (no `--stdio`) succeeds against the same workspace.
## Suggested fix
`ExistsViaCoderConnect` (or its caller in `cli/ssh.go`) should additionally
confirm the tunnel is actually carrying traffic before committing to the Connect
path. Options, roughly in increasing order of invasiveness:
1. **Fast liveness probe** — after the DNS check returns a `fd60::/48` address,
attempt a very short TCP dial (e.g. to the agent API port `4`, or to `22`)
with a sub-second timeout. If that fails, treat Connect as unavailable and
fall through to the non-Connect path.
2. **Local IPC hint from Coder Desktop** — have Coder Desktop expose a local
"tunnel healthy" sentinel (named pipe / UDS / loopback HTTP endpoint, or a
sentinel hostname that only resolves when the dataplane is live) that the
CLI can check cheaply.
3. **Tear-down on Coder Desktop side** — Coder Desktop should remove its NRPT
rule / DNS entries when the tunnel becomes unhealthy, so `LookupIP` returns
NXDOMAIN and `ExistsViaCoderConnect` correctly returns `false`. (Tracked
separately on the Desktop side as coder/coder-desktop-windows#171.)
Option 1 is the smallest, most defensive change here and would prevent the
hang regardless of what Coder Desktop does.
## Related
* coder/coder-desktop-windows#171 — companion Desktop-side issue: "Tray reports
Coder Connect as healthy while tunnel/DNS is broken; stale NRPT rule causes
downstream tooling to hang or fail."
* coder/coder-desktop-windows#170 — "DNS publishing silently fails — workspace
lookups return empty answers despite Connect being on" (the inverse failure
mode of the same underlying state-management gap).
* [coder/coder-desktop-windows#149]() — "Not connected after waking from sleep"
(one of the triggers for the broken-but-not-cleaned-up state).
## Notes
* Reported by a user in the field; reboot resolved it, confirming residual
Coder Desktop / OS-resolver state as the trigger.
* The `~/.ssh/config` produced by `coder config-ssh` is not at fault — the same
hang reproduces with any config that routes through `coder ssh --stdio`.
* This is independent of the `coder` agent-name collision class of bug; in the
observed case the agent was named `main`.
Created on behalf of @mdanter
Contributor guide
Assessment
This issue has not been assessed yet.