guestSocket forwards die with the SSH ControlMaster and are never re-created (silent on vz)
- Dominant language
- Go
- Stars
- 21.9k
- Forks
- 957
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 53
Description
### Description
Unix-socket forwards declared in `portForwards` (`guestSocket` rules, e.g. the docker socket in the docker template, or colima's `docker.sock`) are created exactly once at instance start, as `ssh -O forward` registrations against the SSH ControlMaster. If the master later dies, every forward it owned dies with it. The hostagent never detects this and never re-creates the forwards, so the host-side socket file is left behind as a stub that refuses every connection until the instance is restarted.
Host sleep is the reliable trigger (the master's TCP connection does not survive it). We also see the same failure on machines that have not slept, which points at idle-flow reaping in the usernet stack or network transitions killing a long-idle, keepalive-less connection — flagged as suspected rather than proven; the sleep case reproduces deterministically below.
On vz the failure is completely silent: the guest agent connection rides vsock (`GuestAgentConn`), which survives sleep, so the reconnect logic in `watchGuestAgentEvents` never fires. The hostagent keeps logging routine activity (time sync, guest agent events) with no error while the socket is dead. Inside the VM everything is healthy — when we first hit this in the field, dockerd had 24 h of uptime and 15 healthy containers while the host-side socket had been dead for hours. Users typically "fix" it with a full instance restart, which is how it hides inside vaguer reports.
### Steps to reproduce
Any instance with a `guestSocket` port forward works; we used a docker runtime instance on vz.
1. Start the instance and verify the socket forward works:
```console
$ curl --unix-socket ~/.lima//sock/docker.sock http://localhost/_ping
OK
```
2. Find the ControlMaster:
```console
$ ssh -F ~/.lima//ssh.config -O check lima-
Master running (pid=13185)
```
3. Simulate what host sleep does to the master's connection:
```console
$ kill -9 13185
```
4. The socket is now dead and stays dead. We probed every 10 s for 120 s: 13/13 attempts refused, no recovery, and `ha.stderr.log` shows no error, no reconnect attempt, no re-forward — the hostagent has not noticed.
5. `limactl shell ` still works, and (since #4913) revives a fresh master — but the socket forward stays dead even after that. Reviving the master does not re-issue the forward registrations; they must be re-issued explicitly.
### Root cause (from reading master @ 0990395)
- `watchGuestAgentEvents` sets up the `guestSocket` forwards once, outside any retry or reconcile path: https://github.com/lima-vm/lima/blob/0990395/pkg/hostagent/hostagent.go#L750-L758
- The only re-forward logic on reconnect covers the guest agent's own `ga.sock`, and it only runs when the guest agent connection breaks. On vz the guest agent uses vsock (https://github.com/lima-vm/lima/blob/0990395/pkg/driver/vz/vz_driver_darwin.go#L542-L548), so a dead SSH master never surfaces there.
- Nothing keeps the master alive or notices its death: `SSHOpts` sets `ControlMaster=auto` / `ControlPersist=yes` with no `ServerAliveInterval` (https://github.com/lima-vm/lima/blob/0990395/pkg/sshutil/sshutil.go#L595-L598), and the guest sshd has `ClientAliveInterval 0`. A connection that exchanges no traffic for hours is exposed to everything that reaps idle flows.
- The stale-master helpers added for #4913 (`IsControlMasterRunning`, `RemoveStaleControlMaster`) are only wired into `limactl shell` — the hostagent never consults them.
- Two sharp edges for anyone attempting a fix: `ssh -O forward` cannot create a master (it fails if none exists, which is exactly the post-sleep state), and combining `-f -N` with `-O` fails silently.
### Proposed fix
Two small host-side changes; no guest changes needed.
**1. Keepalives (prevention).** The master should notice its connection is gone instead of persisting as a zombie, and regular traffic protects it from idle reaping:
```go
// pkg/sshutil/sshutil.go, SSHOpts()
opts = append(opts,
fmt.Sprintf("User=%s", username),
"ControlMaster=auto",
controlPath,
"ControlPersist=yes",
"ServerAliveInterval=15",
"ServerAliveCountMax=4",
)
```
**2. Reconcile loop in the hostagent (heal).** Pseudo-code:
```
every 10s, while socket forwarding is active:
if IsControlMasterRunning(instDir):
continue # near-free in the healthy case
RemoveStaleControlMaster(instDir)
run a trivial remote command over the normal ssh path
# re-spawns the master via ControlMaster=auto;
# `ssh -O` alone cannot create one
for each rule in portForwards where rule.GuestSocket != "":
forwardSSH(verbCancel, rule) # best-effort, releases stale registration
forwardSSH(verbForward, rule)
```
We have built and field-tested exactly this (plus the keepalives) on a fork, based on v2.2.0: https://github.com/MartyPine/lima/tree/fix/hostagent-socket-forward-reconcile — including a unit test in the style of the existing ga.sock forward tests. Measured results: the baseline above never recovers; with the patch, detection happens within one 10 s tick and restore takes about a second (kill-to-healed of 3–9 s across repeated runs, 5 s in production use; the reconciler triggers exactly once per kill and never while healthy). Sharing the branch as evidence that the approach works, not as a finished PR — happy for it to be adapted or rewritten however the maintainers prefer.
Longer term, vz could route `guestSocket` forwards over the gRPC forwarder the way TCP ports already go — `pkg/portfwd` already has host-side unix listener support — which would remove SSH from the socket path entirely on vz. The reconciler would still matter for qemu and any fallback path, so the two are complementary.
### Related issues
- #2099 — X11 forwarding breaks after sleep. Same mechanism, different symptom; the first maintainer comment there ("We need a way to restore the SSH connections after sleep") describes this issue's subject.
- #4913 — fixed stale `ssh.sock` handling for `limactl shell` only; the forwards were left unhealed, and this issue is the remaining half.
- #442 — "VM randomly loses network" likely absorbs some reports of this fault (host-side forwards dead, VM fine).
- abiosoft/colima#1033 — collects many docker-flavoured reports of this same fault, mixed with unrelated VM freezes. Worth noting for triage there: colima currently pins `LIMA_SSH_PORT_FORWARDER=true`, so those users also lose all TCP forwards when the master dies, making the failure look much bigger than the socket forwards alone.
- abiosoft/colima#460 — "Colima seems to die after sleep", open since 2022: the same symptom pattern (works again after `colima stop && colima start`), consistent with dead forwards rather than a dead VM.
Contributor guide
Assessment
This issue has not been assessed yet.