godaddy / godaddy/sshenc

Foreground sshenc-agent can remain running but become unreachable after a failed socket takeover

Open
#255 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
25
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Description

On Unix, a supervised `sshenc-agent --foreground` process can remain running but become unreachable when a replacement `sshenc-agent` process takes over its live socket and does not bind a replacement listener.

On Unix, [`--foreground` skips the daemonization path](https://github.com/GoDaddy/sshenc/blob/18c1584a1f2e6aec2ba4b6f1106397b7c5071f61/crates/sshenc-agent/src/main.rs#L432-L443) that writes `~/.sshenc/agent.pid` ([source](https://github.com/GoDaddy/sshenc/blob/18c1584a1f2e6aec2ba4b6f1106397b7c5071f61/crates/sshenc-agent/src/main.rs#L545-L562)). No PID file was present in this setup. When a replacement process found the original process's live socket, [`prepare_socket_path`](https://github.com/GoDaddy/sshenc/blob/18c1584a1f2e6aec2ba4b6f1106397b7c5071f61/crates/sshenc-agent/src/server.rs#L309-L380) followed the no-PID branch and unlinked the socket. This happened before the replacement process initialized its backend, attempted to warm the identity cache, and bound a new listener ([`run_agent`](https://github.com/GoDaddy/sshenc/blob/18c1584a1f2e6aec2ba4b6f1106397b7c5071f61/crates/sshenc-agent/src/server.rs#L162-L188)). If the replacement process exits or is terminated before a replacement listener is available, the original process keeps running, but the socket pathname is gone and new clients cannot reach it. In this setup, systemd still saw the original process as running, so `Restart=on-failure` did not restart it.

I encountered this with an `sshenc-agent` process managed by a systemd user service:

```ini
[Service]
ExecStart=/path/to/sshenc-agent --foreground
Restart=on-failure
```

This systemd unit is custom, but sshenc's generated macOS LaunchAgent also uses [supervised `--foreground` operation](https://github.com/GoDaddy/sshenc/blob/18c1584a1f2e6aec2ba4b6f1106397b7c5071f61/crates/sshenc-cli/src/launchagent.rs#L44-L76).

The incident occurred during a sandboxed client invocation. The sshenc client treated the socket as not ready and auto-started a replacement `sshenc-agent` process. The replacement process detected the original process's socket as live, logged the following warning, and removed it:

```text
WARN sshenc_agent::server: live agent socket found but no PID on record; taking over (old agent will lose socket but keep running) socket=/home//.sshenc/agent.sock
```

No replacement socket appeared afterward. The supervised process remained active, but `~/.sshenc/agent.sock` no longer existed. Restarting the systemd service restored the socket and signing functionality.

The readiness check reduces the connection result to a Boolean, and the auto-started process has its standard error redirected to null ([`ensure_daemon_ready_unix`](https://github.com/GoDaddy/sshenc/blob/18c1584a1f2e6aec2ba4b6f1106397b7c5071f61/crates/sshenc-agent-proto/src/client.rs#L132-L172)). I therefore could not determine either the exact error from the first connection attempt or why the replacement process did not reach listener binding. The available diagnostics also do not establish whether the sandbox caused the initial readiness failure.

## Steps to Reproduce

I do not yet have a standalone reproducer. The incident occurred during a sandboxed client invocation while `sshenc-agent --foreground` was running under a systemd user service with a live socket and no PID file.

The warning, missing socket, surviving original process, and recovery after restarting the service are described above. The linked source shows the socket-removal ordering that can produce this state. The same ordering is present in v0.6.101.

## Expected Behavior

A failed replacement should leave either the original `sshenc-agent` process reachable or the supervisor able to detect and recover from the failure. This should preserve automatic startup and the stale or cross-binary socket recovery introduced in #213; it does not require preserving every live socket without a PID file.

## Actual Behavior

The live socket is unlinked before the replacement `sshenc-agent` process has initialized and bound its listener. If the replacement process fails before binding the listener, the original process remains alive but unreachable. In this setup, that required a manual service restart.

## Environment

- sshenc: 0.6.101
- OS: NixOS 26.11 under WSL2
- Kernel: Linux 6.18.35.2-microsoft-standard-WSL2, x86_64
- OpenSSH: 10.4p1
- Process management: custom systemd user service running `sshenc-agent --foreground`
- The same socket-removal ordering is present on current `main` at `18c1584a1f2e6aec2ba4b6f1106397b7c5071f61`

Contributor guide

Open the contributing guide

Research direction

Start by tracing prepare_socket_path and run_agent in crates/sshenc-agent/src/server.rs, then inspect ensure_daemon_ready_unix in crates/sshenc-agent-proto/src/client.rs. Reproduce or model a failed takeover before listener binding and verify that the original agent remains reachable or the supervisor can recover without leaving the socket path unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.