[Bug]: vminitd: parallel clients of the forwarded ssh-agent hang on live connections; the container's terminal can freeze with them
- Dominant language
- Swift
- Stars
- 49.9k
- Forks
- 1.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 22
Description
### I have done the following
- [x] I have searched the existing issues
- [x] If possible, I've reproduced the issue using the 'main' branch of this project
### Steps to reproduce
the report below was generated by using `claude`
## Summary
When several guest processes talk to `/var/host-services/ssh-auth.sock` at the same time, some of them never get an answer.
The connections are live: the host agent holds them open, no bytes are queued on any host socket, and killing the host agent releases every hung client at once with "communication with agent failed".
Only the data event is lost, on the guest side of the relay.
In two catches on a real machine the container's interactive terminal and every `container exec` froze a few seconds after the clients started to hang, and came back the moment the host agent was killed.
## Steps to reproduce
```sh
container run --detach --name t --ssh debian:12 sleep infinity # any image with ssh-keygen
container exec t sh -c 'apt-get update && apt-get install -y openssh-client'
container exec t sh -c 'ssh-keygen -q -t ed25519 -N "" -f /tmp/k && for i in $(seq 16); do echo hi > /tmp/m.$i; done'
```
Then rounds of sixteen parallel signatures with the file key.
`ssh-keygen -Y sign` first asks the agent whether it holds the key, then reads the file, so each sign is one short agent conversation:
```sh
for r in $(seq 30); do
echo "round $r"
timeout 10 container exec t sh -c 'for i in $(seq 16); do ssh-keygen -Y sign -q -f /tmp/k -n file /tmp/m.$i & done; wait' || break
done
container exec t ps -o pid,etime,stat,args | grep ssh-keygen
```
Result on our machine: a round below 30 (run 4), round 8 (run 5, agent holding one key) and round 1 (run 6, agent holding no key) never returned.
Inside, 14 of the 16 `ssh-keygen` sit in `S`, blocked in `read` on the agent socket, for as long as we let them (20 minutes).
Meanwhile `container exec t echo alive` still answers, so the relay as a whole is not stuck.
On the host, while they hang:
- `lsof -p $(pgrep -x ssh-agent) | grep -c unix` → 16: the agent holds one connection per hung client.
- `netstat -f unix | awk '$3>0 || $4>0'` → no unix socket with anything queued: the requests were consumed and no answer is waiting anywhere on the host.
- `killall ssh-agent` → every hung `ssh-keygen` exits at once with `error fetching identities: communication with agent failed`.
## Negative controls
- One client against a silent agent (a `nc -l -U` socket that accepts and never answers): the request crosses (5 bytes reach `nc`), `exec` keeps answering, the client fails when `nc` dies. No hang.
- Eight clients at once against the same silent agent: `exec` keeps answering. But in every run one of the eight outlives `nc` by 40 s and more without ever seeing EOF (`ssh-add -l` in `S` inside) — a smaller instance of the same loss.
- Thirty rounds of eight parallel `ssh-add -l` against the real agent, in alpine, with and without a key in the agent: clean.
Sixteen `ssh-keygen -Y sign` per round in the Debian image is what hangs.
- Serialising the clients (one at a time) over thirty full test-suite runs on the original workload: clean.
## The terminal half, from two catches on a real machine
Same container kind, a test suite running `ssh-keygen -Y sign` from several threads:
- 07.09: 13 `ssh-keygen` blocked in `read` on the agent socket, the oldest three started within 60 ms of each other; three seconds after that the last byte reached the terminal master (`/dev/pts/0` mtime), and zellij stayed blocked in `write` on it — a non-blocking write to the pty returned `EAGAIN`, the master side full and unread. `container exec` did not return either. `ssh-add -l` on the mac answered in 5 ms throughout. `killall ssh-agent` on the mac: terminal back at once, the 13 signers gone, no restart.
- 30.08: zellij in `write` for 159 consecutive 2-second samples; on the mac `wezterm-gui` in `PtyFd::read`, all 21 threads of `container-runtime-linux` in `kevent`.
The scripted repro has no tty attached and its terminal half never triggered; we could not make the second stage deterministic.
## Where we looked in the sources
- `vminitd/Sources/VminitdCore/ProcessSupervisor.swift:52-64`: one thread runs `epoll_wait` and calls every relay handler inline; handlers are looked up by descriptor number at dispatch time.
- `Sources/ContainerizationOS/Linux/Epoll.swift:121-124`: descriptors are set `O_NONBLOCK` and registered edge-triggered.
- `VsockProxy.swift:257` and `:309`: the client and server descriptors of each proxied connection register on that thread with `[.input, .output]`; `transferData` (`:373`) splices until `EAGAIN` via `OSFile+Splice.swift:60`.
- `IOPair.swift:146`: the terminal relay's write to the other side is a plain blocking `write` on the same thread.
We could not identify by reading which edge is missed; the empirical shape — live connection, EOF still delivered, data not — is what we can offer.
### Problem description
m
### Environment
```markdown
- macOS 26.6.2, container 1.3.1 (containerization 0.42.0), guest kernel 6.18.35
- container created with `--ssh`; the host agent is Apple's `ssh-agent` (launchd) — reproduced with it holding one key and with it empty
- guest image: Debian 12 with OpenSSH 9.2p1 (reproduced there); alpine with `openssh-client` was used for the negative controls below
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Reproduce the parallel ssh-agent workload with the provided Debian container commands, then inspect ProcessSupervisor.swift:52-64, Epoll.swift:121-124, VsockProxy.swift:257, :309 and :373, OSFile+Splice.swift:60, and IOPair.swift:146. Trace edge-triggered events and relay behavior for live connections where requests are consumed but responses are lost. Done means the concurrent signing workload completes without hung clients and the terminal relay remains responsive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, macos, swift
- Domain
- cli, networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100