host UDS: recvmsg fails on an SCM_RIGHTS-imported socket with SO_PASSCRED enabled
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
**Background.** We are evaluating gVisor with nvproxy to run GPU video-inference workloads for several tenants on one node, sharing the GPU through NVIDIA MPS so each tenant gets a device-memory limit. MPS clients register with a host-side control daemon over a Unix socket, and the daemon hands each client its server connection via `SCM_RIGHTS`. Tracing that handshake from inside a sandbox (runsc's own strace) showed the CUDA client dying on the first `recvmsg` of that passed socket; the socket had `SO_PASSCRED` enabled. Everything below is the minimal, GPU-free version of that failure, arrived at by eliminating socket type, `MSG_CMSG_CLOEXEC`, nested `SCM_RIGHTS` and the receiving flags one at a time (all of those work).
## Summary
A sandbox receives a connected `AF_UNIX` socket from a host process via `SCM_RIGHTS` over a mounted host Unix socket (`--host-uds=open`). If the host side enabled `SO_PASSCRED` on that socket before passing it, `recvmsg` with a control buffer on it fails with `EINVAL` inside the sandbox, whether the host attached only `SCM_CREDENTIALS` or `SCM_CREDENTIALS` plus `SCM_RIGHTS`; a plain `recv` without a control buffer works, and with `SO_PASSCRED` off everything works. Under runc all cases work. The kernel attaches `SCM_CREDENTIALS` to every message delivered on a `SO_PASSCRED` socket, and the sentry's receive path for host-imported sockets hands all host control messages to a parser that only accepts `SCM_RIGHTS`.
## Reproducer (no GPU, no Kubernetes; Linux)
`uds_passcred_repro.py`: https://gist.github.com/hansent/e0645f1e2cdbac07f0ac9728bdd51bba (pinned: https://gist.github.com/hansent/e0645f1e2cdbac07f0ac9728bdd51bba/9bf3a03488107ea3e92b787aa3ed1e4c76b99cb1); the same file is embedded below. The server accepts a `SOCK_SEQPACKET` control connection, creates one `SOCK_STREAM` socketpair per client, enables `SO_PASSCRED` on both ends (unless `--no-passcred`), passes one end via `SCM_RIGHTS`, waits for "hi" on it, and then sends exactly one message on it. Each client invocation runs one receive case on that passed socket, so stream framing cannot mix cases.
Dedicated runsc runtime in `/etc/docker/daemon.json` (Docker has no per-run runtime-flag option), then restart Docker:
```json
{ "runtimes": { "runsc-uds-repro": { "path": "/usr/local/bin/runsc", "runtimeArgs": ["--host-uds=open", "--directfs=false"] } } }
```
```bash
mkdir -p /tmp/uds-repro && python3 uds_passcred_repro.py server /tmp/uds-repro # host, leave running
for c in recv recvmsg-creds recvmsg-creds-rights; do
docker run --rm --runtime=runc -v /tmp/uds-repro:/uds -v "$PWD":/r:ro python:3.11-slim python3 /r/uds_passcred_repro.py client /uds $c
done
for c in recv recvmsg-creds recvmsg-creds-rights; do
docker run --rm --runtime=runsc-uds-repro -v /tmp/uds-repro:/uds -v "$PWD":/r:ro python:3.11-slim python3 /r/uds_passcred_repro.py client /uds $c
done
# control: stop the server, restart it with --no-passcred, repeat the runsc loop
```
(`--directfs=false` is only so that the sandbox's own `SCM_CREDENTIALS` hello is accepted by the server, see the companion credentials issue; it does not affect the receive behaviour reported here.)
Expected (runc) vs actual (runsc `release-20260817.0`, kvm platform); the same script was run under containerd with the same two runsc flags:
| case (one client invocation each) | runc | runsc |
|---|---|---|
| `recv`: `recv()` without a control buffer | OK `payload` | OK `payload` |
| `recvmsg-creds`: `recvmsg()` with a control buffer, only host-attached credentials | OK, cmsg `SCM_CREDENTIALS` | **`EINVAL`** |
| `recvmsg-creds-rights`: `recvmsg()` with a control buffer, credentials + `SCM_RIGHTS` pipe | OK, both cmsgs, pipe usable | **`EINVAL`** |
| all three with the server started `--no-passcred` | OK | OK |
Earlier runs with `SOCK_SEQPACKET` and `SOCK_DGRAM` socketpairs gave the same `EINVAL` for the two `recvmsg` cases. Inside the sandbox `getsockopt(SO_PASSCRED)` on the imported socket returns 0 (runc: 1).
`uds_passcred_repro.py`:
```python
#!/usr/bin/env python3
"""Reproducer: recvmsg fails on an SCM_RIGHTS-imported Unix socket that has SO_PASSCRED enabled (gVisor).
Linux only (SO_PASSCRED / SCM_CREDENTIALS). One receive case per client invocation, so SOCK_STREAM framing cannot
mix the cases:
server: python3 uds_passcred_repro.py server DIR [--no-passcred]
client: python3 uds_passcred_repro.py client DIR {recv,recvmsg-creds,recvmsg-creds-rights}
Per client connection the server: accepts on DIR/sock (SOCK_SEQPACKET, SO_PASSCRED on the listener), reads the
client's "hello:" (with SCM_CREDENTIALS), creates ONE SOCK_STREAM socketpair, enables SO_PASSCRED on both ends
(unless --no-passcred), passes one end via SCM_RIGHTS, waits for "hi" on it, then sends exactly ONE message on it:
recv / recvmsg-creds -> b"payload" (no ancillary data attached by the server; the kernel adds SCM_CREDENTIALS
to the receiver if the socket has SO_PASSCRED)
recvmsg-creds-rights -> b"payload" with one pipe write-end attached via SCM_RIGHTS; the server then reports
whether the client wrote through that pipe.
The client receives that one message with the selected shape and prints OK or the error.
"""
import argparse, array, os, select, socket, struct, sys
CASES = ("recv", "recvmsg-creds", "recvmsg-creds-rights")
def serve(dirpath, passcred):
path = os.path.join(dirpath, "sock")
try: os.unlink(path)
except FileNotFoundError: pass
srv = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); srv.bind(path); os.chmod(path, 0o777); srv.listen(4)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
print(f"server: listening on {path}, SO_PASSCRED on passed sockets = {passcred}", flush=True)
while True:
c, _ = srv.accept(); c.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
try:
msg, anc, _, _ = c.recvmsg(256, socket.CMSG_SPACE(64))
creds = [struct.unpack("3i", d[:12]) for lvl, typ, d in anc if typ == socket.SCM_CREDENTIALS]
case = msg.decode(errors="replace").split(":", 1)[-1]
print(f"server: client case={case!r} kernel creds (pid, uid, gid) = {creds}", flush=True)
a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
if passcred:
a.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1); b.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
c.sendmsg([b"FD"], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", [b.fileno()]))]); b.close()
a.settimeout(10); print(f"server: passed a SOCK_STREAM socket, got {a.recv(16)!r} on it", flush=True)
if case == "recvmsg-creds-rights":
r, w = os.pipe()
a.sendmsg([b"payload"], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", [w]))]); os.close(w)
rl, _, _ = select.select([r], [], [], 5)
print("server: sent payload + SCM_RIGHTS pipe; pipe read:", os.read(r, 64) if rl else b"", flush=True); os.close(r)
else:
a.send(b"payload"); print("server: sent payload (no ancillary data from the server)", flush=True)
a.close()
except Exception as e: print("server: error", type(e).__name__, e, flush=True)
finally: c.close(); print("server: --- done ---", flush=True)
def run_client(dirpath, case):
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); s.connect(os.path.join(dirpath, "sock"))
s.sendmsg([f"hello:{case}".encode()], [(socket.SOL_SOCKET, socket.SCM_CREDENTIALS, struct.pack("3i", os.getpid(), os.getuid(), os.getgid()))])
print(f"client[{case}]: uid={os.getuid()} sent SCM_CREDENTIALS ok", flush=True)
s.settimeout(10); _, anc, _, _ = s.recvmsg(16, socket.CMSG_SPACE(64))
fd = [x for lvl, typ, d in anc if typ == socket.SCM_RIGHTS for x in array.array("i", d).tolist()][0]
ps = socket.socket(fileno=fd); ps.settimeout(6)
print(f"client[{case}]: received passed socket fd={fd} type={ps.type} SO_PASSCRED as seen here={ps.getsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED)}", flush=True)
ps.send(b"hi")
try:
if case == "recv":
m, anc2 = ps.recv(64), []
elif case == "recvmsg-creds":
m, anc2, _, _ = ps.recvmsg(64, 4096, 0)
else:
m, anc2, _, _ = ps.recvmsg(64, 4096, socket.MSG_CMSG_CLOEXEC)
fds = [x for lvl, typ, d in anc2 if typ == socket.SCM_RIGHTS for x in array.array("i", d).tolist()]
print(f"client[{case}]: OK msg={m!r} cmsg_types={[typ for _, typ, _ in anc2]} passed_fds={fds}", flush=True)
for x in fds: os.write(x, b"written-through-passed-pipe"); os.close(x)
except OSError as e:
print(f"client[{case}]: FAILED {type(e).__name__}: [Errno {e.errno}] {e.strerror}", flush=True); sys.exit(1)
finally: ps.close(); s.close()
if __name__ == "__main__":
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="mode", required=True)
ps_ = sub.add_parser("server"); ps_.add_argument("dir"); ps_.add_argument("--no-passcred", action="store_true", help="do not enable SO_PASSCRED on the passed socket (control)")
pc = sub.add_parser("client"); pc.add_argument("dir"); pc.add_argument("case", choices=CASES)
args = p.parse_args()
serve(args.dir, not args.no_passcred) if args.mode == "server" else run_client(args.dir, args.case)
```
## Likely source path
- `pkg/sentry/socket/unix/transport/host.go`: an `SCM_RIGHTS`-imported socket becomes a host-backed connected endpoint; `SO_PASSCRED` is read at init, and the receive path assumes the imported fd will not have it enabled.
- A receive with a requested control buffer passes the host control messages to `pkg/unet/unet.go` `ExtractFDs`, which calls `ParseUnixRights` on every message and errors on any other type; the host-attached `SCM_CREDENTIALS` therefore surfaces as `EINVAL`.
- Same on `master` at 80bb741 (2026-09-01) after the socket refactor; the receive and control-message behaviour is unchanged.
## Impact
NVIDIA MPS: the control daemon hands each CUDA client its server connection this way, with `SO_PASSCRED` enabled (a runc client trace shows `SCM_CREDENTIALS` on every message received on it). A sandboxed CUDA client that has already authenticated (`NEW CLIENT … from user 65534`, `NEW SERVER … Ready` in the daemon log) fails on the first `recvmsg` of that passed socket and reports CUDA error 805 "MPS client failed to connect"; runsc strace: `recvmsg(0x9 host:[7], {…, control_len=4096}, MSG_CMSG_CLOEXEC) = -1 errno=22`. The MPS sockets visible in the daemon's network namespace are `SOCK_SEQPACKET`; the minimal `SOCK_SEQPACKET` case produces the same `EINVAL`.
## Environment
```
$ runsc --version
runsc version release-20260817.0
spec: 1.2.1
```
runsc `release-20260817.0` (spec 1.2.1), platform kvm, `--host-uds=open --directfs=false`; validated under containerd 2.1.4 (`io.containerd.runsc.v1` shim) on Kubernetes 1.33.4, Ubuntu 22.04, no user namespaces; the Docker commands above follow Docker's alternative-runtime configuration. Client uid does not matter for this bug (root and uid 1000 behave the same).
## Ask
When the imported socket is presented to the sandbox with `SO_PASSCRED=0`, ignore host-attached `SCM_CREDENTIALS` while continuing to extract and deliver `SCM_RIGHTS`. Alternatively, preserve the inherited `SO_PASSCRED` state and translate the credentials into sandbox-visible credentials. Either would be preferable to returning `EINVAL` for otherwise valid data. Happy to test a patch.
Contributor guide
Research direction
Start by running the provided uds_passcred_repro.py against runc and runsc, then inspect pkg/sentry/socket/unix/transport/host.go and pkg/unet/unet.go, especially ExtractFDs and ParseUnixRights. Trace how host-attached SCM_CREDENTIALS is handled when the imported socket has SO_PASSCRED disabled. Done means recvmsg no longer returns EINVAL for the reported cases while SCM_RIGHTS remains usable and the existing control cases continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux, python
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100