host UDS: non-root SCM_CREDENTIALS sends fail with EPERM under DirectFS, blocking NVIDIA MPS
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
**Background.** Same MPS use case as #14594. The MPS daemon authenticates each client with `SCM_CREDENTIALS`. A root sandbox client is rejected by design, so we tried the obvious alternative, one fixed non-root uid for every sandboxed workload, and that failed too; it took a minimal reproducer and a look at the sentry's host capability set to see why. With `--directfs=false` credentials do pass because the sentry claims its own uid, which is how we got far enough to find #14594, but every sandbox then presents uid 65534 and file I/O loses directfs, so it is a diagnostic, not a configuration we want to run.
## Summary
With `--host-uds=open` and the default DirectFS configuration, a sandbox application running as a fixed non-root uid connects to a mounted host Unix socket but cannot send `SCM_CREDENTIALS`: the sentry substitutes credentials built from the connected socket fd's owner (the application's uid/gid, set by `ConnectWithCreds`) and the host `sendmsg` returns `EPERM`, because the DirectFS sentry keeps only `CAP_CHOWN`, `CAP_DAC_OVERRIDE`, `CAP_DAC_READ_SEARCH`, `CAP_FOWNER` and `CAP_FSETID` and lacks the `CAP_SETUID`/`CAP_SETGID` the kernel requires to present another uid/gid. Root applications fail separately and explicitly (`fdWriteVec` returns `EINVAL` for uid or gid 0). With DirectFS and no pod user namespace, these two behaviours leave no tested root or fixed non-root application identity that can authenticate to the host service. The production impact is NVIDIA MPS, whose control daemon authenticates CUDA clients this way.
## Minimal reproducer (no GPU)
Server: a `SOCK_SEQPACKET` listener with `SO_PASSCRED` on a host directory bind-mounted into the client; it prints the kernel-supplied `(pid, uid, gid)` of each client message. Client: connects and sends one message with `SCM_CREDENTIALS` of its own pid/uid/gid. This is the first step of the `uds_passcred_repro.py` client from #14594 (https://gist.github.com/hansent/e0645f1e2cdbac07f0ac9728bdd51bba, `client recv`): the client's credentialed hello is the send that fails here, before any descriptor is received.
| client | runtime | send `SCM_CREDENTIALS` | server-side `(pid, uid, gid)` |
|---|---|---|---|
| uid 1000 (pod `runAsUser`) | runc | ok | (, 1000, 1000) |
| root | runc | ok | (, 0, 0) |
| uid 1000 | runsc `--host-uds=open`, DirectFS on | **`EPERM`** | (nothing received) |
| uid 1000 + container caps `SETUID,SETGID` (also tried `+SYS_ADMIN`) | runsc, same | `EPERM` | |
| root container that calls `setuid(1000)` before connecting | runsc, same | `EPERM` | |
| root | runsc, same | `EINVAL` (explicit) | |
| root or uid 1000 | runsc `--host-uds=open --directfs=false` | ok | (, 65534, 65534) for both |
In the Kubernetes runs the observed pid field was 0 because server and client were in separate PID namespaces; the uid/gid fields are the values relevant to this report.
Host-side capability sets while a uid-1000 sandbox runs (`/proc//status`): `runsc-sandbox` uid 0, `CapEff 0x1f` (CHOWN, DAC_OVERRIDE, DAC_READ_SEARCH, FOWNER, FSETID); `runsc-gofer` uid 0, `CapEff 0x400df` (adds SETGID, SETUID, SYS_CHROOT). Adding capabilities to the container does not change the sentry's host capability set.
## Where this comes from
- `runsc/fsgofer/lisafs.go` `ConnectWithCreds` (#11340): the host socket is created and connected under the application's uid/gid, so the connected socket fd is owned by that identity.
- `pkg/sentry/socket/unix/transport/host_unsafe.go` `fdWriteVec`: when the application attaches `SCM_CREDENTIALS`, the sentry replaces them with its own pid and the uid/gid from `fstat` of that fd, returns `EINVAL` if either is 0, otherwise calls the host `sendmsg`.
- `runsc/cmd/sentry/sentrycmd/boot.go`: the DirectFS sentry's capability set (the five above), no `CAP_SETUID`/`CAP_SETGID`, so the host kernel's `scm_check_creds` rejects a non-root uid/gid that is not the sentry's own → `EPERM`.
- `--directfs=false` is a diagnostic workaround, not a solution: the sentry then runs as uid 65534 and can claim its own identity, which proves the protocol and the host service accept non-root credentials, but it changes the filesystem path and its performance, collapses every sandbox's identity to 65534 at the host service, and the MPS client then hits a separate bug on the socket the daemon passes back (issue #14594, cross-referenced below).
## Production impact: NVIDIA MPS
Kubernetes 1.33 with the NVIDIA GPU Operator's device plugin in `mps` mode: the MPS control daemon (`nvidia-cuda-mps-control -d`, root, ordinary mode) runs on the host, and each container gets `CUDA_MPS_PIPE_DIRECTORY` plus bind mounts of the pipe directory and its shm directory. Plain runc containers work. Under runsc:
| runsc config | app uid | daemon log | client |
|---|---|---|---|
| `host-uds=open`, DirectFS on | root | `User did not send valid credentials` only | hangs in CUDA init (`EINVAL` path) |
| `host-uds=open`, DirectFS on | uid 1000 (also root→`setuid(1000)`) | `User did not send valid credentials` only | hangs in CUDA init (`EPERM` path) |
| `host-uds=open`, `directfs=false` | root | `NEW CLIENT … from user 65534` → `Starting new server … for user 65534` → `NEW SERVER … Ready` | fails on the passed server socket: issue #14594 |
Same daemon with the same worker under runc, all pods sharing one uid (root or 65532): works.
Two further MPS prerequisites are outside this issue: (a) `recvmsg` on the `SO_PASSCRED` socket the daemon passes to the client returns `EINVAL` (issue #14594, the companion report with the public reproducer); (b) MPS shares memory through the host's `/dev/shm`, and the shim's `configureShm` intentionally rewrites any `/dev/shm` bind mount into a sandbox tmpfs; a bind of the host directory over `/dev/shm` from inside the sandbox works as an experiment, and the NVIDIA proxy proposal already lists host-backed `/dev/shm` (`statfs::f_type` pass-through) as an unimplemented MPS prerequisite. Would you want a separate tracking issue for that as an enhancement?
Why MPS rather than time-slicing: GPU sharing between sandboxes already works under nvproxy with the device plugin's time-slicing (several runsc sandboxes on one L40S at throughput within a few percent of runc). MPS adds per-client resource control that time-slicing lacks: an enforced per-client device memory limit (a time-sliced neighbour took 43.5 GB of 46 GB unopposed; under MPS each client saw its 11.5 GB limit), per-client active-thread limits, and concurrent kernels. We are after that partitioning, not security isolation; NVIDIA documents that MPS is not an isolation boundary.
## Environment
```
$ runsc --version
runsc version release-20260817.0
spec: 1.2.1
```
runsc `release-20260817.0` (spec 1.2.1), platform kvm, nvproxy enabled for the MPS runs (not needed for the reproducer), `--host-uds=open`, DirectFS default unless stated; containerd 2.1.4 shim; Kubernetes 1.33.4 on Ubuntu 22.04, no pod user namespaces (`hostUsers` default), pod `securityContext` as stated per row (no added capabilities unless stated); GPU Operator v25.3.1 / k8s-device-plugin v0.17.2 (`mps`, 4 replicas), driver 570.133.20, one L40S; MPS pipe directory `control` socket `srw-rw-rw- root root`. Checked against `master` at 80bb741 (2026-09-01): the credential construction and the DirectFS capability set are unchanged.
## Ask
Is non-root `SCM_CREDENTIALS` forwarding on host Unix sockets intended to work with DirectFS? `ConnectWithCreds` creates the host socket under the application's non-root uid/gid, and `fdWriteVec` uses that ownership when constructing credentials, but the DirectFS sentry lacks `CAP_SETUID` and `CAP_SETGID`, so the host `sendmsg` returns `EPERM`.
We would like a supported configuration or implementation for this non-root path. If root sandbox clients are also intended to work with explicitly exposed host sockets, guidance on a safe opt-in identity mapping would be useful, but making the fixed non-root path work would satisfy our MPS use case. We are happy to test a patch on a staging GPU node.
Contributor guide
Research direction
Read runsc/fsgofer/lisafs.go: ConnectWithCreds, pkg/sentry/socket/unix/transport/host_unsafe.go: fdWriteVec, and runsc/cmd/sentry/sentrycmd/boot.go for DirectFS capabilities. Run the minimal uds_passcred_repro.py scenario with host-uds=open and DirectFS enabled, comparing root and non-root clients. Done means a supported non-root SCM_CREDENTIALS send succeeds under DirectFS without relying on directfs=false; issue #14594 remains a separate prerequisite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- networking, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100