bug: sandbox bind-mount ownership breaks after host reboot (unpinned rootless userns mapping, no launch-time reconcile)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
Summary
On the rootless Podman and Docker sandbox drivers, OpenShell attaches no user-namespace ID mapping to the container and mounts host state directories as plain (non-idmapped) rbinds. The host-side ownership of a bind-mounted state dir is therefore just the projection of a container-relative UID (e.g. sandbox) through whatever rootless subuid range the runtime happens to assign at create time. OpenShell never pins, persists, or reconciles that range. If the range shifts across a host reboot / runtime reconfig, previously-written inodes keep their old raw numbers and now resolve to unrelated host users — and OpenShell never repairs them.
Result: after a host reboot the sandbox's persisted state tree comes back owned by root / systemd-network / dnsmasq (group systemd-journal) with modes like drwx------ and -r--r--r--, and the sandbox process can no longer read/write its own state (readonly sqlite, EACCES). Reported downstream as NVIDIA/NemoClaw#6972 on a GB10 / Ubuntu 24.04 box: the agent could not recover the gateway and a rebuild's host-side backup could not read the mis-owned dirs.
Reproduction
- Launch a sandbox (rootless Podman driver) with a bind-mounted persistent state dir under
~/.../mounts/<sandbox>/. - Let the sandbox write state (owned, on the host, by the runtime's rootless subuid projection of the container
sandboxuser). - Reboot the host (or otherwise cause the rootless subuid base /
userns-remapto shift — daemon reconfig, storage//etc/subuidchange). - Re-launch / recover the sandbox. Observe on the host: the state tree is now owned by unrelated low-numbered users (
root,systemd-network,dnsmasq) with restrictive modes; inside the container the process hitsEACCES/attempt to write a readonly database.
Root cause (code)
Confirmed against current main:
- No userns / idmap on the container spec.
crates/openshell-driver-podman/src/container.rsContainerSpec(~L174–222) has nouserns/idmappingsfield and runs the supervisor asuser: "0:0"(~L908). The Docker driver sets noUsernsModeand also runs0:0(crates/openshell-driver-docker/src/lib.rs~L908). Mapping is fully delegated to the runtime default. - Bind mounts are plain rbinds, not idmapped. Podman:
crates/openshell-driver-podman/src/container.rs~L562–585 builds options["ro"|"rw","rbind"](+ optional SELinuxz/Z) — noidmap, noU. Docker:crates/openshell-driver-docker/src/lib.rs~L1794–1829 emitssource:target[:ro,z|Z]only. - No subuid base is pinned or persisted. The only subuid handling is
crates/openshell-driver-podman/src/driver.rscheck_subuid_range()(~L807, called ~L242), which is warn-only — it neither allocates, chooses, nor records a base. - Launch-time re-chown does not repair a persisted tree for these drivers.
crates/openshell-supervisor-process/src/process.rsprepare_filesystem()(~L1147): read-write paths are chowned only when newly created (~L1183–1194,prepare_read_write_pathreturns true only on create), and the recursive/sandboxre-chown (~L1201–1207 →chown_sandbox_home~L1105) runs only whenOPENSHELL_SANDBOX_UIDis set — which the rootless Podman/Docker path does not set (only K8s/VM drivers do). So an existing.hermestree with a shifted mapping is never re-owned. - No reboot/recreate reconcile exists anywhere for the bind-mounted tree.
Impact
- Permanent loss of access to sandbox state after a benign host reboot; user-visible as "impossible to recover the sandbox," data-loss on rebuild, and readonly-DB gateway failures.
- Security-adjacent: host ownership of sandbox-written files is non-deterministic and, on some runtimes, can collapse onto low host UIDs.
Proposed fix directions (for maintainer input)
- Durable / deterministic (preferred, architectural): attach an idmapped mount to the host state binds, or pass explicit deterministic
--uidmap/--gidmapto the container so the host-side ownership of the bind-mounted tree is stable and independent of the runtime's default rootless allocation across reboots. This pins the mapping OpenShell currently leaves implicit. - Mitigation (contained, self-healing at launch): extend the existing
prepare_filesystemre-chown so the sandbox state tree is reconciled to the configured sandbox identity on every launch for the Podman/Docker drivers too (not only whenOPENSHELL_SANDBOX_UIDis injected). This makes a mis-owned tree readable again under the current mapping without changing the mapping model. It heals the symptom each start but does not pin the mapping. - Make
check_subuid_range()fail-closed (or emit a clearer remediation) when the range is missing/ambiguous rather than warn-only.
Related: #1909 (rootless Podman UID-mapping issues on Fedora 44) touches adjacent territory but is a distinct scenario.
I'm happy to open a PR for the contained mitigation (#2) if that direction is acceptable, and defer the idmapped-mount design (#1) to maintainers.
Filed while triaging NVIDIA/NemoClaw#6972; root cause traced into OpenShell.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Podman and Docker container specifications in crates/openshell-driver-podman/src/container.rs and crates/openshell-driver-docker/src/lib.rs, then inspect check_subuid_range() in crates/openshell-driver-podman/src/driver.rs. Read prepare_filesystem() in crates/openshell-supervisor-process/src/process.rs to compare creation-time ownership handling with the proposed launch-time reconciliation. Done requires maintainer agreement on either deterministic mapping or self-healing re-chown, plus a reproducible regression test for recovery after the mapping shifts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, rust
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100