lablup / lablup/backend.ai

Translate kernel-uid/kernel-gid through the rootless user-namespace mapping

Open
#13,844 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

On a rootless Podman socket the agent chowns scratch paths using kernel-uid as a host uid, while the same value is handed to the container as LOCAL_USER_ID and consumed as a container uid. Under a user namespace those are different coordinate systems, so the kernel ends up unable to access the files the agent prepared for it. The agent needs to translate the id through the container's uid/gid map.

### Why the two values diverge

A rootless namespace owned by an unprivileged user maps only the owner's uid and their subuid range:

```
host 1000 -> container 0
host 100000..165535 -> container 1..65536
anything else -> nobody (65534)
```

So a container uid N lives at host uid 100000+N-1, but the agent writes N itself onto the host. On a rootful socket the map is the identity, which is why this never surfaced there.

### Only reachable with a root agent

_chown_paths_if_root() in stage/kernel_lifecycle/docker/scratch.py is guarded by os.geteuid() == 0. An unprivileged agent skips the chown entirely, so the files keep the agent's own uid (1000), which maps to container 0. That is why an unprivileged agent with kernel-uid=0 is currently the only working combination on a rootless socket.

### Observed behaviour (verified in BA-7389)

- kernel-uid = 0, -1 or 2000: host ownership falls outside the map and shows as nobody(65534) inside, so crun cannot create the vfolder mount point. Container creation fails with "crun: mkdir /home/work/: Permission denied: OCI permission denied".
- kernel-uid = 1000: the container starts and the session reports RUNNING, but the kernel (container uid 1000) cannot write /home/work, cannot write mounted vfolders, and cannot read its own dropbear host key. dropbear and ipykernel never start, so the session looks healthy while being unusable.
- Note that -1 resolves to the agent process uid, not the image default, via UserID.check_and_return() -> os.getuid().

### Applies to the per-user override too

users.container_uid / container_main_gid override the agent-level kernel-uid and feed both LOCAL_USER_ID and the chown. Verified on a rootful socket: with the agent set to kernel-uid=2000 and the user set to container_uid=4000, the kernel ran as 4000 and the scratch work/ tree was owned 4000:4000. The same translation therefore has to cover values arriving from the per-user override, not just the agent config.

### Proposed work

- Detect whether the target socket is rootless. Podman exposes this as Host.Security.Rootless in its info endpoint, and the concrete map is readable per container. See the comment on this issue for the measured field values and the recommended call.
- When rootless, resolve the host id corresponding to the intended container id from that map and chown to it, instead of chowning to kernel-uid directly.
- Limit translation to paths the kernel must own: the work/ tree and the ssh private keys. Paths that are mounted read-only and already world-readable (the config/ tree, seccomp.json) need no ownership change.
- Decide and document the behaviour when the requested id cannot be represented in the map. With the observed allocation (subuid 100000:65536) only container ids 0 and 1..65536 are expressible, so a larger container_uid has no host counterpart. This should surface as a clear error rather than an opaque crun failure at container create time.

### Acceptance

With the agent running as root against a rootless socket and a non-zero kernel-uid, whether it comes from the agent config or from the per-user override: the kernel process runs as that container uid, the scratch paths it must own are chowned to the corresponding host id so the kernel sees them as its own, it can write /home/work, it can read its own config and ssh keys, the session reaches RUNNING with dropbear and ipykernel started, and the scratch directory is fully removed on termination with no CLEAN event exception. A requested id that cannot be represented in the map fails with a clear error.

### Out of scope

Write access to mounted vfolders is not covered here and will not be fixed by this change alone. vfolders are created by the storage proxy in host id space, while the kernel's uid and gids live in container id space. On a rootful socket the two coincide, which is why aligning a vfolder group with users.container_gids works there. On a rootless socket a kernel supplementary gid of 3000 is host gid 102999, so the same alignment does not carry over, and the storage proxy has no knowledge of the container's namespace. That needs its own handling at vfolder creation or at mount time and should be tracked as a separate issue under this epic.

JIRA Issue: BA-7407

Contributor guide

Open the contributing guide

Research direction

Start in stage/kernel_lifecycle/docker/scratch.py, tracing _chown_paths_if_root(), UserID.check_and_return(), and the agent and per-user uid overrides. Read Podman's Host.Security.Rootless info and the container uid/gid map before deciding how representability errors should surface. Done means rootless non-zero container ids work for the required scratch and key paths, cleanup succeeds, and unrepresentable ids fail clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.