lablup / lablup/backend.ai

Support rootless container runtimes

Open
#13,845 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Backend.AI currently works against a rootless container runtime in exactly one narrow configuration, and that configuration gives up isolation inside the container. This epic collects the work needed to make rootless a first-class deployment shape.

### Background

Under a rootless user namespace the host and container id spaces are different. On a host with the usual subuid allocation the map looks like this:

```
host 1000 -> container 0
host 100000..165535 -> container 1..65536
anything else -> nobody (65534)
```

Several components write an id in one space and have it consumed in the other. The agent chowns scratch paths using host ids, the kernel receives the same numbers as container ids, and the storage proxy creates vfolders in host id space with no knowledge of the namespace at all. On a rootful socket the map is the identity, so none of this is visible.

### Current state (verified in BA-7388 and BA-7389)

- An unprivileged agent with kernel-uid=0 is the only working combination. It works because the chown is skipped entirely (it is guarded by os.geteuid() == 0), so scratch keeps the agent's own uid, which maps to container 0, and the kernel runs as container 0. vfolders need no extra setup because the kernel, the storage proxy and the vfolder owner all resolve to the same host uid.
- The cost is that user code runs as root inside the container. The kernel entrypoint itself warns about this. Host-side isolation is preserved, but in-container isolation is not - which undercuts the usual reason for choosing rootless in the first place.
- Any non-zero kernel-uid fails. With an unprivileged agent the kernel cannot write its own /home/work. With a root agent, container creation fails outright with an OCI permission error, or - at kernel-uid=1000 - the session reports RUNNING while the kernel cannot write /home/work, cannot write mounted vfolders, and cannot read its own ssh host key.
- Scratch cleanup can fail permanently. An unprivileged agent cannot remove files the kernel wrote under a subuid, and the CLEAN handler dies with an unhandled PermissionError, leaving the scratch directory on disk.

### Scope

- Translate kernel-uid/kernel-gid, and the per-user container_uid/container_main_gid overrides, through the namespace map instead of writing them to the host verbatim.
- Align vfolder ownership and permissions across the host and container id spaces so a kernel running as a non-zero uid can actually use its vfolders. This spans the agent and the storage proxy.
- Make scratch cleanup survive files owned by mapped ids, rather than aborting the CLEAN handler and stranding the directory.
- Decide and document the supported deployment shape: whether the agent runs privileged or unprivileged against a rootless socket, and what kernel-uid values are valid for a given subuid allocation.

### Out of scope

Docker-API dialect differences that are unrelated to id mapping stay in BA-7382.

JIRA Issue: BA-7408

Contributor guide

Open the contributing guide

Research direction

Start by reviewing BA-7388 and BA-7389, then trace the agent's id handling, the storage proxy's vfolder ownership, the kernel entrypoint, and the CLEAN handler under a rootless socket. Done means mapped kernel and vfolder ids work for supported subuid allocations, non-root kernels retain access, cleanup does not strand scratch directories, and the supported deployment shape is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.