Add agent dockerfile using the official Docker CLI image for DooD support
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Motivation
The agent is the only service component without a dockerfile in `docker/`. A previous attempt (PR [#9596](https://github.com/lablup/backend.ai/pull/9596), branch `feat/BA-2327/dockerize-agent`, Jira BA-2327) was auto-closed as stale in a draft state with unresolved review findings — notably a broken `curl`/GPG/apt dance to install `docker-ce-cli`. The agent needs the Docker CLI in-image because it shells out to `docker load` / `docker exec` (`src/ai/backend/agent/docker/utils.py:110`, `docker/kernel.py:427,518,557,620`); everything else goes through the aiodocker API socket.
## Objective
A `docker/backend.ai-agent.dockerfile` following the same wheel-install pattern as the other six service dockerfiles, with the Docker CLI obtained via multi-stage `COPY --from` from the official CLI image — no apt repository setup.
## Details
1. Start from the dead branch's dockerfile (`origin/feat/BA-2327/dockerize-agent`, commits `12406ade12` / `400bb44856`) as reference, but rebuild on the standard pattern: `FROM python:${PYTHON_VERSION`} + wheels from `dist/` + root `requirements.txt`, build args `PYTHON_VERSION` / `PKGVER`.
1. Obtain the CLI with `COPY --from=docker:-cli /usr/local/bin/docker /usr/local/bin/docker`. Do NOT use `docker:cli` as the base image — it is Alpine/musl and cannot install our manylinux wheels; the CLI binary itself is statically linked and runs fine on the Debian base. CLI↔daemon API compatibility is wide, so exact host-version match is not required; pin the tag.
1. DooD deployment requirements are conventions, not code — document them in the compose service / install docs:
- bind-mount `/var/run/docker.sock`
- path parity: scratch dirs, `/tmp/backend.ai/ipc`, and vfolder mount sources must be mounted at identical absolute paths on host and in the agent container (the host daemon resolves kernel bind-mount sources in the host filesystem)
- host network mode (agent↔kernel ZMQ/service ports)
- `pid: host` (host PID namespace for process visibility)
- **host cgroup namespace (**`cgroupns: host` / compose `cgroup: host`) — REQUIRED, not optional. The PID-translation logic (`host_pid_to_container_pid` in `src/ai/backend/agent/utils.py:321` via `get_cgroup_of_pid` / `get_container_id_of_cgroup` in `src/ai/backend/common/cgroup.py`) parses `/proc/{pid}/cgroup` expecting host-rooted paths (`docker/` or `system.slice/docker-.scope`), and `get_cgroup_mount_point` resolves the cgroupfs from `/proc/mounts`. A private cgroupns renders sibling-container paths non-host-rooted and mounts a namespaced cgroupfs, breaking both PID translation and sysfs metrics. Note: on cgroup v2 hosts Docker defaults to `private` cgroupns even with `--privileged`, so this must be set explicitly. With host cgroupns, host `/sys/fs/cgroup` visibility follows automatically.
1. Address the other findings from the dead PR: slim final image where feasible, correct RST indentation if docs are touched.
1. Backlog note (separate issue, not blocking): refactor the `docker load` / `docker exec` subprocess calls to aiodocker API equivalents so the CLI copy can be dropped entirely.
## Success Criteria
- [ ] `docker build` succeeds with the same build-arg contract as the other service dockerfiles.
- [ ] Containerized agent with socket mount + path parity + host network + `pid: host` + `cgroupns: host` spawns and terminates a kernel session end-to-end against a host daemon.
- [ ] Host-PID↔container-PID translation works from inside the agent container (exercises `get_cgroup_of_pid` against a sibling kernel container).
- [ ] No apt/curl/GPG Docker repository setup remains in the dockerfile.
- [ ] The image is picked up by the dockerfile-discovery script (BA-7267) without changes.
JIRA Issue: BA-7271
Contributor guide
Research direction
Compare the dead branch dockerfile (commits 12406ade12 and 400bb44856) with the six existing service dockerfiles, then read src/ai/backend/agent/docker/utils.py and docker/kernel.py for the Docker CLI calls. Check src/ai/backend/agent/utils.py and src/ai/backend/common/cgroup.py alongside the compose service and install docs. Done means the image builds with the stated args, discovery finds it, and the socket-mounted deployment passes the listed end-to-end and PID-translation checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100