lablup / lablup/backend.ai

Publish Backend.AI service Docker images via GitHub Actions with Docker Hub OIDC authentication

Open
#13,582 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Objective

Automatically bake and publish the Backend.AI service Docker images from `docker/*.dockerfile` to the `lablup` Docker Hub organization on every release tag, authenticating with **Docker Hub OIDC connections** (no stored PAT/password secrets in GitHub).

Reference: [Docker OIDC connections for GitHub Actions](https://www.docker.com/blog/docker-oidc-connections-for-github-actions-available-for-docker-orgs/) / [Docker docs](https://docs.docker.com/enterprise/security/oidc-connections/create-manage/)

## Scope

|Dockerfile|Image|
|---|---|
|`docker/backend.ai-manager.Dockerfile`|`lablup/backend.ai-manager`|
|`docker/backend.ai-agent.dockerfile` (new, BA-7271)|`lablup/backend.ai-agent`|
|`docker/backend.ai-appproxy-coordinator.dockerfile`|`lablup/backend.ai-appproxy-coordinator`|
|`docker/backend.ai-appproxy-worker.dockerfile`|`lablup/backend.ai-appproxy-worker`|
|`docker/backend.ai-webserver.dockerfile`|`lablup/backend.ai-webserver`|
|`docker/backend.ai-storage-proxy.dockerfile`|`lablup/backend.ai-storage-proxy`|
|`docker/backend.ai-client.dockerfile`|`lablup/backend.ai-client`|

Out of scope:

- Infra images (`krunner-extractor`, `socket-relay`, `linuxkit-nsenter`) — self-contained in backend.ai-krunner, not published here.
- Refactoring agent `docker load`/`docker exec` subprocess calls to aiodocker (backlog follow-up noted in BA-7271).
- Not related to the isla-sorna pipeline; that dispatch stays as is.

## Design decisions (confirmed)

- **Auth**: Docker Hub OIDC connection (lablup org is a Team account with the feature available). `docker/login-action@v4` (≥ 4.5.0), `username: lablup`, no password, `DOCKERHUB_OIDC_CONNECTIONID` env from a GitHub Actions repo variable, `permissions: id-token: write`.
- **Ruleset subject claim**: `repo:lablup/backend.ai:environment:deploy-to-dockerhub` — push jobs run inside a protected GitHub environment (mirrors the existing `deploy-to-pypi` pattern used for PyPI trusted publishing).
- **Trigger**: tag push (`refs/tags/*`), chained after the existing `build-wheels` job in `ci.yml`; the image build reuses the `wheels` artifact (dockerfiles install wheels from `dist/` + root `requirements.txt`, build context = repo root, build args `PYTHON_VERSION` from `pants.toml`, `PKGVER` from `VERSION`). Manual reruns via `workflow_dispatch`.
- **Multi-arch**: linux/amd64 + linux/arm64 (`scripts/build-wheels.sh` already produces aarch64 wheels).
- **Tagging**: version tag from the git tag; `latest` only for final (non-rc) releases, reusing `scripts/determine-release-type.py`.
- **Agent image (BA-7271)**: DooD via the host Docker socket; Docker CLI obtained with `COPY --from=docker:-cli` (NOT as base image — Alpine/musl cannot install our manylinux wheels). Path parity, host network, and cgroupfs mount are deployment conventions documented in compose/docs, not code.
- Follow `.github/AGENTS.md` guardrails: logic in `scripts/`, no `${{ `}} interpolation into shell, third-party actions pinned to commit SHAs.

JIRA Issue: BA-7264

Contributor guide

Open the contributing guide

Research direction

Read .github/AGENTS.md and the existing deploy-to-pypi pattern in ci.yml first. Inspect the listed docker/*.dockerfile files, the wheels artifact flow, scripts/build-wheels.sh, scripts/determine-release-type.py, pants.toml, and VERSION. Done means release tags publish all scoped amd64 and arm64 images through Docker Hub OIDC, with latest only for final releases and manual reruns supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, python
Domain
ci-cd, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.