lablup / lablup/backend.ai

Container vulnerability scanning of published Backend.AI images

Open
#11,388 1 comment 0 reactions 0 assignees View on GitHub
area:infrastructure area:security ci-cd
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Related to #11379.

The `backend.ai-{client,storage-proxy,webserver,appproxy-coordinator,appproxy-worker}.dockerfile` files cannot be built by the OSV-Scanner workflow in standalone mode (they consume pre-built wheels produced by the release pipeline). The current hotfix excludes them from the `scan-containers` matrix, which closes the immediate CI failure but leaves a real gap: **the actual deployed Backend.AI container images are not vulnerability-scanned at all**.

Container scanning is meant to verify what's deployed. We should close that gap with a workflow that scans the *published* images.

## Options

### A. Pull-and-scan published images

New workflow that pulls `lablup/backend.ai-{component}:` from the registry and runs `osv-scanner scan image --archive` on each.
- **Pros:** scans exactly what's deployed; no coupling to the build pipeline; simple workflow.
- **Cons:** requires knowing where images are published (Docker Hub? GHCR?) and which tags to scan; doesn't cover PR-time builds; depends on the release pipeline having pushed.

### B. Hook into the release pipeline

Add an OSV-Scanner step to `.github/workflows/ci.yml` (deploy job, around the docker-image build) or `release.yml`. Build the image with the wheels and `PKGVER` already in scope, then scan immediately.
- **Pros:** wheel build already exists at that point; runs on every release.
- **Cons:** only scans tagged releases, not `main`; image stays in CI's docker daemon, not what users pulled from the registry.

### C. Wheel-build inside OSV-Scanner workflow

Add `pants package` + `PKGVER` derivation to `osv-scanner.yml` so the `backend.ai-*` dockerfiles can be built standalone.
- **Pros:** keeps scanning in one workflow; runs on every push.
- **Cons:** heavy — adds significant minutes per matrix entry; pants cache cold-starts on GH runners; redundant with what release CI does.

## Recommendation

**Option A** is the right architectural fit (scan what ships, decoupled from build). **Option B** is a defensible alternative if "scan exactly what we just published" is the more important property. **Option C** is a stopgap and probably not worth the runner cost.

## Scope to define before implementing

- **Inventory:** which `backend.ai-*` images are actually published, where (`lablup/backend.ai-*` on Docker Hub? GHCR?), which tags matter (`latest`, `stable`, version pins, ephemeral PR builds).
- **Trigger:** schedule-only (weekly?), on-tag, or on-publish webhook.
- **Auth:** for image pulls if registries require it.
- **SARIF categorization:** keep alerts grouped per image so they're actionable in the Security tab.
- **Failure policy:** should a vuln in a published image fail the workflow, or just report? (Probably "report only" — fixing requires a new release, not a workflow change.)

## Out of scope

- The immediate `scan-containers` matrix failure (tracked in the hotfix issue).
- Source-level dep bumps to clear `scan-scheduled` alerts (tracked in the source-pin follow-up issue).

Contributor guide

Open the contributing guide

Research direction

Start by reading .github/workflows/osv-scanner.yml, then compare the image build and publish paths in .github/workflows/ci.yml and release.yml. Inventory the backend.ai-* Dockerfiles and published image names, tags, and registry location before choosing a trigger and pull strategy. Done means published images are scanned with OSV-Scanner and actionable, per-image SARIF results are reported without relying on standalone Dockerfile builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions
Domain
ci-cd, devops, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.