Container vulnerability scanning of published Backend.AI images
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Related to #11379.
The `backend.ai-{client,storage-proxy,webserver,appproxy-coordinator,appproxy-worker}.dockerfile` files cannot be built by the OSV-Scanner workflow in standalone mode (they consume pre-built wheels produced by the release pipeline). The current hotfix excludes them from the `scan-containers` matrix, which closes the immediate CI failure but leaves a real gap: **the actual deployed Backend.AI container images are not vulnerability-scanned at all**.
Container scanning is meant to verify what's deployed. We should close that gap with a workflow that scans the *published* images.
## Options
### A. Pull-and-scan published images
New workflow that pulls `lablup/backend.ai-{component}:` from the registry and runs `osv-scanner scan image --archive` on each.
- **Pros:** scans exactly what's deployed; no coupling to the build pipeline; simple workflow.
- **Cons:** requires knowing where images are published (Docker Hub? GHCR?) and which tags to scan; doesn't cover PR-time builds; depends on the release pipeline having pushed.
### B. Hook into the release pipeline
Add an OSV-Scanner step to `.github/workflows/ci.yml` (deploy job, around the docker-image build) or `release.yml`. Build the image with the wheels and `PKGVER` already in scope, then scan immediately.
- **Pros:** wheel build already exists at that point; runs on every release.
- **Cons:** only scans tagged releases, not `main`; image stays in CI's docker daemon, not what users pulled from the registry.
### C. Wheel-build inside OSV-Scanner workflow
Add `pants package` + `PKGVER` derivation to `osv-scanner.yml` so the `backend.ai-*` dockerfiles can be built standalone.
- **Pros:** keeps scanning in one workflow; runs on every push.
- **Cons:** heavy — adds significant minutes per matrix entry; pants cache cold-starts on GH runners; redundant with what release CI does.
## Recommendation
**Option A** is the right architectural fit (scan what ships, decoupled from build). **Option B** is a defensible alternative if "scan exactly what we just published" is the more important property. **Option C** is a stopgap and probably not worth the runner cost.
## Scope to define before implementing
- **Inventory:** which `backend.ai-*` images are actually published, where (`lablup/backend.ai-*` on Docker Hub? GHCR?), which tags matter (`latest`, `stable`, version pins, ephemeral PR builds).
- **Trigger:** schedule-only (weekly?), on-tag, or on-publish webhook.
- **Auth:** for image pulls if registries require it.
- **SARIF categorization:** keep alerts grouped per image so they're actionable in the Security tab.
- **Failure policy:** should a vuln in a published image fail the workflow, or just report? (Probably "report only" — fixing requires a new release, not a workflow change.)
## Out of scope
- The immediate `scan-containers` matrix failure (tracked in the hotfix issue).
- Source-level dep bumps to clear `scan-scheduled` alerts (tracked in the source-pin follow-up issue).
Contributor guide
Research direction
Start by reading .github/workflows/osv-scanner.yml, then compare the image build and publish paths in .github/workflows/ci.yml and release.yml. Inventory the backend.ai-* Dockerfiles and published image names, tags, and registry location before choosing a trigger and pull strategy. Done means published images are scanned with OSV-Scanner and actionable, per-image SARIF results are reported without relying on standalone Dockerfile builds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions
- Domain
- ci-cd, devops, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100