releng: a new component image fails its first release because its GHCR package does not exist
- Dominant language
- Go
- Stars
- 28
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 55
Description
## Symptom
The first release that includes a new component image fails, four minutes into
the job, with:
```
ERROR: failed to push ghcr.io/azure/token-refresher:v0.5.0: denied: permission_denied: write_package
```
[run 33655638847, attempt 1, `component-images (token-refresher)`](https://github.com/Azure/unbounded/actions/runs/33655638847/job/100333512009).
This was the failure that started the `v0.5.0` cascade: `fail-fast` cancelled
seven healthy sibling builds, and the pipeline then could not be re-run for
unrelated reasons (fixed in #700). The release was abandoned and `v0.6.0` cut
instead.
## Cause
`release.yaml` grants `packages: write` at the workflow level
(`.github/workflows/release.yaml:44-52`), which is correct. But GHCR requires
the *package* to already exist and be linked to the repository before
`GITHUB_TOKEN` can push to it. For a package that has never been pushed, the
first push from Actions is denied.
`token-refresher` was added in 896cc73 (#646) and is not in `images.yaml`,
whose only triggers are:
```yaml
on:
push:
tags:
- "images/*/*"
workflow_dispatch:
```
so nothing had ever pushed `ghcr.io/azure/token-refresher` before the release
tried to. Someone created the package out of band between attempts 1 and 3, and
the job succeeded from attempt 3 onward, which confirms the diagnosis.
## Why it matters
This recurs for **every** new component image, and it fails in the worst place:
late, during a release, after other images have already been pushed, signed and
attested. The component matrix currently has 11 entries and grows.
## Options
1. **Preflight the matrix.** A cheap job that checks every component's package
is pushable before any of them build, so a new image fails in seconds with a
message naming the package, instead of four minutes in with a raw registry
error. Does not remove the manual step, but makes it obvious and cheap.
2. **Build component images on merge to `main`.** Extend `images.yaml` (or add a
path-filtered job) so a component's package is created the first time its
Containerfile lands, long before a release needs it. Removes the failure
entirely; costs CI minutes on every merge that touches `images/`.
3. **Create the package as part of adding a component.** Document it as a step
in whatever checklist covers adding an image, and leave the pipeline alone.
Cheapest, and the one most likely to be forgotten.
Option 2 removes the class of failure; option 1 makes it survivable and is
compatible with either of the others.
## Notes
- Needs someone with package admin on the org to confirm the current settings,
specifically whether org policy permits `GITHUB_TOKEN` to create packages, or
whether every package must be created and linked by hand.
- #700 makes the release re-runnable, so a recurrence is now recoverable rather
than terminal. That reduces the severity but does not fix this.
Contributor guide
Research direction
Start with .github/workflows/release.yaml:44-52 and the trigger and job definitions in images.yaml, then review run 33655638847 to reproduce the first-release failure. Confirm the organization's GHCR package-creation policy before choosing among the preflight, merge-build, or documented setup options. Done means a new component image can reach its first release without an out-of-band package step, or fails earlier with an actionable package-specific message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions
- Domain
- ci-cd, devops, release
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100