Azure / Azure/unbounded

releng: a new component image fails its first release because its GHCR package does not exist

Open
#701 0 comments 0 reactions 0 assignees View on GitHub
bug github_actions releng
Dominant language
Go
Stars
28
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
55

Description

## Symptom

The first release that includes a new component image fails, four minutes into
the job, with:

```
ERROR: failed to push ghcr.io/azure/token-refresher:v0.5.0: denied: permission_denied: write_package
```

[run 33655638847, attempt 1, `component-images (token-refresher)`](https://github.com/Azure/unbounded/actions/runs/33655638847/job/100333512009).
This was the failure that started the `v0.5.0` cascade: `fail-fast` cancelled
seven healthy sibling builds, and the pipeline then could not be re-run for
unrelated reasons (fixed in #700). The release was abandoned and `v0.6.0` cut
instead.

## Cause

`release.yaml` grants `packages: write` at the workflow level
(`.github/workflows/release.yaml:44-52`), which is correct. But GHCR requires
the *package* to already exist and be linked to the repository before
`GITHUB_TOKEN` can push to it. For a package that has never been pushed, the
first push from Actions is denied.

`token-refresher` was added in 896cc73 (#646) and is not in `images.yaml`,
whose only triggers are:

```yaml
on:
push:
tags:
- "images/*/*"
workflow_dispatch:
```

so nothing had ever pushed `ghcr.io/azure/token-refresher` before the release
tried to. Someone created the package out of band between attempts 1 and 3, and
the job succeeded from attempt 3 onward, which confirms the diagnosis.

## Why it matters

This recurs for **every** new component image, and it fails in the worst place:
late, during a release, after other images have already been pushed, signed and
attested. The component matrix currently has 11 entries and grows.

## Options

1. **Preflight the matrix.** A cheap job that checks every component's package
is pushable before any of them build, so a new image fails in seconds with a
message naming the package, instead of four minutes in with a raw registry
error. Does not remove the manual step, but makes it obvious and cheap.
2. **Build component images on merge to `main`.** Extend `images.yaml` (or add a
path-filtered job) so a component's package is created the first time its
Containerfile lands, long before a release needs it. Removes the failure
entirely; costs CI minutes on every merge that touches `images/`.
3. **Create the package as part of adding a component.** Document it as a step
in whatever checklist covers adding an image, and leave the pipeline alone.
Cheapest, and the one most likely to be forgotten.

Option 2 removes the class of failure; option 1 makes it survivable and is
compatible with either of the others.

## Notes

- Needs someone with package admin on the org to confirm the current settings,
specifically whether org policy permits `GITHUB_TOKEN` to create packages, or
whether every package must be created and linked by hand.
- #700 makes the release re-runnable, so a recurrence is now recoverable rather
than terminal. That reduces the severity but does not fix this.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/release.yaml:44-52 and the trigger and job definitions in images.yaml, then review run 33655638847 to reproduce the first-release failure. Confirm the organization's GHCR package-creation policy before choosing among the preflight, merge-build, or documented setup options. Done means a new component image can reach its first release without an out-of-band package step, or fails earlier with an actionable package-specific message.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions
Domain
ci-cd, devops, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.