registrystack / registrystack/registry-stack
Encode the lean beta release path in automation
- Lingua principale
- Rust
- Stelle
- 2
- Fork
- 0
- Merge medio
- 2h 55m
- PR unite (30g)
- 130
Descrizione
## Context
#427 established the operating rule for beta releases: preserve the supply-chain
controls on every release, but make duplicate rebuilds and exhaustive manual proof
exception-based. It delivered planning, telemetry, immutable-publication checks, and a
leaner manual procedure.
State verified against `main` on 2026-07-25:
- `release.yml` is tag-triggered. The `images` job (`.github/workflows/release.yml:368`)
builds and **pushes** both images to GHCR in one step (`:437`), scans them (`:476`),
and only then enforces the advisory policy (`:502`). The vulnerability gate is
therefore post-publication for images. A blocking finding today leaves a permanently
published, publicly pullable image and no GitHub Release, and under the fix-forward
rule that image stays published. Closing this hole is the primary reason for this
work.
- Every product binary and image is built after the tag exists. There is no candidate,
comparison, or promotion machinery in the workflow.
- Repeatability proof is coordinated by an operator outside the tag workflow and written
up by hand in `release/REPEATABLE-BUILDS.md`.
- Reviewed Grype dispositions are bound to `reviewed_rootfs_digest`, a SHA-256 over the
ordered layer-digest list (`check_advisory_baselines.py:211`), compared at `:441` and
fail-closed at `:507`. Any application-layer change invalidates every review. The three
current libc6 dispositions in both baselines are bound to the beta-17 rootfs and expire
2026-08-25, so the next release carrying any code change fails this gate *after* the
images are already public.
Design constraint discovered while scoping, which shapes workstreams 2 and 3:
`release/scripts/build-release-image.sh:50-65` has two mutually exclusive output modes.
The OCI-layout export sets `--provenance=false` because timestamped BuildKit attestations
make retained comparison layouts vary; registry-pushed release images keep BuildKit
provenance (#440). `compare_layouts` requires exactly one manifest in `index.json`, so it
structurally cannot ingest a provenance-carrying layout. **The compared bytes are
deliberately not the published bytes.** A promoted image must therefore be the
registry-pushed artifact, and the equality claim must be scoped to rootfs layers and
image config rather than the full manifest.
Related work:
- #427 defines the lean operating boundary and is complete.
- #127 remains the tracker for the external-reviewer repeatable-build procedure and
public evidence.
- #242 tracks the public provenance and verification contract.
## Goal
Make the standard beta path publish an exact, verified pre-tag candidate instead of
rebuilding it after tagging, and move every gate ahead of the first public write. Keep a
separate extended proof level for release-system changes and stable milestones. Scope
vulnerability dispositions to the vulnerable component and a machine-checked exposure
invariant rather than unrelated rootfs bytes.
This is a safety and toil change, not a latency change. End-to-end wall clock for the
standard path increases, because the operator dispatches a candidate, checks it, and then
tags; runner-seconds increase by roughly the cost of one canonical build. What disappears
is the manual closeout described in #427: independently verifying the full asset
inventory, every cosign signature, every SLSA subject, and byte equality against both
retained builds, then hand-writing the proof into `release/REPEATABLE-BUILDS.md`.
This issue is an epic. It is tracked through the five workstreams below, which are
intended to land as separate area-scoped changes.
Recommended sequence: **workstream 1 and slice 3a first**, then run a beta on them before
committing to the rest. Together they remove the per-release CVE re-review and close the
publish-before-scan hole without a new operator flow, a second build, or any new
machinery. Workstream 1 is independent of everything else and blocks the next release that
carries any code change; slice 3a is independent of the candidate architecture.
## Assurance invariants
The implementation must remain fail-closed and preserve all of these controls:
- the release target is one exact commit on protected `main`;
- the version, release ID, manifest, lockfiles, and tag target agree;
- **no artifact reaches a public location before its scan gate passes**;
- two independent clean native-Linux builds, on separate runners with no shared compiled
output, produce byte-identical canonical Linux binaries, and produce OCI images whose
ordered rootfs layer digests and image config are identical. The published manifest
additionally carries BuildKit provenance and is intentionally not byte-comparable, so
the comparison runs without `--exact-image`;
- the full publish candidate is scanned by immutable digest and fixable or unreviewed
blocking findings fail the release;
- the tag is created only after candidate comparison and scan succeed;
- promotion consumes an explicitly selected candidate run and verifies its source SHA,
workflow identity, run attempt, inputs, builder/recipe fingerprints, artifact
inventory, and digests, with no "latest successful run" ambiguity;
- the post-tag workflow publishes the exact candidate bytes and OCI digests, not rebuilt
equivalents;
- checksums, file and image SBOMs, keyless cosign signatures, tag-bound SLSA provenance,
OCI labels, non-root execution, immutable publication, and final asset reconciliation
remain release gates;
- untrusted PR artifacts cannot become release artifacts;
- a stale, expired, mismatched, or partially uploaded candidate cannot be promoted;
- a release identity that has been promoted once can never be promoted again.
## Workstream 1: component-scoped advisory review binding
Owning areas: `products/notary`, `crates/registry-relay`. Independent of the rest.
**Blocks the next release that carries any code change.**
Introduce a versioned advisory-baseline schema whose security invariant is: a disposition
remains usable only while the exact vulnerable component and the reviewed exposure
condition are unchanged.
The current checker already enforces more than the issue's first draft implied. Schema v2
must state, per field, whether it is **enforced** or **recorded**, because
`evidence_image_digest` and `evidence_revision` are shape-validated at `:348` and then
never compared:
| Field | Today | v2 |
| --- | --- | --- |
| `tool\|subject\|CVE\|package\|version\|type` fingerprint | enforced | enforced, unchanged |
| severity | enforced (`mismatched`) | enforced, unchanged |
| fix availability | enforced (`block_fixable`) | enforced, unchanged |
| `expires_at`, `reviewed_at` | enforced | enforced, unchanged |
| `reviewed_rootfs_digest` | enforced | **removed** |
| `evidence_image_digest`, `evidence_revision` | recorded only | recorded only, documented as such |
| vulnerable component layer identity | absent | **enforced** |
| pinned runtime base identity | absent | **enforced** |
| exposure assertion | absent (prose in `reason`) | **enforced** |
| re-review triggers | absent (prose in `reason`) | recorded, structured |
For the component binding, prefer evidence already present in the Grype report: match
artifacts carry `locations[].layerID` from the syft package model, which is invariant to
application-layer changes by construction. Verify this against a real Grype report before
committing to it; the fixture at `advisory_baseline_check_test.py:121` only synthesizes
`source.target.layers`.
Exposure assertions must come from a **closed vocabulary**, evaluated by the checker
against the candidate image, not the source tree:
- `dynamic_symbol_absent`
- `package_absent_from_executable_closure`
- `file_digest_equals`
An assertion kind outside the vocabulary, or one that cannot be evaluated, fails closed.
The three current libc6 dispositions are all expressible as `dynamic_symbol_absent`.
The checker must force re-review when the component layer identity, base image, severity,
fix availability, exposure assertion definition, or exposure result changes or cannot be
established. It must continue to reject all fixable findings. An unrelated application
change may reuse an unexpired disposition only when the full-image scan still reports the
same finding and the exposure assertion re-evaluates successfully.
`crates/registry-relay/scripts/check_advisory_baselines.py` and
`products/notary/scripts/check_advisory_baselines.py` are currently byte-identical
557-line copies with separate tests. Default decision: keep both copies, land v2 in both
within one commit, and add a CI assertion that the two files are byte-identical, so
schema drift on the only vulnerability gate fails a check rather than a release.
Consolidating into one shared implementation is acceptable if it can be done without
splitting ownership across a third area.
Acceptance criteria:
- [ ] Negative tests in both `advisory_baseline_check_test.py` files cover every
invalidation condition: component layer change, base image change, severity change,
fix becoming available, assertion definition change, assertion evaluating false,
assertion unevaluable, unknown assertion kind, expiry, future-dating.
- [ ] A positive test proves an unrelated application-layer change retains a valid
disposition when component and exposure invariants still hold.
- [ ] The three current libc6 dispositions are migrated to v2 with
`dynamic_symbol_absent` assertions and no loss of coverage.
- [ ] No blanket suppressions; no automatic acceptance of new findings.
- [ ] The two checker copies are byte-identical and CI asserts it.
## Workstream 2: trusted pre-tag candidate workflow
Owning area: `.github/`, `release/`.
- Add a manually dispatched candidate workflow for an explicit `--version`,
`--release-id`, and exact protected-`main` commit.
- Reject a non-`main` target, unresolved or reused release identity, unexpected workflow
revision, and an existing tag or GitHub Release.
- Build the canonical Linux release outputs twice, as **two jobs on separate runners**
with no shared compiled-output state. Build A restores the exact-key Cargo cache
(`release.yml:207`); build B builds cold. The cache-state difference turns the duplicate
build into a cache-taint check instead of a near-tautology.
- Build other platform-specific release assets once in the same trusted candidate run.
- Publish the candidate images from build A with `type=registry,push=true` (BuildKit
provenance on) into **separate private GHCR staging packages**
(`registry-notary-candidate`, `registry-relay-candidate`). This is the publishable
artifact; nothing lands in a public package.
- Export build B as an OCI layout (`RELEASE_IMAGE_OCI_LAYOUT`, provenance off), pull
build A's manifest back into a layout, and compare with
`compare-release-image-layouts.py` **without** `--exact-image`.
- Compare canonical binaries byte for byte.
- Scan the staging images by immutable registry digest after equality is established. The
advisory checker needs no changes for this path: its `repoDigests` / `userInput@digest`
requirement (`check_advisory_baselines.py:189-203`) is already satisfied.
- Emit one machine-readable candidate receipt containing at least the source commit,
workflow/run/attempt identity, release inputs, builder and recipe fingerprints, exact
artifact inventory and SHA256 values, staging image references and manifest/config/layer
digests, per-build cache state, scan evidence coordinates, and comparison results.
- Attest the receipt and the candidate artifacts with `actions/attest-build-provenance` in
the candidate run. This is the truthful build attestation, bound to the run that
actually compiled, verifiable with `gh attestation verify`. Because the receipt covers
every other artifact by digest, attesting the receipt anchors the whole set.
- Retain candidate artifacts and staging images for 7 days.
Acceptance criteria:
- [ ] One candidate run builds the canonical Linux payload exactly twice, on separate
runners, with differing cache state, and the comparison passes.
- [ ] No public GHCR package or GitHub Release is written by the candidate workflow.
- [ ] The scan runs against the staging digest and fails closed on fixable, new blocking,
expired, or invalidly dispositioned findings.
- [ ] The receipt is attested and `gh attestation verify` succeeds against it.
## Workstream 3: tag-bound promotion
Owning area: `.github/`, `release/`.
### Slice 3a: move publication behind the scan gate (independent, land early)
The publish-before-scan hole does not depend on the candidate architecture and should not
wait for it. In the existing `images` job:
- push both images to the private staging packages instead of the public ones;
- run Syft and Grype against the staging digests and enforce the advisory policy there;
- `crane copy staging@ → public:` only after the policy passes, then assert
that the published digest equals the scanned digest.
This keeps the single build, adds no workflow, and needs no receipt. The tag still
precedes the gates, so a blocking finding still burns the version; that is an accepted
cost until the rest of this workstream lands, and workstream 1 makes it much less likely
by removing the rootfs binding that invalidates dispositions on every code change.
Caveats: the image SBOM and OCI-label checks move to the staging digest, which is the same
content and the same digest; release capsule metadata keeps referencing the public
repository, which is safe precisely because the copy does not change the digest.
Acceptance criteria for this slice:
- [ ] No public GHCR write occurs before the advisory policy passes.
- [ ] The published digest equals the scanned staging digest, asserted in the workflow
rather than by inspection.
- [ ] Existing SBOM, OCI label, non-root, and reconciliation checks pass unchanged.
### Full promotion
- **The operator pushes the tag; automation never writes refs.** No workflow gains
`contents: write` for refs. The property enforced technically is "no publication
without a verified candidate," not "no tag can exist."
- `registry-release finalize --candidate-run ` verifies the receipt, attestation, and
artifact digests locally, and prints the exact `git tag` command only on success. That
gives the operator a machine check before the tag exists, without granting CI ref-write.
- The tag workflow requires an explicit candidate-run binding, downloads and verifies the
receipt and every candidate artifact, and verifies the receipt's attestation identity
(repository, workflow path, ref, run id) before any write to GHCR or GitHub Releases.
- Promote images with `crane copy staging@ → public:`, then assert the
published digest equals the candidate digest. Cross-repo copy preserves manifest and
index bytes, including the provenance attestation manifest, so the digest is identical
by construction. Never `docker push` a loaded image; that rewrites the manifest.
Note: images are not cosign-signed today (`release.yml:709` signs blobs only), so no
signature or referrer artifacts need copying. If image signing is added later,
promotion must copy the signature tags too.
- Promote release files without rebuilding product binaries.
- Generate tag-bound checksums, release capsule metadata, and cosign signatures from the
promoted bytes. **Keep `generator_generic_slsa3` in the promotion run exactly as it is
today** (`release.yml:786`): its source binding stays truthful because the candidate is
built from the tag's exact commit, `slsa-verifier --source-tag` and the documented
commands in `release/VERIFY.md` keep working, and Scorecard Signed-Releases is
unaffected. Only the run identity is imprecise, and the receipt closes that gap.
- Publish the signed candidate receipt as a GitHub Release asset, so
tag → candidate run → build attestation is a public chain.
- Reconcile the final GitHub Release asset inventory and hashes against the verified
candidate plus the expected tag-bound evidence files.
- Reject a candidate older than 72 hours, a candidate whose release identity has already
been promoted, and any receipt whose digests do not match the downloaded bytes.
- Failure states: promotion failing **before** any public write is retry-safe from the
same candidate. A failure **after** any public write fixes forward with a new patch
version, as today. A tag with no published release is a normal, non-fatal state: the
version number is burned and the next attempt uses a new patch version.
Acceptance criteria:
- [ ] Unit tests over the promotion verifier with synthetic receipts prove that changing
any one of: a payload byte, source SHA, workflow identity, run attempt, release
input, builder fingerprint, artifact digest, OCI digest, scan coordinate, or
receipt attestation identity, fails promotion before any public write. One live
tamper drill exercises the same path end to end.
- [ ] Stale-candidate (>72h), already-promoted-identity, and partially-uploaded-candidate
tests fail before any public side effect.
- [ ] A completed beta release publishes the exact expected inventory, SBOMs, checksums,
cosign material, and SLSA provenance, and final reconciliation succeeds.
- [ ] The published image digest equals the candidate staging digest, verified in the
workflow, not by inspection.
## Workstream 4: standard and extended proof levels
Owning area: `.github/`, `release/`.
Make the standard candidate-plus-promotion path the default for beta releases.
Automatically require, or allow the operator to explicitly select, the extended proof
level when any of these apply:
- release workflow, build recipe, packaging, signing, provenance, or advisory-policy code
changed;
- builder, BuildKit, runtime base image, or other trust anchor changed;
- the two candidate builds disagree or evidence is incomplete or ambiguous;
- a stable or 1.0 milestone is being prepared;
- a security or external audit explicitly requests the extra proof.
Add a scheduled and manually dispatchable repeatability job that rebuilds a selected
published tag from clean state and compares it with published hashes and digests.
State plainly, in the workflow and in the docs, that two builds inside one candidate run
prove **build determinism**, not environment independence. #127's claim is carried by this
scheduled rebuild job, not by the in-run duplicate build, and the public evidence must not
overclaim.
Acceptance criteria:
- [ ] One end-to-end extended proof validates the new machinery on a real published tag
(target: the first release after workstream 3 lands); subsequent unchanged betas use
the standard path.
- [ ] The scheduled repeatability job runs against a published tag and refreshes #127
evidence without operator coordination.
## Workstream 5: operator, storage, measurement, and public evidence
Owning areas: `release/`, `docs/site/`.
- Add a preflight that measures available storage against a documented release-workspace
budget and fails before expensive builds when the runway is insufficient. Size the
budget from a real measurement of peak disk on a candidate run, not an estimate. Note
that splitting the two builds across separate runners materially lowers per-runner
pressure. If no disk exhaustion has actually been observed, record the gate as
preventive.
- Bound artifact and staging-image retention to 7 days and document safe cleanup. Do not
require multi-gigabyte local evidence caches for the standard hosted path.
- Update `release/VERIFY.md` and
`docs/site/src/content/docs/security/openssf-evidence.mdx` to the current release model.
That page is stale: its newest examples are v0.8.x while `main` is at v0.13.0. Add one
paragraph stating that release provenance binds source and published bytes, and that
the build run identity lives in the candidate receipt.
- Record the OpenSSF Best Practices Silver answers in-tree, with criterion names and
evidence URLs, so this issue's badge criteria are checkable against something and a
docs edit cannot silently drift from the badge. At minimum `build_repeatable` (Met,
cite `release/REPEATABLE-BUILDS.md`) and `signed_releases` (Met). Signed version tags
are recorded as not implemented at `openssf-evidence.mdx:63` and `:86`; adopting
`git tag -s` in workstream 3's operator step would flip that, and is worth considering
while the operator is the one creating tags. Verify the exact criterion identifiers
against the project's page on bestpractices.dev rather than from memory.
Acceptance criteria:
- [ ] The release does not regress any answer recorded in `openssf-evidence.mdx`, and
preserves the inputs required for Scorecard Signed-Releases score 10.
- [ ] Insufficient-storage tests fail before expensive builds.
- [ ] Public verification docs and #127 evidence are updated with commands an external
reviewer can replay.
## Measurement
Baseline, from beta-16/v0.12.2 (#427, 2026-07-20): 2,728 seconds wall clock to the
telemetry collector, 4,482 completed runner-seconds (74.7 minutes), canonical Linux binary
build 2,307 seconds, `exact_key_hit: false`.
Budgets for the standard path, excluding GitHub runner queue time:
- candidate run wall clock: 60 minutes
- promotion run wall clock: 20 minutes
- total runner-seconds: 8,000 (the second canonical build adds roughly 2,300)
These are **review triggers recorded by telemetry, not release gates**. A slow release
must not fail closed. Per-job timing, cache status, runner occupancy, and peak storage are
recorded on every release; queue delay is reported separately.
## Non-goals
- Removing source/tag binding, vulnerability scanning, signatures, provenance, SBOMs, or
reconciliation.
- Reusing artifacts from pull-request workflows.
- Granting any workflow write access to Git refs.
- Treating OpenSSF aggregate-score improvement as evidence of release integrity.
- Claiming OpenSSF Best Practices Gold or universally reproducible builds.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start by choosing one area-scoped workstream, then read .github/workflows/release.yml, release/scripts/build-release-image.sh, and the two check_advisory_baselines.py copies with their advisory_baseline_check_test.py tests. Run the existing checker and release-image comparison tests first; done means the selected workstream's acceptance criteria and listed assurance invariants are covered without weakening existing gates.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- docker, github-actions, python, rust
- Ambito
- build-system, ci-cd, devops, release, security
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 28/100