Epic: Software supply chain security — verifiable release artifacts

Problem

Every artifact this project publishes is consumed by operators who run it with cluster-admin-adjacent RBAC on GPU fleets. Today a consumer cannot answer "did NVIDIA build this, from what source, containing what?" for most of what we ship.

The container image is signed and carries an SBOM. Nothing else is. And the image's own coverage has defects:

• Wrong SBOM subject. publish.yml:96 generates one SBOM against the multi-architecture index digest. Syft resolves an index to a single platform, so the linux/arm64 image is described by a linux/amd64 SBOM. An SBOM describes exactly one root filesystem; its subject must be a per-platform manifest digest.
• No build provenance anywhere. No SLSA statement links any artifact to the source commit, the workflow, or the run that produced it. A signature proves "someone in this repo signed it," not "this came from tag vX.Y.Z."
• The published verification command does not verify what it claims. SECURITY.md:68 instructs users to pass --certificate-identity-regexp='https://github.com/NVIDIA/cluster-readiness-engine'. That regexp is unanchored and workflow-agnostic. publish.yml also runs on push to main (publish.yml:7-9), signing main-<sha> images under .../publish.yml@refs/heads/main — which that regexp accepts. A user following our own documentation will accept an unreleased main-branch build as a release.
• The signing toolchain is not pinned. anchore/sbom-action@v0.24.2 (publish.yml:94) is a floating tag while every other action in the repo is SHA-pinned. cosign-installer (publish.yml:86) has no cosign-release, so the cosign version — and with it the attestation bundle format and Rekor backend — is an inherited default that moves without a commit here.
• Nothing ever verifies. The post-publish gate at release.yml:274-304 checks SHA-256 digests and the version string. It never runs cosign verify. A release where signing silently no-ops ships green.
• The release notes make a claim the release never checked. release.yml:226 tells users the image is .../manager:<tag>. That image was published by a different workflow run (publish.yml) with no digest handoff. release.yml never confirms it exists, never learns its digest, and never verifies its signature.

Unsigned entirely: the Helm chart, all four nvcrectl binaries, and the installer script that our own README tells users to curl | bash.

Status — 2026-09-04

Phases 0–3 are complete. Phase 4 is underway. v0.2.0-rc.1 ran the whole pipeline end to
end in 10m28s: run 33753017998,
release. The gate
drafted the release, verified every artifact, published it, and re-checked it anonymously.
v0.2.0-rc.2 proved the gate rejects a tampered artifact; #302 then proved it rejects an
absent one, which is a different set of branches — the ones that run when a signing step
silently no-ops.

	Count	Issues
Closed	9	#263, #264, #265, #266, #267, #268, #269, #271, #276
Open — 9 of 11 criteria met	1	#270
Open — enforced only on a fork	1	#274 (draft PR #295, verified sound)
Open — parked, needs rework	1	#272
Open — not started	3	#273, #275, #301

Every artifact this epic set out to sign is signed, attested, verified at release time, and
documented with commands that CI executes verbatim. The identity hole in SECURITY.md:68 — a
regexp that accepted a main build as a release — is closed and the fix is enforced by tests.

Landed since the last review

• #276 closed (#305). The weekly image scan ships. Dispatched against main on 2026-09-04
  and posted a real Slack alert, so the notification path is verified end to end rather than
  assumed. It found a genuine issue on its first run: v0.1.0 ships
  google.golang.org/grpc@v1.83.0 with fixable High GHSA-vp52-pcj8-j9qc, already fixed on
  main by #261 the day after v0.1.0 shipped. That is precisely the gap this epic's Phase 4
  exists to close — a CVE published against a release nobody would otherwise notice. A release
  is planned for Monday 2026-09-08, well inside SECURITY.md's 30-day window.
• Suppression policy consolidated (#306, #307). .grype.yaml ignore rules were replaced by
  .openvex.json, consumed as grype --vex, with .grype.yaml enforced empty of suppressions.
  Grype applies both in the same run, so two homes meant the impact analysis for a CVE could
  live in either file and the weaker-enforced one was the easier to reach for. OpenVEX has no
  expiry field, so the deadline became a 180-day re-affirmation rule, checked before the scan
  fans out. Triage procedure in .claude/skills/managing-openvex.md. This changes a non-goal
  below — see the corrected bullet.
• #302 proved the release gate rejects absence, not only tampering, and corrected an ADR-074
  claim that described logic which was never built.

What the RC proved

What the RC proved

Re-verified from a workstation with only cosign v3.1.3, crane and jq, rather than trusting
the pipeline's own output.

Check	Result
Image index signature and SLSA provenance	OK
Provenance predicate	repository=NVIDIA/cluster-readiness-engine, ref=refs/tags/v0.2.0-rc.1, commit=934f472a… — the tagged commit exactly
Per-platform CycloneDX SBOMs (amd64, arm64)	OK, bound to their own manifest digests
blob subject kind — first ever execution	13 bundles, 25 assets, no filename collision
All 5 binaries + installer, all 4 SBOM sidecars	OK
Chart signature and provenance	OK
All 4 digests in the release notes	resolve
Gate: draft → verify → publish → anonymous re-check	passed; retraction step correctly skipped
Gate rejects a tampered artifact (rc.2)	ok checksums.txt above two signature failures

The identity does real work:

identity for v0.1.0     -> REJECTED
old documented regexp   -> ACCEPTED   (the hole this epic closed, still reproducible)

What remains

1. The gate rejects tampering; it has not been shown to reject absence. #270's three open
   criteria — a deleted asset, a skipped signing step, a legacy-format bundle — are different
   code paths from the one rc.2 exercised. The first two are testable without cutting a tag,
   using the stub-harness pattern test/releasepolicy already uses for the installer.
2. Four hardcoded asset lists can drift from what the release uploads (#274). Adding a
   platform ships an unsigned binary with every gate green.
3. A consumer-compatibility gap that was not in the original plan. The bundle format we
   publish is not readable by Kyverno's legacy verifyImages path or by Sigstore
   policy-controller at its chart default. Documented on the verification page; enforcement
   samples are #272.
4. Nothing re-verifies after publication (#273) and nothing watches the transparency log
   for signatures we did not make (#275).

The recurring failure in this epic

Worth recording, because it happened at least seven times and every instance was caught by
review or by running the thing rather than by reading it: checks that look protective and are
not. A 2>&1 that let a gh upgrade notice corrupt a draft-state variable. A skip-verify test
searching for a string the file never contains. A cosign stub that ignored its arguments. A docs
gate that skipped and read as green. A grype suppression test bypassable two ways. In each case
the code read correctly. Reading correctly is what all of them had in common.

Per-issue evidence is posted on #267, #268, #269, #270 and #271.

Goal

Every artifact a user can download or pull from a tagged release carries a Sigstore signature, a SLSA Build Provenance v1 statement, and an SBOM — produced by a tamper-resistant builder, verified by the release itself before publication, verified again daily after publication, and documented with an identity contract precise enough that following it rejects anything we did not release.

Post-release assurance extends to vulnerability posture: the image we shipped is scanned weekly against current advisories, because a CVE published the day after a release affects every operator running it and nothing in this repository would otherwise notice.

Non-goals

• Migrating the release to GoReleaser. Separate epic if we ever want it.
• A private Sigstore / KMS signing path. Public-good Sigstore keyless only.
• Signing the docs site, test fixtures, or UAT artifacts.
• An nvcrectl verify subcommand. cosign and gh cover it; a bespoke verifier is surface area we would have to keep correct.
• OpenVEX vulnerability triage. Adopted 2026-09-04 (#306). The original reasoning was that triage needed a process to exist first, with .grype.yaml suppressions starting it. That process ran for exactly one finding before it became clear that keeping two suppression mechanisms was the problem, not the sequencing: grype applies .grype.yaml ignores and OpenVEX statements in the same run, so the impact analysis for a CVE could live in either file and the weaker-enforced one was the easier to reach for. .openvex.json is now the single home, consumed as --vex, with .grype.yaml enforced empty of suppressions. Publishing a signed OpenVEX attestation remains out of scope — ADR-074's artifact contract still has no OpenVEX row, and decision 5 keeps both publication routes open.

Target artifact contract

Artifact	Signature	SBOM	Provenance	Subject
manager image, per platform	cosign, new bundle format	CycloneDX-JSON, one per platform	—	per-platform manifest digest
manager image, index	cosign	—	SLSA Build Provenance v1	index digest
Helm chart (OCI)	cosign	—	SLSA Build Provenance v1	chart OCI digest
nvcrectl-{linux,darwin}-{amd64,arm64}	cosign attest-blob bundle	CycloneDX-JSON per binary	SLSA Build Provenance v1	file digest
installer	cosign attest-blob bundle	—	SLSA Build Provenance v1	file digest
Every *.cyclonedx.json release asset	cosign attest-blob bundle	—	—	file digest

Verification identity contract

All release attestations are produced by one reusable workflow, so a consumer pins exactly one identity:

--certificate-oidc-issuer https://token.actions.githubusercontent.com
--certificate-identity   https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/<TAG>

Exact, not a regexp. It names the workflow and the tag. A main build cannot satisfy it, and neither can a signature from release vX presented as release vY.

Key design decisions

#	Fork	Decision	Rationale
D1	Adopt GoReleaser vs. extend the existing hand-rolled build	Extend what's there	release.yml carries hard-won guards from #194/#195 (version stamping, installer 404). A GoReleaser migration churns all of it and is a separate epic. Reversible later.
D2	Attest inline vs. from a reusable workflow	Reusable attest.yml, targeting SLSA Build L2 (corrected 2026-09-02)	It collapses signing into one stable certificate identity users pin — which is what fixes the main-branch-looks-like-a-release hole — and isolates the signing token from caller-defined build steps. It does not reach L3: the builds stay in the callers, and L3 requires the build itself to run in the protected reusable workflow. Reaching L3 is deferred to its own decision. See ADR-074 decision 4.
D3	SBOM format	Stay on CycloneDX-JSON (reversed 2026-09-02)	Both are first-class cosign predicate types and both are formally standardized (CycloneDX ECMA-424, SPDX ISO/IEC 5962), so the original "first-class type" and "the ISO standard" arguments for switching favored neither. What remained was cross-repo consistency, worth one flag value — against a breaking change for anyone parsing the predicate today, and the loss of CycloneDX's native VEX carriage, which matters given #276. See ADR-074 Alternatives Considered #5.
D4	Binary attestation transport	cosign attest-blob .sigstore.json sidecars as release assets	Verifiable with cosign alone, offline, no GitHub API dependency. Registry-attached actions/attest-build-provenance is used for the image, where it's the right fit.

Phases

Phase	Outcome	Subtasks
0 — Foundation	Design of record; toolchain pinned so results are reproducible	#263, #264
1 — Produce	Every artifact signed, attested, SBOM'd	#265, #266, #267, #268, #269
2 — Verify	A release that does not verify does not publish	#270
3 — Document	Users and admission controllers can act on it	#271, #272
4 — Assure	Tampering and new vulnerabilities after publication are detected	#273, #274, #275, #276

Subtasks

#	Subtask	Phase	Depends on	Size	Status
#263	ADR-074: supply chain artifact and verification contract	0	—	S	✅ Closed — #277
#264	Pin and harden the release signing toolchain	0	—	S	✅ Closed — #278
#265	Reusable attestation workflow (attest.yml) — isolated signer	1	#263, #264	L	✅ Closed — #281, #282
#266	Container image: SLSA provenance and per-platform SBOMs	1	#265	M	✅ Closed — #286
#267	Helm chart: signature and provenance	1	#265	M	✅ Closed — #289
#268	CLI binaries and installer: provenance, SBOMs, signed sidecars	1	#265	L	✅ Closed — #290, #298
#269	Digest handoff between publish and release	1	#266, #267	M	✅ Closed — #292
#270	Release-time verification gate	2	#266, #267, #268, #269	M	🟡 9 of 11 criteria. #293 built it, #302 proved it rejects absence. Two open — see below
#271	User-facing artifact verification documentation	3	#270	M	✅ Closed — #299
#272	Admission-policy samples (Kyverno, Sigstore policy-controller)	3	#271	S	⬜ Parked — needs rework for the bundle-format gap
#273	Daily release re-verification	4	#270	L	⬜ Not started
#274	Workflow policy regression tests	4	#265, #270	M	🟢 Draft PR #295 satisfies all 5 criteria, mutation-verified; blocked on 9 lint findings
#275	Rekor signing-identity monitoring (stretch)	4	#273	M	⬜ Not started
#276	Weekly image vulnerability scan with Slack notification	4	#266	M	✅ Closed — #305, #306, #307
#301	ADR-074 D2: correct the Build Level reasoning, and gate the reusable-workflow boundary	4	#265, #266	M	⬜ Not started — not blocking v0.2.0

#270's two open criteria.

• "Skipping any signing step fails the gate (test by dispatch with the step disabled)." #302
  covers the job level: the three needs.<job>.result guards are driven through skipped,
  failure and cancelled. A signing step disabled inside a job that still succeeds passes
  all three. verify-release re-checks signatures against the registry and would catch it, but
  that path is untested and the criterion prescribes a dispatch.
• "A legacy-format bundle is rejected rather than read." #302 established that no such logic
  exists anywhere in the release path — ADR-074 claimed it did, and that claim has been
  corrected. This criterion therefore describes work that was never done, not a test that is
  missing. It needs a decision: implement it, or strike it with reasoning.

#274 is close, and the work is not ours. Draft PR #295 (asivanadi0, on a fork) satisfies
all five acceptance criteria. Verified on 2026-09-04 by mutating each defect rather than
reading the tests, since this issue's own standard is that a test which does not fail on the
reintroduced defect is decoration: a rogue cosign sign in publish.yml, a rogue
actions/attest-build-provenance in release.yml, a second trigger on attest.yml,
id-token: write on a non-signing job, an action unpinned to @v7, and a fifth platform in
NVCRECTL_PLATFORMS — every one caught. No network, no cluster, passes alongside the existing
suite.

It is blocked only on lint: 9 mechanical findings (5 lll, 2 modernize, 1 prealloc, 1
goconst), no logic changes. Reviewed with specifics at
https://github.com/NVIDIA/cluster-readiness-engine/pull/295#pullrequestreview-5114169047.

This issue stays open until that merges. Nothing is enforced on main today — a rogue
signer, an unpinned action, or a new platform would all pass CI. The C4 pinning check is not
hypothetical: dependabot moves anchore/scan-action weekly under patterns: ["*"], and #306
had to add a runtime assertion because nothing guaranteed that pin stays a pin.

Phases 0–2 are this epic's definition of done. Phases 3–4 are in scope but cuttable if the release date moves.

Sequencing

#263 and #264 are parallel. #265 gates all of Phase 1. #266/#267/#268 are parallel once it lands. #269 needs #266 and #267. #270 needs all of Phase 1. #271 → #272 and #273 → #275 are parallel tails. #276 needs only #266 and can run alongside either tail. #274 can start as soon as #265 exists and should land before #270 merges, so the gate is born with a regression test.

Validation checkpoint

Most of this epic changes CI workflows whose real behaviour only appears when a tag is pushed — the attestation certificate identity, the digest handoff, and the release-time gate cannot be fully proven by a dry run. #269 has landed, so the RC is ready to cut once #270 merges; taking it after #270 means one tag exercises the gate as well, and a gate defect withholds a draft rather than publishing an unverified release. vX.Y.Z-rc.N clears all three tag validators and is marked as a prerelease automatically.

Findings from that run feed back into #266–#270 before any of them is considered done.

Acceptance criteria deferred to that RC run, carried here so they survive their issue closing:

From	Criterion
#264	cosign version printed in the publish job log matches the pinned COSIGN_VERSION
#264	A published image carries attestations discoverable via cosign download attestation on the referrers path
#267	The chart's signature and provenance verify against the tag identity, and helm pull by digest returns the chart that was verified
#267	Whether Flux OCIRepository (.spec.verify.provider: cosign) and Argo CD can consume the signed chart — believed yes for Flux, unconfirmed against a real version
#268	The blob subject kind has never executed; the RC is its first run. All 13 bundles emit with no filename collision
#268	The installer's own fail-closed verification, cosign bootstrap and --skip-verify — deferred until the bundles exist to test against
#269	release.yml builds the image it advertises, and the release notes carry digests that resolve
#270	The gate runs, the release is drafted, verification passes, and only then does it become visible
#270	A deliberately broken artifact is withheld — the gate must be shown to reject, not only to pass

Tool pins that no updater currently watches are tracked separately in #279.

Epic acceptance criteria

• For a freshly cut tag, every row of the artifact contract table verifies against the pinned identity from a clean machine with only cosign and gh installed.
• Deleting any signing step from any workflow fails CI.
• Deleting a published release asset, or repointing a tag to a different digest, opens a security issue within 24 hours.
• SECURITY.md and the new verification page contain no unanchored identity regexps.
• No release artifact is produced by an unpinned tool version.
• A HIGH+ vulnerability in the released image reaches a human within a week, with the package, the advisory ID, and a triage path.

Risks

Risk	Mitigation
Sigstore outage fails a release	Retry with backoff around every cosign call; liveness probes in the daily monitor so an outage reads as operational, not as tampering
SBOM format switch breaks a consumer	Repo is pre-1.0; announce in release notes for the first release that carries it
Reusable-workflow refactor breaks the release path	Land it behind a dry-run dispatch and cut a -rc tag before the first real release uses it
ghcr.io does not implement the OCI referrers API	Attestations land via the spec's referrers tag fallback; transparent to cosign, but documented commands must use cosign rather than a raw referrers call

Open questions

1. SPDX vs. CycloneDX (D3). Resolved 2026-09-02: staying on CycloneDX. Recorded as a rejected alternative in ADR-074 so the reasoning is not relitigated.
2. Digest handoff approach. Resolved 2026-09-02 (#292): the image build moved into a reusable build-image.yml. release.yml calls it on a tag and receives the index and both platform digests as job outputs; publish.yml triggers on main only and owns main-<sha> dev images. The two workflows were not deliberately independent — they both fired on a tag and raced, and the release asserted an image tag it had never resolved. The race is removed rather than mitigated.
3. installer behavior without cosign. Resolved 2026-09-02: fails closed. The soft fallback contradicted the epic's own premise that a same-origin unsigned checksum proves nothing. installer now bootstraps a digest-pinned cosign, and skipping verification requires an explicit --skip-verify flag. See ADR-074 Consequences.
4. Rekor identity monitoring (#275) — stretch, or in the committed set? Still open. Phases 0–2 are the definition of done, so this is cuttable.
5. Milestone. Proposed v0.2.0 for Phases 0–2. No milestones exist in the repo yet.
6. Slack webhook secret (#276). Needs a maintainer with repo admin to create SLACK_WEBHOOK_PATH and pick a destination channel. Blocks the notification path, not the scan itself.
7. Scan target (#276). Proposal scans both the latest released image and main. If only one, keep the released image.