stacklok / stacklok/toolhive

E2E coverage for the cosign key-pair signing path (skills and plugins)

Open
#6,642 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage skills testing
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Problem

The --key / --public-key path shipped for skills (#6447, #6478) and plugins (#6524, #6525, #6528) with unit tests only, by agreement on #6442 — the push side did not exist on plugins until the last PR, and cosign is not on the E2E runner. Every test of the path drives the mocked verifier.Verifier; nothing has ever exercised the real signer → registry → verifier chain for a key-pair signature end to end.

That leaves a specific class of bug invisible: a mismatch between what signer.SignOCI writes for a key-pair signature (simple-signing payload, .sig manifest layout, annotations) and what verifier.VerifyOCIWithKey / VerifyBundleOfflineWithKey expect to read back. The mocks agree with each other by construction. bundleSignsPayload (the anti-transplant check) and the "payload recovered from the stored bundle, not rebuilt from a reference" fix in #6525 are exactly the kind of logic that only a real round trip validates.

Proposed scope

Add one Ginkgo spec per surface to test/e2e/ (the plugins side can extend cli_plugins_lock_test.go, which already has the registry + project harness):

  1. cosign generate-key-pair into a temp dir (or check in a fixed test key pair; a checked-in key removes the cosign binary dependency and is fine for a test that asserts behaviour, not secrecy).
  2. thv skill build / thv ai-plugin build, then push --key cosign.key to the local registry with COSIGN_PASSWORD set on the thv serve process.
  3. install --scope project --public-key cosign.pub; assert the lock entry has publicKey and no signerIdentity, and info renders "cosign key pair".
  4. install again with no flag → uses the pinned key; install with a different --public-key → 403 conflict.
  5. sync --checkAlreadyCurrent; tamper with the on-disk tree → drift → sync repairs via the pinned key.
  6. upgrade after pushing a new version signed with the same key → upgraded; after pushing one signed with a different key → failed [signature-invalid] with the reinstall remedy; after pushing keyless → signer-change-blocked.
  7. install --allow-unsigned on the key-signed artifact with no key → 403 naming --public-key, not a recorded unsigned exception.

Also assert the key-signing capability from #6560: a push --key from a client without the discovery capability is refused.

Related

  • #6442
  • test/e2e/cli_plugins_lock_test.go (existing harness)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/e2e/cli_plugins_lock_test.go and its registry and project harness, then trace the existing skill and plugin build, push, install, sync, and upgrade specs. Add end-to-end coverage for the key-pair signing round trip and the listed conflict, drift, upgrade, unsigned, and discovery-capability outcomes; run the E2E suite to verify the real signer-to-registry-to-verifier path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
security, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.