Help wanted: integrate SkillOpt with Superpowers for measurable agent efficiency

Summary

We are looking for a contributor to explore a narrow, reproducible integration between SkillOpt and obra/superpowers.

Superpowers supplies composable software-development workflows through SKILL.md files, session bootstrap, and harness-specific tool mappings. SkillOpt proposes bounded edits to a skill and accepts a candidate only after validation. The hypothesis, not a current performance claim, is that SkillOpt could help evaluate or improve an individual Superpowers skill while preserving workflow correctness and measuring execution cost.

A result where no candidate beats the current skill is still useful.

Important integration constraint

A valid evaluation must run the normal Superpowers bootstrap and harness integration, with only the selected skill replaced through an isolated candidate overlay. Testing one SKILL.md as a standalone system prompt is not equivalent to running Superpowers and should not be reported as Superpowers support.

This work should reuse the generic existing-skill targeting in #120 rather than duplicate it.

Proposed MVP

1. Pin a released Superpowers version and commit SHA.
2. Select one skill with checkable behavior, such as verification-before-completion or systematic-debugging.
3. Keep the rest of the pinned Superpowers installation unchanged.
4. Run baseline and candidate skills with the same coding-agent harness, model, settings, tasks, seeds, permissions, and budgets.
5. Keep training, validation, and final test scenarios separate.
6. Stage and report candidates only; never modify an installed Superpowers checkout or a live user skill automatically.
7. Record provenance and raw results: source ref/path, harness/model/settings, split, SkillOpt config, accepted/rejected edits, quality, tokens, tool calls, turns, latency, and failures.

The implementation can be a small runner/benchmark adapter plus documentation and tests. It may use the existing --target-skill-path support.

Acceptance criteria

• A documented command works from a clean checkout with a pinned Superpowers ref and one target skill.
• Candidate injection uses an isolated overlay; source and installed Superpowers trees remain unchanged.
• Baseline and candidate runs use identical settings and held-out validation.
• Required task behavior and workflow compliance are the primary gate; lower cost cannot compensate for a quality regression.
• Validation fails closed on timeout, non-zero exit, malformed output, missing score, or incomplete trajectory.
• Deterministic offline tests cover optimization, rejection, cleanup, and source-tree non-mutation.
• At least one opt-in real-harness smoke test is documented for Codex or Claude Code.
• Reports provide enough provenance and raw data to reproduce the comparison.
• No general efficiency claim is made from a single task, seed, model, or harness.

Safety, privacy, and attribution

• Run agents in a throwaway workspace and isolated HOME.
• Prefer synthetic or public tasks; treat trajectories as potentially sensitive.
• Do not expose credentials or run live model calls in public CI.
• Candidate adoption must remain an explicit human decision.
• Preserve the MIT license and attribution for any redistributed Superpowers material, and label generated candidates as derivatives rather than official releases.
• Do not copy or vendor evaluation code or scenarios that lack an explicit compatible license.

Non-goals

• Optimizing every Superpowers skill at once
• Editing bootstrap, manifests, hooks, or auxiliary scripts in the first MVP
• Automatically changing a user installation
• Requiring a particular API provider
• Claiming an official partnership with Superpowers maintainers
• Automatically opening an upstream Superpowers PR

Related SkillOpt work

• #120 — target existing hand-written skills
• #111 — related exploration for Cursor-hosted skills
• #54 — prior discussion of skills with auxiliary files

Interested?

Before substantial implementation, please comment with:

1. the target Superpowers skill and pinned tag/SHA;
2. the target harness and model;
3. the proposed task/evaluation source; and
4. whether changes beyond a runner, documentation, and tests appear necessary.

Start by reading issue #120 and the existing --target-skill-path support, then identify the runner and benchmark entry points. Choose and document the target skill, pinned Superpowers ref, harness, model, and evaluation source; done means isolated candidate injection, reproducible baseline comparisons, fail-closed validation, offline tests, and provenance-rich reports.