Plan: skill-verify — AI Skill Quality Scorer

# skill-verify — Plan

## Problem

AI skills are proliferating fast (awesome-copilot has 300+ skills), but quality varies wildly. There is no single tool that tells a skill author "here is exactly what to fix and why." The existing tools each cover a slice:

| Tool | What it checks | Gaps |
|------|---------------|------|
| **Sensei** (spboyer/sensei) | Frontmatter quality, trigger phrasing, routing clarity, token budget, bad patterns | No body structure analysis, no repo-context verification, no cross-file consistency |
| **awesome-copilot validator** | Name format, description exists, assets exist + <5MB, folder match | No quality scoring — pass/fail only, no recommendations |
| **Agent Skills Spec** | Authoring guidelines (scoped, actionable, unambiguous, tested) | Spec is prose — not machine-checkable |
| **ai-ready** (this repo's sibling) | Repo-level AI readiness (12 assets) | Checks repos, not skills themselves |

**skill-verify unifies all four** into one actionable skill.

---

## Research Findings

### What "High quality" means (synthesized from all sources)

#### Frontmatter (from Sensei)
- Description: 150–1024 chars, ≤60 words, action-verb-led
- Starts with skill-type prefix: `**ANALYSIS SKILL**`, `**WORKFLOW SKILL**`, `**UTILITY SKILL**`
- Contains `WHEN:` trigger phrases (quoted, specific)
- Contains `INVOKES:` (tools used) and `FOR SINGLE OPERATIONS:` (when NOT to use)
- Name: lowercase-hyphenated, matches folder, ≤64 chars
- No secrets, API keys, TODO/FIXME/HACK

#### Body Structure (from awesome-copilot top skills)
- `## Steps` or `## Rules` sections
- Self-contained instructions (no external context needed per step)
- Completion criteria defined
- Edge cases addressed
- Token budget: soft 500, hard 5000 (SKILL.md itself)
- Progressive disclosure via `references/` for detailed content

#### Repo Context (from major repos research)
- Skills that reference external repos should validate those refs still exist
- Cross-file consistency (if SKILL.md mentions references/, they must exist)
- If bundled assets listed in frontmatter, verify they exist and are <5MB

#### Real-World Patterns (from Angular, React, vscode-peacock, johnpapa repos)
- PR templates with checklists → skills should have "done means done" criteria
- Conventional commit discipline → skills should have clear naming
- Maintenance matrix awareness → skills should state what else changes when they change
- The best skills (security-review, create-specification) are narrowly scoped, define when to use AND when not to, give explicit workflows, and include self-verification steps

---

## What skill-verify Will Check (Scoring Rubric)

### 5 Scoring Dimensions

| Dimension | Weight | What it checks |
|-----------|--------|---------------|
| **Frontmatter** | 25% | Description quality (length, words, action-verb, triggers, routing), name validity, optional fields (license, version) |
| **Structure** | 25% | Has Steps/Rules sections, code blocks with language IDs, progressive disclosure (references/), token budget compliance |
| **Clarity** | 20% | Self-contained steps, no ambiguous pronouns, completion criteria, edge cases, "when NOT to use" guidance |
| **Safety** | 15% | No secrets/bad patterns, file guards ("check if exists"), no destructive defaults, idempotent operations |
| **Discoverability** | 15% | Skill-type prefix, WHEN triggers match likely user phrasing, description works as standalone pitch |

### Scoring Output

| Score | Medal | Meaning |
|-------|-------|---------|
| 90–100 | 🏆 Exemplary | Publishable as a reference skill |
| 75–89 | 🥇 Strong | Ready for production use |
| 60–74 | 🥈 Adequate | Works but has improvement areas |
| 40–59 | 🥉 Developing | Needs significant work |
| 0–39 | ⚠️ Draft | Not ready for use |

### Per-Check Output Format

```
✅ Frontmatter: description is 187 chars, 42 words — concise and action-led
⚠️ Structure: no `## Steps` section found — add explicit step numbering
❌ Safety: found TODO on line 34 — remove before publishing
💡 Discoverability: add WHEN: trigger phrases for better matching
```

---

## Architecture

```
skill-verify/
├── skills/
│ └── skill-verify/
│ ├── SKILL.md # The verification skill (~200 lines)
│ └── references/
│ ├── scoring-rubric.md # Detailed scoring criteria
│ ├── sensei-checks.md # Sensei-derived frontmatter checks
│ ├── spec-checks.md # Agent Skills Spec compliance checks
│ └── examples.md # Before/after examples of improvements
├── .github/
│ ├── copilot-instructions.md
│ └── workflows/ci.yml
├── AGENTS.md
├── CHANGELOG.md
├── README.md
└── LICENSE
```

---

## How It Works

1. User says: "verify this skill" or "score my SKILL.md" or "is this skill ready to publish?"
2. skill-verify locates SKILL.md files (glob for `**/SKILL.md`)
3. Runs 5-dimension analysis
4. Produces a scored report with ✅/⚠️/❌ per check
5. Offers actionable fixes: "Add WHEN: triggers to reach High frontmatter score"
6. Optionally applies fixes (with user confirmation)

---

## Differentiation from Sensei

| | Sensei | skill-verify |
|---|--------|-------------|
| Scope | Frontmatter + triggers | Full skill (frontmatter + body + repo context) |
| Output | Score label (Low/Med/High) | 5-dimension score + per-check actionable fixes |
| Fixes | Recommendations only | Can apply fixes directly (with confirmation) |
| Context | Standalone evaluation | Checks cross-file consistency (references exist, assets valid) |
| Integration | External tool | Copilot CLI skill (runs where you work) |

---

## Implementation Phases

### Phase 1: Core Skill (MVP)
- [ ] SKILL.md with frontmatter checks (Sensei-equivalent)
- [ ] Body structure checks (Steps/Rules, code blocks, token count)
- [ ] Safety checks (secrets, bad patterns, destructive defaults)
- [ ] Scored report output
- [ ] README + AGENTS.md + copilot-instructions.md

### Phase 2: Deep Analysis
- [ ] Clarity scoring (self-contained steps, completion criteria)
- [ ] Discoverability scoring (trigger phrase analysis)
- [ ] Cross-file consistency (references/ exist, assets valid)
- [ ] Before/after examples in references/

### Phase 3: Auto-Fix
- [ ] Offer to rewrite description to hit High score
- [ ] Offer to add missing sections (Steps, completion criteria)
- [ ] Offer to restructure for progressive disclosure
- [ ] Submit to awesome-copilot wrapper generation

---

## Open Questions

1. Should skill-verify also check `.github/copilot-instructions.md` quality (not just SKILL.md)?
2. Should it integrate with awesome-copilot's validator directly (run `skill:validate` equivalent)?
3. Should it produce an HTML report like ai-ready does?
4. Should it have a "compare to top skills" mode that benchmarks against security-review, create-specification, etc.?

---

## Next Steps

1. Review this plan
2. Build Phase 1 (SKILL.md + scoring)
3. Test on ai-ready's own SKILL.md + 5 awesome-copilot skills
4. Submit to awesome-copilot as wrapper