microsoft / microsoft/hve-core

feat: Run skill-bench against existing hve-core skills to establish effectiveness baselines

Open
#1,505 0 comments 0 reactions 0 assignees View on GitHub
feature needs-triage skills
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

Part of the skill-bench integration epic: #1504

Depends on #1507 (npm integration complete).

Execute the bench against the current hve-core skill library to produce baseline effectiveness metrics and surface gaps where new or improved skills are needed.

**Scope:**
- Run bench in EVAL and COMPARE modes against all skills in production collections
- Identify skills scoring 0% baseline (requiring skill investment) vs. patterns learnable without skills
- Surface any undocumented architectural decisions revealed by failing expectations (candidates for ADRs or instructions)

**Acceptance criteria:**
- Baseline pass-rate table published for each skill
- Skills with 0% baseline (without-skill) flagged as candidates for new skills or skill improvements
- At least one ADR or instruction issue created for any design decision exposed by failing expectations in both conditions

Contributor guide

Open the contributing guide

Research direction

Start with the skill-bench integration epic in #1504 and confirm the npm integration dependency in #1507. Run EVAL and COMPARE against all skills in the production collections, then publish a per-skill baseline pass-rate table and create follow-up skill or ADR/instruction issues for the identified gaps and design decisions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.