microsoft / microsoft/hve-core

feat: Integrate skill-bench with Prompt Builder workflow and npm scripts

Open
#1,507 0 comments 0 reactions 0 assignees View on GitHub
feature needs-triage skills
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

Part of the skill-bench integration epic: #1504

Depends on #1508 (core collection created).

Wire the skill-bench into the existing hve-core toolchain so it is runnable from the command line and optionally in CI.

**Scope:**
- Add `bench:run` and `bench:report` npm scripts to `package.json`
- Integrate bench EVAL mode into the Prompt Builder → Prompt Evaluator → Prompt Updater cycle so eval results can feed the authoring feedback loop
- Document CI output format decision (JUnit XML, TAP, or custom JSON matching existing `logs/` patterns)

**Acceptance criteria:**
- `npm run bench:run` executes the bench orchestrator
- `npm run bench:report` outputs a summary of results
- Integration with Prompt Evaluator is documented (even if fully automated wiring is deferred)
- Scripts work on both Windows (PowerShell) and Unix (bash)

Contributor guide

Open the contributing guide

Research direction

Start with package.json and locate the existing bench orchestrator, Prompt Builder, Prompt Evaluator, and Prompt Updater entry points. Check dependency #1508 before wiring the workflow. Done means both bench scripts run on PowerShell and bash, report results, and document the evaluator integration and chosen CI output format.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, cli, testing-qa, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.