Integrate automated skill effectiveness testing (agent-skill-bench)

Summary

Integrate an automated effectiveness testing framework for skills, agents, and instructions into hve-core. This directly implements the Q4 2026 roadmap item "Build automated testing for agent and instruction effectiveness" with a proven multi-agent orchestration system that has already demonstrated measurable results against hve-core artifacts.

Motivation

hve-core currently has zero quality gates beyond structural validation (validate:skills). There is no mechanism to verify that a skill actually improves agent output quality. An external prototype (agent-skill-bench) has been evaluated against hve-core artifacts and demonstrates that automated effectiveness testing is both feasible and high-impact.

Key Benchmark Analysis: What the Data Reveals About HVE Core

An hve-artifact-authoring skill was benchmarked using blind A/B comparison (with-skill vs baseline) across 8 evaluations containing 34 expectations:

Critical Findings for hve-core

1. Skills are essential for custom conventions, not general patterns. Three evals (collection manifest authoring, orchestrator agent creation, delegation model design) show 0% baseline → 100% with-skill. These test HVE-specific patterns that an LLM cannot infer from general knowledge alone.

2. Some patterns are already learnable without skills. One eval (add-skill-with-scripts) scored 100% in both conditions, meaning the patterns it tests are redundant with information already in copilot-instructions.md.

3. The discriminating ratio (53%) identifies where skill investment matters. Of 34 expectations, 18 uniquely measure skill value. The remaining 15 pass in both conditions (redundant) or fail in both (undocumented design decisions). This metric enables lean eval suites.

4. One undocumented architectural decision was surfaced. A "parallel CI jobs" expectation fails in both conditions, revealing a design-fork that should become an ADR or instruction rather than a skill expectation.

5. Skills dramatically outperform baseline on frontmatter contracts and packaging conventions — exactly the areas where hve-core has custom conventions that deviate from defaults.

Strategic Implications

• Three patterns urgently need skills: collection manifest authoring, orchestrator handoffs/delegation, and the agents: frontmatter contract. All scored 0% baseline.
• The +53% delta validates hve-core's skill model. The investment in artifact packaging and distribution is justified by measurable quality improvement.
• Eval results can drive the authoring feedback loop. Bench results could feed into the existing Prompt Builder → Prompt Evaluator → Prompt Updater cycle.

Proposed Approach

Integrate the bench framework as a new collection (initially at experimental maturity) following a phased contribution:

1. Phase 0: Fix critical functional blockers and frontmatter compliance
2. Phase 1: Core eval framework — agents, SKILL.md, and collection manifest
3. Phase 2: Integration with Prompt Builder workflow and npm scripts
4. Phase 3: Run bench against existing hve-core skills to establish baselines

The bench uses a 5-mode lifecycle (CREATE → EVAL → IMPROVE → COMPARE → ASSESS) with a parent orchestrator dispatching specialized subagents, following the same architectural patterns already used throughout hve-core.

Architectural Alignment

Known Gaps (from analysis)

A 27-finding quality review identified prerequisites before integration:

• 3 Critical: Missing subagent file, workspace naming mismatch, missing agents: frontmatter
• 12 Major: SKILL.md over 500-line limit, missing name: frontmatter, Unix-only commands, non-standard section headers
• 12 Minor: Style and documentation issues

All criticals are fixable in 1-2 days. Total conformance effort estimated at 3-4 weeks.

Quick Win (Independent of Framework Integration)

The hve-artifact-authoring skill itself could land in the coding-standards collection immediately with minimal integration friction. It demonstrates 96% pass rate and follows standard skill structure.

Discussion Points for Maintainers

• Should the bench land as its own collection (skill-bench) or be distributed across existing collections?
• What maturity level at launch: experimental or preview?
• Should benchmark results gate PR merges for skill changes?
• How should the bench interact with the existing Prompt Builder / Evaluator / Updater cycle?
• Preferred CI output format: JUnit XML, TAP, or custom JSON matching existing logs/ patterns?

References

• Q4 2026 Roadmap: "Build automated testing for agent and instruction effectiveness"
• Source repository: gim-home/agent-skill-bench (EMU-protected)
• Related: Prompt Builder agent workflow, validate:skills npm script