Azure / Azure/git-ape

[meta] Eval suite coverage for agents and skills

Open
#93 0 comments 0 reactions 0 assignees View on GitHub
AI-evals enhancement
Dominant language
JavaScript
Stars
269
Forks
45
Avg merge
18h 41m
Merged PRs (30d)
11

Description

## Goal

Track per-skill and per-agent eval suite authoring on top of the eval harness work tracked in #61. Each sub-issue covers one skill or one agent and ships as a small, independently reviewable PR using the contributor loop documented in `CONTRIBUTING.md` and `docs/WAZA.md` (both land with the harness PRs).

## Contributor loop

1. Pick an unclaimed sub-issue below and assign yourself.
2. Run `/skill-bench ` or `/agent-bench ` to draft `eval.yaml` + tasks from the live `SKILL.md` / `.agent.md`.
3. Run `waza run .github/evals//eval.yaml -v` locally (`copilot-sdk` executor, requires `copilot login`).
4. Run `/skill-improve ` or `/agent-improve ` to iterate on graders and fix false positives.
5. Open a PR adding the suite and a `manifest.yaml` entry.
6. CI runs the mock executor; a maintainer dispatches a real-model run for final review.
7. After the suite is stable, `/skill-promote` (or `/agent-promote`) bumps it from `expanded` to `pilot` tier.

## Skills

Good-first-issue candidates (no Azure CLI / no live calls):

- Skill: `azure-naming-research`
- Skill: `azure-rest-api-reference`
- Skill: `azure-role-selector`

Standard skills:

- Skill: `azure-cost-estimator`
- Skill: `azure-policy-advisor`
- Skill: `azure-security-analyzer`
- Skill: `git-ape-onboarding` (most complex — defer until others land)

(`prereq-check` ships with the harness PR as the proof-of-pipe — no separate sub-issue needed.)

## Agents

Good-first-issue candidates:

- Agent: `azure-principal-architect`

Standard agents:

- Agent: `azure-iac-exporter`
- Agent: `azure-policy-advisor`
- Agent: `azure-requirements-gatherer`
- Agent: `azure-resource-deployer` (safety-sensitive — grade refusal / plan-only path, not real deploy)
- Agent: `azure-template-generator`
- Agent: `git-ape` (orchestrator — depends on most sub-agent suites being stable first)
- Agent: `git-ape-onboarding`

## Conventions

- One suite per PR. Don't bundle.
- Use the authoring prompts; don't hand-write YAML.
- Default new suites to `expanded` tier in `manifest.yaml`; promote after at least one clean real-model run.
- Mock executor is the only thing fork PRs can run. Real-model runs are maintainer-dispatched.

## Related

- Eval harness work: #61

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.