Azure / Azure/git-ape

Author eval suite for agent `git-ape-onboarding`

Open
#106 0 comments 0 reactions 1 assignee Claimed by @arnaudlh View on GitHub
AI-evals enhancement
Dominant language
JavaScript
Stars
269
Forks
45
Avg merge
18h 41m
Merged PRs (30d)
11

Description

## Agent

`git-ape-onboarding` — source: `.github/agents/git-ape-onboarding.agent.md`

## Scope

Author the eval suite at `.github/evals/agents/git-ape-onboarding/`:

- [ ] `eval.yaml` — suite config (executor, model, graders)
- [ ] At least 2 positive tasks under `tasks/positive-*.yaml`
- [ ] At least 1 negative task under `tasks/negative-*.yaml`
- [ ] Entry added to `.github/evals/manifest.yaml` at `tier: expanded`

## Notes

The onboarding agent gates on `az login` + input collection before scaffolding. Grade the gated step-1 reply (prereq table shown, missing prereqs flagged, inputs requested, plan preview) rather than expecting full execution in a single response. All-or-nothing criteria like "must mention OIDC AND envs AND workflows" will fail because the agent correctly stops at the auth gate.

Author this suite after the skill-level `git-ape-onboarding` suite has landed so they don't diverge.

## Procedure

1. `/agent-bench git-ape-onboarding` drafts the suite from the live `.agent.md`.
2. `waza run .github/evals/agents/git-ape-onboarding/eval.yaml -v` locally.
3. `/agent-improve git-ape-onboarding` to iterate on graders.
4. Open PR.
5. Mock CI runs automatically. A maintainer will dispatch a real-model run before merge.

## Acceptance

- [ ] Suite runs cleanly in `mock` executor.
- [ ] At least one positive task passes in a real-model run.
- [ ] All negative tasks produce a refusal or out-of-scope acknowledgement.
- [ ] `manifest.yaml` entry added; PR description includes the real-model run summary.

## Conventions to follow

- Persona lock: refusal graders should accept the agent's own scope language.
- Prompt graders need `continue_session: true` in their grader config.

## Related

- Umbrella: #93
- Harness: #61

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.